NDAK23003U Simulation-based Reinforcement Learning (SiRL)
Combining physical simulation with machine learning opens up the possibility of solving a large repertoire of challenges that previously have been impossible to solve. It is a future-enabling technology. This simulation-based learning approach is highly relevant in scenarios where data is limited or not easily available. In such a scenario the simulator provides a model or prior for the learning process. The learning-based process can on the other hand bridge the gap to reality as it allows the simulator to incorporate knowledge of the real world better through data. In combination, both flavours combined can provide solutions for the control of very advanced systems, typical robots, for solving quite complex problems. As a whole, this field of simulation-based learning addresses many interesting issues such as motion control, motion imitation, and system identification like parameter estimation, or digital twinning, for instance creating data-driven models of humans, etc. These are highly relevant for solving actual real-world challenges. Such as advanced autonomous robotics manipulation and navigation, prediction or detection of human motion for improved health-care services, creation of realistic computer animations for XR, movies, and computer games, or in digital twining such as learning the physics of the carbon dioxide cycle in soil which is an important aspect of predicting climate consequences from farming. Many more examples exist. This course will prepare students for working with this world-changing technology.
This course is ideal for students who wish to work with simulation-based learning in their master's thesis. It is particularly ideal for students who want to work with robotics or character animation, or content creation for XR.
Students are expected to solve weekly practical exercises mostly programming with python, but some C++ could be involved, as well as some lab-space work as well. Students are expected to be self-driven in getting the software to work on their own devices, like installing an auto differentiation framework and making it run. Students could be exposed to working with vision or motion capture data.
- Fundamentals of articulated body dynamics.
- Using computer vision and motion capture data as labels on loss functions
- Concepts in optimal control such as open and closed loop control, PD controllers, etc.
- Computational Graphs, forward- and reverse-mode automatic differentiation.
- The adjoint method, adjoint state equation, and sensitivity analysis.
- Techniques for program transformation
- Auto-differentiation tools & differentiable simulator frameworks such as DiffTaiChi, Phyton Warp, JAX, MuJoCo, and more.
- Simulation-based reinforcement learning, model-predictive control (MPC).
- Augmenting differentiable simulators with neural networks.
- Using a differentiable simulator to compute gradients.
- Deriving the models for trajectory optimization and policy optimization and explaining their differences.
- Solve motion imitation problems, such as inferring dynamics from video or motion capture data.
- Solve motion control problems such as grasping, manipulation, locomotion, or navigation.
- Solve system identification problems such as parameter estimation from video.
See Absalon when the course is set up.
We recommend students to have passed courses that involve python programming, linear algebra, calculus (analytical computing derivatives of functions), and solving minimization problems with gradient descent methods.
- Project work
- Exam Preparation
PhD’s can register for MSc-course by following the same procedure as credit-students, see link above.
- 7,5 ECTS
- Type of assessment
- Oral examination, 30 minutes
- Type of assessment details
- Oral examination of 30 minutes without preparation. The exam takes an outset in theory taught over the course. Students should be able to derive theory/math on the blackboard during the examination.
- Exam registration requirements
To qualify for the exam the student must complete 5 out of a maximum of 7 short reports which can be made as a group or as individual reports. The written reports should be maximally 10 pages.
- All aids allowed
- Marking scale
- 7-point grading scale
- Censorship form
- No external censorship
Several internal examiners
The reexam format and qualification for reexam are the same as for the regular exam.
Criteria for exam assesment
In order to obtain the grade 12 the student should convincingly and accurately demonstrate the knowledge, skills and competences described under Learning Outcome.
- Course code
- 7,5 ECTS
- Full Degree Master
- 1 block
- Block 4
- Course capacity
- Maximum limit of 30 students.
The number of seats may be reduced in the late registration period.
- Study Board of Mathematics and Computer Science
- Department of Computer Science
- Faculty of Science
- Kenny Erleben (5-6f6972727d44686d326f7932686f)