NDAK23003U Simulation-based Reinforcement Learning (SiRL)
Combining physical simulation with machine learning opens up the possibility of solving a large repertoire of challenges that previously have been impossible to solve. It is a future-enabling technology. This simulation-based learning approach is highly relevant in scenarios where data is limited or not easily available. In such a scenario the simulator provides a model or prior for the learning process. The learning-based process can on the other hand bridge the gap to reality as it allows the simulator to incorporate knowledge of the real world better through data. In combination, both flavours combined can provide solutions for the control of very advanced systems, typical robots, for solving quite complex problems. As a whole, this field of simulation-based learning addresses many interesting issues such as motion control, motion imitation, and system identification like parameter estimation, or digital twinning, for instance creating data-driven models of humans, etc. These are highly relevant for solving actual real-world challenges. Such as advanced autonomous robotics manipulation and navigation, prediction or detection of human motion for improved health-care services, creation of realistic computer animations for XR, movies, and computer games, or in digital twining such as learning the physics of the carbon dioxide cycle in soil which is an important aspect of predicting climate consequences from farming. Many more examples exist. This course will prepare students for working with this world-changing technology.
This course is ideal for students who wish to work with simulation-based learning in their master's thesis. It is particularly ideal for students who want to work with robotics or character animation, or content creation for XR.
Students are expected to solve weekly practical exercises mostly programming with python, but some C++ could be involved, as well as some lab-space work as well. Students are expected to be self-driven in getting the software to work on their own devices, like installing an auto differentiation framework and making it run. Students could be exposed to working with vision or motion capture data.
Knowledge of
- Fundamentals of articulated body dynamics.
- Using computer vision and motion capture data as labels on loss functions
- Concepts in optimal control such as open and closed loop control, PD controllers, etc.
- Computational Graphs, forward- and reverse-mode automatic differentiation.
- The adjoint method, adjoint state equation, and sensitivity analysis.
- Techniques for program transformation
- Auto-differentiation tools & differentiable simulator frameworks such as DiffTaiChi, Phyton Warp, JAX, MuJoCo, and more.
- Simulation-based reinforcement learning, model-predictive control (MPC).
- Augmenting differentiable simulators with neural networks.
Skills in
- Using a differentiable simulator to compute gradients.
- Deriving the models for trajectory optimization and policy optimization and explaining their differences.
Competences to
- Solve motion imitation problems, such as inferring dynamics from video or motion capture data.
- Solve motion control problems such as grasping, manipulation, locomotion, or navigation.
- Solve system identification problems such as parameter estimation from video.
See Absalon when the course is set up.
We recommend students to have passed courses that involve python programming, linear algebra, calculus (analytical computing derivatives of functions), and solving minimization problems with gradient descent methods.
- Category
- Hours
- Lectures
- 28
- Preparation
- 52
- Exercises
- 28
- Project work
- 70
- Exam Preparation
- 27
- Exam
- 1
- Total
- 206
As
an exchange, guest and credit student - click here!
Continuing Education - click here!
PhD’s can register for MSc-course by following the same procedure as credit-students, see link above.
- Credit
- 7,5 ECTS
- Type of assessment
- Oral examination, 30 minutes (no preparation time)
- Type of assessment details
- The exam takes an outset in theory taught over the course. Students should be able to derive theory/math on the blackboard during the examination.
- Exam registration requirements
To qualify for the exam, the student must complete all reports. There will be 3-4 short reports during the course. The reports can be made as a group or as individual reports. Each report should be a maximum of 5 pages.
- Aid
- Without aids
- Marking scale
- 7-point grading scale
- Censorship form
- No external censorship
Several internal examiners
- Re-exam
Same as the ordinary exam.
Criteria for exam assesment
In order to obtain the grade 12 the student should convincingly and accurately demonstrate the knowledge, skills and competences described under Learning Outcome.
Course information
- Language
- English
- Course code
- NDAK23003U
- Credit
- 7,5 ECTS
- Level
- Full Degree Master
- Duration
- 1 block
- Placement
- Block 4
- Schedule
- A
- Course capacity
- 30
The number of places might be reduced if you register in the late-registration period (BSc and MSc) or as a credit or single subject student.
Study board
- Study Board of Mathematics and Computer Science
Contracting department
- Department of Computer Science
Contracting faculty
- Faculty of Science
Course Coordinators
- Hang Yin (4-777088784f73783d7a843d737a)