NDAK23003U Simulation-based Reinforcement Learning (SiRL)

Volume 2024/2025
Content

Combining physical simulation with machine learning opens up the possibility of solving a large repertoire of challenges that previously have been impossible to solve. It is a future-enabling technology. This simulation-based learning approach is highly relevant in scenarios where data is limited or not easily available. In such a scenario the simulator provides a model or prior for the learning process. The learning-based process can on the other hand bridge the gap to reality as it allows the simulator to incorporate knowledge of the real world better through data. In combination, both flavours combined can provide solutions for the control of very advanced systems, typical robots, for solving quite complex problems. As a whole, this field of simulation-based learning addresses many interesting issues such as motion control, motion imitation, and system identification like parameter estimation, or digital twinning, for instance creating data-driven models of humans, etc. These are highly relevant for solving actual real-world challenges. Such as advanced autonomous robotics manipulation and navigation, prediction or detection of human motion for improved health-care services, creation of realistic computer animations for XR, movies, and computer games, or in digital twining such as learning the physics of the carbon dioxide cycle in soil which is an important aspect of predicting climate consequences from farming. Many more examples exist. This course will prepare students for working with this world-changing technology.

This course is ideal for students who wish to work with simulation-based learning in their master's thesis. It is particularly ideal for students who want to work with robotics or character animation, or content creation for XR.

Students are expected to solve weekly practical exercises mostly programming with python, but some C++ could be involved, as well as some lab-space work as well. Students are expected to be self-driven in getting the software to work on their own devices, like installing an auto differentiation framework and making it run. Students could be exposed to working with vision or motion capture data.

Learning Outcome

Knowledge of

  • Fundamentals of articulated body dynamics.
  • Using computer vision and motion capture data as labels on loss functions
  • Concepts in optimal control such as open and closed loop control, PD controllers, etc.
  • Computational Graphs, forward- and reverse-mode automatic differentiation.
  • The adjoint method, adjoint state equation, and sensitivity analysis.
  • Techniques for program transformation
  • Auto-differentiation tools & differentiable simulator frameworks such as DiffTaiChi, Phyton Warp, JAX, MuJoCo, and more.
  • Simulation-based reinforcement learning, model-predictive control (MPC).
  • Augmenting differentiable simulators with neural networks.

 

Skills in

  • Using a differentiable simulator to compute gradients.
  • Deriving the models for trajectory optimization and policy optimization and explaining their differences.


Competences to

  • Solve motion imitation problems, such as inferring dynamics from video or motion capture data.
  • Solve motion control problems such as grasping, manipulation, locomotion, or navigation.
  • Solve system identification problems such as parameter estimation from video.

See Absalon when the course is set up.

We recommend academic qualifications equivalent to a BSc degree.

We recommend students to have passed courses that involve python programming, linear algebra, calculus (analytical computing derivatives of functions), and solving minimization problems with gradient descent methods.
A mixture of lectures and exercise classes on project group work with written hand-ins.
  • Category
  • Hours
  • Lectures
  • 28
  • Preparation
  • 52
  • Exercises
  • 28
  • Project work
  • 70
  • Exam Preparation
  • 27
  • Exam
  • 1
  • Total
  • 206
Written
Oral
Continuous feedback during the course of the semester
Credit
7,5 ECTS
Type of assessment
Oral examination, 30 minutes (no preparation time)
Type of assessment details
The exam takes an outset in theory taught over the course. Students should be able to derive theory/math on the blackboard during the examination.
Exam registration requirements

To qualify for the exam, the student must complete all reports. There will be 3-4 short reports during the course. The reports can be made as a group or as individual reports. Each report should be a maximum of 5 pages. 

Aid
Without aids
Marking scale
7-point grading scale
Censorship form
No external censorship
Several internal examiners
Re-exam

Same as the ordinary exam.

Criteria for exam assesment

In order to obtain the grade 12 the student should convincingly and accurately demonstrate the knowledge, skills and competences described under Learning Outcome.