S-114.4220 Research Seminar on Computational Science
	TKK / Units / LCE / Teaching / Courses / S-114.4220 / s2007

The topic of the course in fall 2007 is

Stochastic and Adaptive Control of Uncertain Systems (5 p) L V

Topic of the seminar is automatic control of uncertain dynamic systems, where a control signal is generated based on observations made from the system and the control signal is used for actively changing the behaviour of the system in order to reach a certain predefined goal. Examples of such control tasks are guidance of a spacecraft, control of a chemical process and steering of a mobile robot.

In the seminar, the purpose is:

1. to present a review of optimal control, optimal estimation and stochastic control. In these classical approaches, the uncertainties are modeled with random processes and controls are selected to optimize certain model criteria either using variational calculus, or Hamilton-Jacobi-Bellman equations and Bayesian decision theory.
2. to present review of adaptive and learning control approaches. In these methods, either the value functions (cost-to-go functions) are directly optimized based on a model or some other means are used for adapting the model to the observed environment.
3. to find the connection between the above two approaches and the connection to the classical pole placement based control design methods.

Teachers

Dr.Tech. Simo Särkkä, Ph.D. Tuomas Lukka

Prerequisites

Basics of probability theory, differential equations, matrix algebra and Bayesian inference. Basic knowledge or ability to learn to use Matlab is needed for completing the exercise.

Place and time

The seminars take place on Mondays at 12:15 in Innopoli 2, B317 (the seminar room of Laboratory of computational engineering, see map). The first lecture is 17.09.2007.

Course Registration

Please register for the course by sending email to ssarkka@lce.hut.fi Remember to mention the student id. If you have an suggestion for the topic of your presentation, you may mention it in the email also.

Language

English if at least one English speaking participant.

Contents

Course requirements:

• Active participation in the seminars
• Seminar presentation
• Homework exercises (pen & paper)
• Exercise work (computer simulation)

Control engineering and optimal control:

• [1] D. E. Kirk (1970), Optimal Control Theory. Prentice Hall (Dover 2004).
• [2] B. D. O. Anderson and J. B. Moore (1989), Optimal Control: Linear Quadratic Methods. Prentice-Hall (Dover 2007). (Scanned PDF).
• [3] R. C. Dorf and R. H. Bishop (2004), Modern Control Systems, Prentice Hall.
• [4] T. Glad and L. Ljung (2000), Control Theory: Multivariable and Nonlinear Methods, Taylor & Francis.

Stochastic optimal control and estimation:

• [5] R. Stengel (1994), Optimal Control and Estimation, Dover Publications, New York.
• [6] B. D. O. Anderson and J. B. Moore (1979), Optimal Filtering. Prentice-Hall (Dover 2005). (Scanned PDF).
• [7] P. Maybeck (1982). Stochastic Models, Estimation and Control, Volume 3. Academic Press.
• [8] M. Aoki (1967), Optimization of Stochastic Systems. Academic Press, New York.

Approximate dynamic programming:

• [9] D. P. Bertsekas (2001), Dynamic Programming and Optimal Control, Volume 1, Athena Scientific.
• [10] D. P. Bertsekas (2001), Dynamic Programming and Optimal Control, Volume 2, Athena Scientific.
• [11] R. S. Sutton, A. G. Barto (1998), Reinforcement Learning: An Introduction. (Online copy).

Adaptive control:

• [12] K. Åström and B. Wittenmark (1995), Adaptive Control, Addison-Wesley (2nd ed.).
• [13] S. Sastry and M. Bodson (1989), Adaptive Control: Stability, Convergence, and Robustness. (Scanned PDF).

Potential topics of presentations include, but are not limited to:

• Deterministic optimal control: Calculus of variations, dynamic programming, Hamilton-Jacobi-Bellman equations, LQ-regulators, numerical methods, approximation methods, H-infinity control, robust optimal control
• Stochastic optimal control: stochastic Hamilton-Jacobi-Bellman equations, stochastic dynamic programming, LQG-regulators, optimal filtering methods, certainty equivalent control, neighboring optimal control, numerical and approximation methods
• Suboptimal and adaptive control: dual control, self tuning regulators, model reference adaptive systems, recursive parameter estimation, system identification
• Approximate dynamic programming: model predictive control, policy iteration, Q-learning, reinforcement learning
• Classical control: pole placement design, PID controllers, observers, connections between classical and optimal/stochastic control
• Applications

Schedule

General instructions

In every seminar talk, there is the author who writes a 2-4 page summary paper about the subject and gives the talk, and an opponent, whose task is to make questions and stimulate discussion after the presentation. Every student is the author for (at least) one talk and the opponent for another talk. The author should send the summary paper to the opponent and to the teachers on the thursday preceding the presentation day. The slides should be sent to the teacher lastest on the day preceding the presentation. The opponent prepares at least 2-3 questions about the topic to stimulate discussion after the talk. Normal talk is 30-40 minutes.

Homework exercises

Available here. DL 31.1.2008.

Exercise work

Available here. DL 29.2.2008.

Additional Online Material:

• MIT OpenCourseWare: Dynamic Programming and Stochastic Control
• IUC's Course: Math 574 Applied Optimal Control.
• Brian D.O. Anderson's list of publications contains, in addition to many classical articles, PDF versions of some classical estimation/control books (e.g., Optimal Filtering and Optimal Control: Linear Quadratic Methods).
• The Kalman Filter Page contains much interesting material on linear optimal estimation, that is, Kalman filtering.
• H.J. Kappen (2006), An introduction to stochastic control theory, path integrals and reinforcement learning. Proceedings 9th Granada seminar on Computational Physics. (PDF)
• Control Theory in Wikipedia
• Control Systems Wikibook

Contact:

When sending email concerning the course, please add "S-114.4220" or "1144220" to subject.

Dr.Tech. Simo Särkkä
ssarkka@lce.hut.fi

Ph.D. Tuomas Lukka
tjl@lce.hut.fi