APP MTH 7090 - Stochastic Decision Theory

North Terrace Campus - Semester 2 - 2019

People make decisions everyday: whether to take an umbrella to work; to take an available park for their car or continue to search for a better one; which of several possible methods to implement to attempt to save a species from extinction; and, which people in the population to give a vaccine to. All of these decisions are being made under uncertainty: there exists a certain chance of rain today; a certain chance all of the car parks are used; uncertainty about how many individuals of the species exist and how they will respond to each of the possible interventions; and, the actual dynamics of the infection and the uptake of the vaccine by the population. This course will focus on formulating problems of this type in a mathematical framework and provide methods for making the best decision possible taking into account the uncertainty. Topics covered are: stochastic linear programming - the extension of linear programming to account for uncertainty; Markov decision processes (MDP) and dynamic programming - the framework for solving problems in which the state of the process up to the time of decision is known but the behaviour of the process is governed by a Markov chain; Hidden Markov models, and Partially-observable MDPs - the extension of MDPs where we can only observe a `noisy' version of the state of the system.

  • General Course Information
    Course Details
    Course Code APP MTH 7090
    Course Stochastic Decision Theory
    Coordinating Unit Mathematical Sciences
    Term Semester 2
    Level Postgraduate Coursework
    Location/s North Terrace Campus
    Units 3
    Contact Up to 3 hours per week
    Available for Study Abroad and Exchange Y
    Prerequisites MATHS 2103 or APP MTH 3001
    Assumed Knowledge Knowledge of linear programming such as would be obtained from APP MTH 2105
    Assessment Ongoing assessment, exam
    Course Staff

    Course Coordinator: Dr Giang Nguyen

    Course Timetable

    The full timetable of all activities for this course can be accessed from Course Planner.

  • Learning Outcomes
    Course Learning Outcomes

    1 Be able to formulate Deterministic Equivalent Problems (DEPs) for Stochastic Linear Programs and solve them under certain assumptions.
    2 Understand the Principle of Optimality and Dynamic Programming and be able to use Dynamic Programming to solve appropriate problems.
    3 Be able to specify a Markov Decision Chain (MDC).
    4 Be able to formulate and solve Finite Horizon MDC Programs, simple Infinite Horizon MDC Programs with Discounting, simple Positive MDC Programs, simple Negative MDC Programs and simple Average-Cost MDC Programs.
    5 Understand Value-Iteration and Policy-Improvement Algorithms, be able to identify and specify a Hidden Markov Chain (HMC) model and to evaluate quantities of interest.
    6 Demonstrate skills in communicating mathematics both orally and in writing.
    University Graduate Attributes

    This course will provide students with an opportunity to develop the Graduate Attribute(s) specified below:

    University Graduate Attribute Course Learning Outcome(s)
    Deep discipline knowledge
    • informed and infused by cutting edge research, scaffolded throughout their program of studies
    • acquired from personal interaction with research active educators, from year 1
    • accredited or validated against national or international standards (for relevant programs)
    Critical thinking and problem solving
    • steeped in research methods and rigor
    • based on empirical evidence and the scientific approach to knowledge development
    • demonstrated through appropriate and relevant assessment
    Career and leadership readiness
    • technology savvy
    • professional and, where relevant, fully accredited
    • forward thinking and well informed
    • tested and validated by work based experiences
  • Learning Resources
    Required Resources
    Recommended Resources
    There are many good books on Stochastic Decision Theory in the Barr Smith Library, with the following texts and articles being recommended for this course (many of which should be available electronically).

    1. "Statistical Modelling and Computation", by D.P. Kroese and J.C.C. Chan (Springer, 2014).
    2. "Probability and Random Processess", by G. Grimmett and D. Stirzaker (Oxford University Press, 2001).
    3. "Stochastic Programming", by P. Kall and S.W. Wallace (John Wiley & Sons, 1994).
    4. "Stochastic Linear Programming", by P. Kall and J. Mayer (Springer, 2011).
    5. "Markov Decision Processes: Discrete Stochastic Dynamic Programming", by M.L. Puterman (John Wiley & Sons).
    6. "Stochastic Dynamics Programming and the Control of Queueing Systems", by L.I. Sennott (John Wiley & Sons).
    7."What HMMs Can Do", by J.A. Bilmes, Bilmes, J. A. (2006), IEICE - Transactions on Information and Systems E89-D(3), 869–891.
    8. "A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition", by L.R. Rabiner (1989), Proceedings of the IEEE 77(2), 257–286.
    9. "Hidden Markov Models: Estimation and Control", by R.J. Elliott, L. Aggoun and J.B. Moore (Springer-Verlag).
    Online Learning
    A version of the course notes will available online for those whowish to download and print prior to attending lectures. The format (either as two or one slide per page) is the same as the presentation slides used in the lectures, with room for you to annotate during lectures.

    All assignments, tutorials, handouts and solutions, where appropriate, will be made available on the course website as the course ensues.

    Please don't hesitate to e-mail the lecturer should anything be missing.
  • Learning & Teaching Activities
    Learning & Teaching Modes
    This course relies on lectures as the primary delivery mechanism for the material. The lecturer will guide the students through the material presented in this course in a total of 33 lectures. Downloading and prereading the online notes will enable the students to more actively engage the material and interact during lectures.

    Tutorials supplement the lectures by providing exercises and example problems to enhance the understanding obtained through lectures. A sequence of written assignments provides assessment opportunities for students to gauge their progress and understanding.

    The information below is provided as a guide to assist students in engaging appropriately with the course requirements.

    Activity Quantity Workload hours
    Lectures 33 99
    Tutorials 5 25
    Assignments 5 30
    Total 154
    Learning Activities Summary
    Lecture outline

    Introduction to Stochastic Decision Theory (1 Lecture)
    Revision of Basic Probability, Discrete-time Markov chains, Linear Programming and Convexity (5 Lectures) Stochastic Linear Programming (9 Lectures), including
         -General Formulation (1 Lecture)
         -Recourse Deterministic Equivalent Problems (DEPs) (5 Lectures)
         -Chance Constrained DEPs (3 Lectures)
    Markov Decision Chains (10 Lectures), including
         -The Principle of Optimality and Dynamic Programming (1 Lecture)
         -Introduction to Markov Decision Chains and Finite Horizon Programming (1 Lecture)
         -Infinite Horizon Programming, with Discounting (1 Lecture)
         -Positive Programming and the Value Iteration Algorithm (2 Lectures)
         -Negative Programming and Optimal Stopping (2 Lectures)
         -Average Cost Programming and the Policy Improvement Algorithm (3 Lectures)
    Hidden Markov Chains (7 Lectures), including
         -Introduction to Hidden Markov Chains (1 Lecture)
         -Smoothing and the Forward-Backward Algorithm (2 Lectures)
         -Optimal State Sequence and the Viterbi Algorithm (2 Lectures)
         -Estimation of Parameters and the Baum-Welch Algorithm (2 Lectures)
    Summary (1 Lecture)

    The first tutorial in Week 3 covers material from the previous two weeks and other material that should be considered revision. Tutorials in Weeks 5, 7, 9 and 11 cover the material of the previous few weeks.
  • Assessment

    The University's policy on Assessment for Coursework Programs is based on the following four principles:

    1. Assessment must encourage and reinforce learning.
    2. Assessment must enable robust and fair judgements about student performance.
    3. Assessment practices must be fair and equitable to students and give them the opportunity to demonstrate what they have learned.
    4. Assessment must maintain academic standards.

    Assessment Summary
    Assessment task Task type When due Weighting Learning outcomes
    Examination Summative Examination period 70% All
    Assignments Formative and summative Weeks 4, 6, 8, 10 and 12 30% All
    Assessment Related Requirements
    An aggregate score of 50% is required to pass this course.  
    Assessment Detail
    Assessment task Set Due Weighting
    Assignment 1 week 3 week 4 6%
    Assignment 2 week 5 week 6 6%
    Assignment 3 week 7 week 8 6%
    Assignment 4 week 9 week 10 6%
    Assignment 5 week 11 week 12 6%
    Assignments must be submitted on time to the designated hand-in box in the School of Mathematical Sciences with a signed assessment cover sheet attached to the assignment.

    Late assignments will not be accepted.

    Assignments will normally be returned within two weeks.

    Students may be excused from an assignment for medical or compassionate reasons. In such cases, documentation is required and the lecturer must be notified as soon as possible.
    Course Grading

    Grades for your performance in this course will be awarded in accordance with the following scheme:

    M10 (Coursework Mark Scheme)
    Grade Mark Description
    FNS   Fail No Submission
    F 1-49 Fail
    P 50-64 Pass
    C 65-74 Credit
    D 75-84 Distinction
    HD 85-100 High Distinction
    CN   Continuing
    NFE   No Formal Examination
    RP   Result Pending

    Further details of the grades/results can be obtained from Examinations.

    Grade Descriptors are available which provide a general guide to the standard of work that is expected at each grade level. More information at Assessment for Coursework Programs.

    Final results for this course will be made available through Access Adelaide.

  • Student Feedback

    The University places a high priority on approaches to learning and teaching that enhance the student experience. Feedback is sought from students in a variety of ways including on-going engagement with staff, the use of online discussion boards and the use of Student Experience of Learning and Teaching (SELT) surveys as well as GOS surveys and Program reviews.

    SELTs are an important source of information to inform individual teaching practice, decisions about teaching duties, and course and program curriculum design. They enable the University to assess how effectively its learning environments and teaching practices facilitate student engagement and learning outcomes. Under the current SELT Policy ( course SELTs are mandated and must be conducted at the conclusion of each term/semester/trimester for every course offering. Feedback on issues raised through course SELT surveys is made available to enrolled students through various resources (e.g. MyUni). In addition aggregated course SELT data is available.

  • Student Support
  • Policies & Guidelines
  • Fraud Awareness

    Students are reminded that in order to maintain the academic integrity of all programs and courses, the university has a zero-tolerance approach to students offering money or significant value goods or services to any staff member who is involved in their teaching or assessment. Students offering lecturers or tutors or professional staff anything more than a small token of appreciation is totally unacceptable, in any circumstances. Staff members are obliged to report all such incidents to their supervisor/manager, who will refer them for action under the university's student’s disciplinary procedures.

The University of Adelaide is committed to regular reviews of the courses and programs it offers to students. The University of Adelaide therefore reserves the right to discontinue or vary programs and courses without notice. Please read the important information contained in the disclaimer.