Steady State & Long-term Probabilities (DP IB Applications & Interpretation (AI)): Revision Note

Steady State & Long-term Probabilities

What is the steady state of a regular Markov chain?

• The vector s is said to be a steady state vector if it does not change when multiplied by the transition matrix

  • Ts = s

• Regular Markov chains have steady states

  • A Markov chain is said to be regular if there exists a positive integer k such that none of the entries are equal to 0 in the matrix T^k

    • For this course all Markov chains will be regular

• The transition matrix for a regular Markov chain will have exactly one eigenvalue equal to 1 and the rest will all be less than 1

• As n gets bigger Tⁿ tends to a matrix where each column is identical

  • The column matrix formed by using one of these columns is called the steady state column matrix s

  • This means that the long-term probabilities tend to fixed probabilities

    • s_n tends to s

How do I use long-term probabilities to find the steady state?

• As Tⁿ tends to a matrix whose columns equal the steady state vector

  • Calculate Tⁿ for a large value of n using your GDC

  • If the columns are identical when rounded to a required degree of accuracy then the column is the steady state vector

  • If the columns are not identical then choose a higher power and repeat

How do I find the exact steady state probabilities?

• As Ts = s the steady state vector s is the eigenvector of T corresponding to the eigenvalue equal to 1 whose elements sum to 1:

  • Let s have entries x₁, x₂, ..., x_n

  • Use Ts = s to form a system of linear equations

  • There will be an infinite number of solutions so choose a value for one of the unknowns

    • For example: let x_n = 1

  • Ignoring the last equation solve the system of linear equations to find x₁, x₂, ..., x_{n – 1}

  • Divide each value x_i by the sum of the values

    • This makes the values add up to 1

• You might be asked to show this result using diagonalisation

  • Write T = PDP^-1 where D is the diagonal matrix of eigenvalues and P is the matrix of eigenvectors

  • Use Tⁿ = PDⁿP^-1

  • As n gets large Dⁿ tends to a matrix where all entries are 0 apart from one entry of 1 due to the eigenvalue of 1

  • Calculate the limit of Tⁿ which will have identical columns

    • You can calculate this by multiplying the three matrices (P, D^∞, P^-1) together