4310 : Intertemporal macroeconomics 1/22 4310 : Intertemporal macroeconomics Espen Henriksen August 27, 2008 4310 : Intertemporal macroeconomics Introduction Administrative 2/22 Administrative 1 Final exam 2 Week without classes 3 Evaluation/feedback ­ class representatives 4 Communication 5 Questions?? 4310 : Intertemporal macroeconomics Introduction Objectives 3/22 Objectives How to model uncertainty. How to compute the value of being at a given state (uncertainty, but no decisions). More specifically 1 Uncertainty and expected utility 2 Probability theory and Markov processes 3 Solving a Markov process: Dynamic programming and value iteration 4310 : Intertemporal macroeconomics Introduction Objectives 4/22 Building the toolbox A B C D No uncertainty Uncertainty No decisions Decisions A : Solow growth model B : Value iteration on a Markov process C : Ramsey growth model D : Stochastic neoclassical growth model 4310 : Intertemporal macroeconomics Preview Expected utility theory 5/22 Expected utility theory Standard textbook in microeconomics. 4310 : Intertemporal macroeconomics Preview Markov chains 6/22 i.i.d. A sequence of random variables is independent and identically distributed (i.i.d.) if each has the same probability distribution as the others and all are mutually independent. Examples All other things being equal, ... ... a sequence of outcomes of spins of a roulette wheel is i.i.d. ... a sequence of dice rolls is i.i.d. ... a sequence of coin flips is i.i.d. 4310 : Intertemporal macroeconomics Preview Markov chains 7/22 Modelling uncertainty The two main types of modelling techniques that macroeconomists make use of are: Markov chains Linear stochastic difference equations, e.g. an AR(1) process Markov property Markov chains and AR(1) processes have the Markov property. The Markov property means that for a given process, knowledge of the previous states is irrelevant for predicting the probability of subsequent states. For example, in the case we would predict a student's grades on a sequence of exams in a course. Taking the model to the measurements. 4310 : Intertemporal macroeconomics Preview Markov chains 8/22 Markov chains: Some terminology A set of states, S = {s1, s2, . . . , sr}. The process moves successively from one state to another. Each move is called a step. If the chain is currently in state si, then it moves to state sj at the next step with a probability denoted by pij. pij does not depend on any other information than that the chain is currently in state si. The probabilities pij are called transition probabilities. The process can remain in the state it is in, and this occurs with probability pii. 4310 : Intertemporal macroeconomics Preview Markov chains 9/22 The Markov property Formally, Pr (Xn+1 = x | Xn = xn, . . . , X1 = x1) = Pr (Xn+1 = x | Xn = xn) 4310 : Intertemporal macroeconomics Preview Markov chains 10/22 Example: Weather transitions R O S 1 4 1 4 1 4 1 4 1 2 1 2 1 2 1 2 where R is rain, O is overcast, and S is sunshine. 4310 : Intertemporal macroeconomics Preview Markov chains 11/22 Represented as a transition matrix t + 1 R O S R 0.50 0.25 0.25 t O 0.25 0.50 0.25 S 0.50 0.50 0.00 Such a square array is called the matrix of transition probabilities, or the transition matrix. We denote the probability that, given the chain is in state i today, it will be in state j n days from now p (n) ij . What is the probability that it will be overcast in two days if it is overcast today? 4310 : Intertemporal macroeconomics Preview Markov chains 12/22 Represented as a transition matrix The weather today is known to be overcast. This can represented by the following vector: x(0) = 0 1 0 The weather tomorrow (one day from now) can be predicted by x(1) = x(0) = 0 1 0 0.50 0.25 0.25 0.25 0.50 0.25 0.50 0.50 0.00 = 0.25 0.50 0.25 The weather two days from now can be predicted by x(2) = x(1) = 0.25 0.50 0.25 0.50 0.25 0.25 0.25 0.50 0.25 0.50 0.50 0.00 = 0.3750 0.4375 0.1875 4310 : Intertemporal macroeconomics Preview Markov chains 13/22 cont'd The weather n days from now can be predicted by x(n) = x(0) n = 0 1 0 0.50 0.25 0.25 0.25 0.50 0.25 0.50 0.50 0.00 n and in the limit lim n x(n) = lim n x(0) n = lim n 0 1 0 0.50 0.25 0.25 0.25 0.50 0.25 0.50 0.50 0.00 n = 0.4 0.4 0.2 4310 : Intertemporal macroeconomics Preview Bellman equation and value iteration 14/22 A glimpse of the Bellman equation v(s) = max s r(s, s ) + Ev(s ) Today simpler v(s) = r(s) + v(s ) Next time leading up to v(k) = max k u(k, k ) + v(k ) 4310 : Intertemporal macroeconomics Lab session 15/22 Seminar sessions this week Repeat basic structures such as scalars, vectors, matrices and for and while loops Implement value function iteration. Use the random variable generator to generate random Markov chains. 4310 : Intertemporal macroeconomics Lab session 16/22 Employment transitions Student Entry level Manager Start-up Unemployed Executive Rich Dead .3 .1 .2 .1 .1 .2 .1 .9 .1 .2 .1 .1 .4 .6 .6 .7 .7 .9 .6 1.0 4310 : Intertemporal macroeconomics Lab session 17/22 Compensation at each realization Current realization Ref. Compensation (x) Student s1 NOK 150,000 Entry level position s2 NOK 300,000 Middle manager s3 NOK 450,000 Start-up company s4 NOK 200,000 Unemployed s5 NOK 150,000 Top-level executive s6 NOK 800,000 Successful entrepreneur s7 NOK 3,000,000 Dead s8 NOK 0 4310 : Intertemporal macroeconomics Lab session 18/22 Markov process We have One state variable (S) which can take eight distinct values/realizations, S = {s1, s2, s3, s4, s5, s6, s7, s8}. A transition probability matrix . A reward/compensation, x, associated with each realization of the state variable. A discount factor . 4310 : Intertemporal macroeconomics Lab session 19/22 Solving the Markov process v(si) = expected discounted sum of future rewards starting in realization si = r(xi) + (expected discounted sum of future rewards starting at next step) = r(xi) + j ijv(sj) 4310 : Intertemporal macroeconomics Lab session 20/22 Vector notation v(S) = v(s1) v(s2) v(s3) v(s4) v(s5) v(s6) v(s7) v(s8) , X = x1 x2 x3 x4 x5 x6 x7 x8 , = 11 12 18 21 ... ... ... ... ... ... ... ... 78 81 87 88 The Bellman equation in vector form v(S) = r(X) + v(S ), where indicates next period. 4310 : Intertemporal macroeconomics Lab session 21/22 Functional equation Notice that v(S) is a function that takes the realization of the state as argument and gives the value. The unknown here is the function. We know r(X), , and the set S, and we want to find the function v() such that v(S) = r(X) + v(S). 4310 : Intertemporal macroeconomics Lab session 22/22 Solution by value function iteration Approach: iterate backwards on the value function by proceeding through the following steps: Pick an initial value function v0(S), e.g. a vector of zeros. Iterative scheme, solving backwards vi+1(S) = r(X) + vi(S), Iterate until convergence, i.e. until vi+1(S) - vi(S) < , where is an arbitrarily small number.