Dynamic Programming For Data Independent Decision Sets

Multistage stochastic optimization problems are oftentimes formulated informally in a pathwise way. These formulations are appealing in a discrete setting and suitable when addressing computational challenges, for example. But the pathwise problem statement does not allow an analysis with mathematical rigor and is therefore not appropriate. R. T. Rockafellar and R. J.-B. Wets [Nonanticipativity and L-1-martingales in stochastic optimization problems, Math. Programming Study 6 (1976) 170-187] address the fundamental measurability concern of the value functions in the case of convex costs and constraints. This paper resumes these foundations. The contribution is a proof that there exist measurable versions of intermediate value functions, which reveals regularity in addition. Our proof builds on the Kolmogorov continuity theorem. It is demonstrated that verification theorems allow stating traditional problem specifications in the novel setting with mathematical rigor. Further, we provide dynamic equations for the general problem. The problem classes covered include Markov decision processes, reinforcement learning and stochastic dual dynamic programming.