Abstract
This paper concerns nonstationary continuous-time Markov control processes on Polish spaces, with the infinite-horizon discounted cost criterion. Necessary and sufficient conditions are given for a control policy to be optimal and asymptotically optimal. In addition, under suitable hypotheses, it is shown that the successive approximation procedure converges in the sense that the sequence of finite-horizon optimal cost functions and the corresponding optimal control policies both converge.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bertsekas, D. P.: Dynamic Programming: Deterministic and Stochastic Models, Prentice-Hall, Englewood Cliffs, NJ, 1987.
Bielecki, T. R.: Adaptive control of continuous-time linear stochastic systems with discounted cost criterion, J. Optim. Theory Appl. 68 (1991), 379-383.
Borkar, V. S.: Optimal Control of Diffusion Processes, Pitman Res. Notes in Math. 203, Longman, Harlow, 1989.
Davis, M. H. A.: Markov Models and Optimization, Chapman & Hall, London, 1993.
Doshi, B. T.: Continuous time control of Markov processes on an arbitrary state space: Discounted rewards, Ann. Statist. 4 (1976), 1219-1235.
Dynkin, E. B.: Markov Processes I, Springer, Berlin, 1965.
Easley, D. and Spulber, D. F.: Stochastic equilibrium and optimality of rolling plans, Internat. Econom. Rev. 22 (1981), 79-103.
Fleming, W. H. and Rishel, R.: Deterministic and Stochastic Optimal Control, Springer, New York, 1975.
Fleming, W. H. and Soner, H. M.: Controlled Markov Processes and Viscosity Solutions, Springer, New York, 1993.
Gihman, I. I. and Skorohod, A. V.: Controlled Stochastic Processes, Springer, Berlin, 1979.
Guo, X. P. and Zhu, W. P.: Denumerable state continuous time Markov decision processes with unbounded reward and transition rates under discounted criterion, J. Australian Math. Soc., to appear.
Hernández-Lerma, O.: Adaptive Markov Control Processes, Springer, New York, 1989.
Hernández-Lerma, O.: Lectures on Continuous-Time Markov Control Processes, Aportaciones Matemáticas, Textos Nivel Avanzado 3, Sociedad Matemática Mexicana, 1994.
Hernández-Lerma, O. and Lasserre, J. B.: Value iteration and rolling plans for Markov control rocesses with unbounded rewards, J. Math. Anal. Appl. 177 (1993), 38-55.
Hernández-Lerma, O. and Lasserre, J. B.: Discrete-Time Markov Control Processes: Basic Optimality Criteria, Springer, New York, 1996
Hernández-Lerma, O. and Lasserre, J. B.: Further Topics on Discrete-Time Markov Control Processes, Springer, New York, 1999.
Hernández-Lerma, O. and Vega-Amaya, O.: Infinite-horizon Markov control processes with undiscounted cost criteria: From average to overtaking optimality, Appl. Math. (Warsaw) 25(1998), 153-178.
Hinderer, K.: Foundations of Non-Stationay Dynamic Programming with Discrete Time Parameter, Lecture Notes in Oper. Res. 33, Springer, Berlin, 1970.
Krylov, N. V.: Controlled Diffusion Processes, Springer, New York, 1980.
Leizarowitz, A.: Controlled diffusion processes on infinite horizon with the overtaking criterion, Appl. Math. Optim. 17 (1988), 61-78.
Pliska, S. R.: Controlled jump processes, Stochastic Process. Appl. 3 (1975), 259-282.
Rishel, R.: Controlled continuous time Markov processes, In: D. P. Heyman and M. J. Sobel (eds), Handbooks in OR and MS 2, Elsevier (North-Holland), 1990, pp. 435-467.
Schäl, M.: Conditions for optimality and for the limit of n-stage optimal policies to be optimal, Z. Wahrs. verw. Geb. 32 (1975), 179-196.
Schäl, M.: Estimation and control in discounted stochastic dynamic programming, Stochastics 20 (1987), 51-71.
Taksar, M. I.: Infinite-dimensional linear programming approach to singular stochastic control, SIAM J. Control Optim. 35 (1997), 604-625.
Tan, H. and Rugh, W. J.: Nonlinear overtaking optimal control: Sufficiency, stability, and approximations, IEEE Trans. Autom. Control 43 (1998), 1703-1718.
Wentzell, A. D.: A Course in the Theory of Stochastic Processes, McGraw-Hill, New York, 1981.
Widder, D. V.: The Laplace Transform, Princeton Univ. Press, Princeton, NJ, 1941.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Hernández-Lerma, O., Govindan, T.E. Nonstationary Continuous-Time Markov Control Processes with Discounted Costs on Infinite Horizon. Acta Applicandae Mathematicae 67, 277–293 (2001). https://doi.org/10.1023/A:1011970418845
Issue Date:
DOI: https://doi.org/10.1023/A:1011970418845