Skip to main content
SpringerLink
Log in

Optimality Conditions for CTMDP with Average Cost Criterion

  • Chapter
Markov Processes and Controlled Markov Chains

Abstract

In this paper, we consider continuous time Markov decision processes with (possibly unbounded) transition and cost rates under the average cost criterion. We present a set of conditions that is weaker than those in [5, 11, 12, 14], and prove the existence of optimal stationary policies using the optimality inequality. Moreover, the theory is illustrated by two examples.

This research has been supported partially by Natural Science Foundation of Guangdong Province, by Foundation of Hongkong Zhongshan University Advanced Research Center, and by the University of Queensland under Grant No. 98/UQNSRG025G.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Bather, J. Optimal stationary policies for denumerable Markov chains in continuous time. Adv, Appl. Prob., 8 (1976), 144–158.

    Article  MathSciNet  MATH  Google Scholar 

  2. Bertsekas, D. P. Dynamic Programming: Deterministic and Stochastic Models. Prentice-Hall, Englewood Cliffs, NJ, 1987.

    MATH  Google Scholar 

  3. Doshi, B. T. Continuous time control of Markov processes on an arbitrary state space: average return criterion. Stochastic Processes and their Applications, 4 (1976), 55–77

    Article  MathSciNet  MATH  Google Scholar 

  4. Howard, R. A. Dynamic Programming and Markov Processes. Wiley, New York, 1960.

    MATH  Google Scholar 

  5. Kakumanu, P. Nondiscounted continuous time Markov decision processes with countable state space. SIAM J. Control, 10 (1972), 210–220.

    Article  MathSciNet  MATH  Google Scholar 

  6. Lippman, S. A. Applying a new device in the optimization of exponential queueing systems. Op. Res. 23 (1975b), 687–710.

    Article  MathSciNet  MATH  Google Scholar 

  7. Miller, R. L. Finite state continuous time Markov decision processes with an infinite planning horizon. J. Math. Anal. Appl., 22 (1968), 552–569.

    Article  MathSciNet  MATH  Google Scholar 

  8. Puterman, M. L. Markov Decision Processes. John Wiley & Sons Inc,1994.

    Google Scholar 

  9. Yushkevich, A. A. and Feinberg, E. A. On homogeneous Markov model with continuous time and finite or countable state space. Theory. Prob. Appl., 24 (1979), 156–161.

    Article  MATH  Google Scholar 

  10. Walrand, J. An Introduction to Queuing Networks. Prentice-Hall, Englewood Cliffs, NJ, 1988.

    Google Scholar 

  11. Dong, Z. Q. Continuous time Markov decision programming with average reward criterion-countable state and action space. Sci. Chi.. SP ISS(II)(1979), 131–148.

    Google Scholar 

  12. Song, J. S. Continuous time Markov decision programming with non-uniformly bounded transition rate. Scientia Sinica, 12 (1987), 1258–1267.

    Google Scholar 

  13. Zheng, S. H. Continuous time Markov decision programming with average reward criterion and unbounded reward rate. Acta Math. Appl. Sinica, 7 (1991), 6–16.

    Article  MATH  Google Scholar 

  14. Guo, X. P. and Liu K. “Optimality inequality for continuous time Markov decision processes with average reward criterion”. Preprint, Zhongshan University, Guangzhou, P.R. China, 1998(Submitted).

    Google Scholar 

  15. Sennott, L. I. Another set of conditions for average optimality in Markov decision processes. System & control letters, 24 (1995), 147–151.

    Article  MathSciNet  MATH  Google Scholar 

  16. Sennott, L. I. Average cost optimal stationary policies in infinite state Markov decision processes with unbounded cost. Op. Res., 37 (1989), 626–633.

    Article  MathSciNet  MATH  Google Scholar 

  17. Cavazos-Cadena, R. and Gaucherand, Value iteration in a class of average controlled Markov chains with unbounded costs: Necessary and sufficient conditions for pointwise convergence, J. Appl. Prob., 33 (1996), 986–1002.

    Article  MATH  Google Scholar 

  18. Guo, X. P. and Zhu, W. P. “Denumerable state continuous time Markov decision processes with unbounded cost and transition rates under discounted cost criterion”, To appear on Journal of Australian Math. Soci. series B.

    Google Scholar 

  19. Wu, C. B. Continuous time Markov decision processes with unbounded reward and non-uniformly bounded transition rate under discounted criterion. Acta Math. Appl. Sinica, 20 (1997), 196–208.

    MathSciNet  MATH  Google Scholar 

  20. Filar, J. A. and Vrieze, K. Competitive Markov Decision Processes. Springer-Verlag, New York, 1996.

    Book  Google Scholar 

  21. Chung, K. L. Markov Chains with Stationary Transition Probabilities. Springer-Verlag, Berlin, 1960.

    MATH  Google Scholar 

  22. Haviv, M. and Puterman, M. L. Bias optimality in controlled queueing systems. J. Appl. Prob.., 35 (1998), 136–150.

    Article  MathSciNet  MATH  Google Scholar 

  23. Serfozo, R. Optimal control of random walks, birth and death processes, and queues. Adv. Appl. Prob.., 13 (1981), 61–83.

    Article  MathSciNet  MATH  Google Scholar 

  24. Anderson, W. J. Continuous Time Markov Chains. Springer-Verlag, New York, 1991.

    Book  MATH  Google Scholar 

  25. Widder, D.V. The Laplace Transform. Princeton University Press. Princeton, NJ, 1946.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics, Zhongshan University, China

    Xianping Guo

  2. Department of Computer Science and Electrical Engineering, The University of Queensland, Australia

    Weiping Zhu

Authors
  1. Xianping Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Weiping Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Research Department, Changsha Railway University, Changsha, China

    Zhenting Hou

  2. School of Mathematics, University of South Australia, Mawson Lakes, SA, Australia

    Jerzy A. Filar

  3. School of Computing and Mathematical Sciences, University of Greenwich, London, UK

    Anyue Chen

Rights and permissions

Reprints and permissions

Copyright information

© 2002 Kluwer Academic Publishers

About this chapter

Cite this chapter

Guo, X., Zhu, W. (2002). Optimality Conditions for CTMDP with Average Cost Criterion. In: Hou, Z., Filar, J.A., Chen, A. (eds) Markov Processes and Controlled Markov Chains. Springer, Boston, MA. https://doi.org/10.1007/978-1-4613-0265-0_10

Download citation

  • DOI: https://doi.org/10.1007/978-1-4613-0265-0_10

  • Publisher Name: Springer, Boston, MA

  • Print ISBN: 978-1-4613-7968-3

  • Online ISBN: 978-1-4613-0265-0

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation