Skip to main content
Log in

On confidence intervals from simulation of finite Markov chains

  • Published:
Mathematical Methods of Operations Research Aims and scope Submit manuscript

Abstract

Consider a finite state irreducible Markov reward chain. It is shown that there exist simulation estimates and confidence intervals for the expected first passage times and rewards as well as the expected average reward, with 100% coverage probability. The length of the confidence intervals converges to zero with probability one as the sample size increases; it also satisfies a large deviations property.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Burnetas AN, Katehakis MN (1996) Finding optimal policies for markovian decision processes using simulation. Prob. Eng. Info. Sci. 10:525–537

    Google Scholar 

  • Burnetas AN, Katehakis MN (1997) Optimal adaptive policies for markovian decision processes. Math. Oper. Res. 22(1): 222–255

    Google Scholar 

  • Dembo A, Zeitouni O (1993) Large deviations techniques and applications. Jones and Bartlett

  • Derman C (1970) Finite state Markovian decision processes. Academic Press

  • Federgruen A, Schweitzer P (1981) Nonstationary markov decision problems with converging parameters. J. Opt. Th. Appl. 34:207–241

    Google Scholar 

  • Fishman GS (1978) Principles of discrete event simulation. Wiley

  • Fishman GS (1994) Markov chain sampling and the product estiamtor. Mgt. Sci. 42:1137–1145

    Google Scholar 

  • Glasserman P, Liu T (1996) Rare-event simulation for multistage production-inventory systems. Mgt. Sci. 42(9): 1292–1307

    Google Scholar 

  • Hernández-Lerma O (1989) Adaptive Markov control processes. Springer-Verlag

  • Hordijk A (1974) Dynamic programming and Markov potential theory. Mathematisch Centrum, Amsterdam

  • Hordijk A, Iglehart DL, Schassberger R (1976) Discrete time methods for simulating continuous time markov chains. Adv. App. Prob. 8:772–788

    Google Scholar 

  • Katehakis MN, Robbins H (1995) Sequential allocation involving normal populations. Proc. Natl. Acad. Sci. USA:8584–8585

    Google Scholar 

  • Kleijnen JPC (1992) Simulation: A statistical perspective. Chichester

  • Van-Dijk N, Puterman ML (1988) Perturbation theory for Markov reward processes with applications to queueing systems. Adv. App. Prob. 20:79–98

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Rights and permissions

Reprints and permissions

About this article

Cite this article

Burnetas, A.N., Katehakis, M.N. On confidence intervals from simulation of finite Markov chains. Mathematical Methods of Operations Research 46, 241–250 (1997). https://doi.org/10.1007/BF01217693

Download citation

  • Received:

  • Revised:

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF01217693

Key words

Navigation