Abstract
Consider a finite state irreducible Markov reward chain. It is shown that there exist simulation estimates and confidence intervals for the expected first passage times and rewards as well as the expected average reward, with 100% coverage probability. The length of the confidence intervals converges to zero with probability one as the sample size increases; it also satisfies a large deviations property.
Similar content being viewed by others
References
Burnetas AN, Katehakis MN (1996) Finding optimal policies for markovian decision processes using simulation. Prob. Eng. Info. Sci. 10:525–537
Burnetas AN, Katehakis MN (1997) Optimal adaptive policies for markovian decision processes. Math. Oper. Res. 22(1): 222–255
Dembo A, Zeitouni O (1993) Large deviations techniques and applications. Jones and Bartlett
Derman C (1970) Finite state Markovian decision processes. Academic Press
Federgruen A, Schweitzer P (1981) Nonstationary markov decision problems with converging parameters. J. Opt. Th. Appl. 34:207–241
Fishman GS (1978) Principles of discrete event simulation. Wiley
Fishman GS (1994) Markov chain sampling and the product estiamtor. Mgt. Sci. 42:1137–1145
Glasserman P, Liu T (1996) Rare-event simulation for multistage production-inventory systems. Mgt. Sci. 42(9): 1292–1307
Hernández-Lerma O (1989) Adaptive Markov control processes. Springer-Verlag
Hordijk A (1974) Dynamic programming and Markov potential theory. Mathematisch Centrum, Amsterdam
Hordijk A, Iglehart DL, Schassberger R (1976) Discrete time methods for simulating continuous time markov chains. Adv. App. Prob. 8:772–788
Katehakis MN, Robbins H (1995) Sequential allocation involving normal populations. Proc. Natl. Acad. Sci. USA:8584–8585
Kleijnen JPC (1992) Simulation: A statistical perspective. Chichester
Van-Dijk N, Puterman ML (1988) Perturbation theory for Markov reward processes with applications to queueing systems. Adv. App. Prob. 20:79–98
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Burnetas, A.N., Katehakis, M.N. On confidence intervals from simulation of finite Markov chains. Mathematical Methods of Operations Research 46, 241–250 (1997). https://doi.org/10.1007/BF01217693
Received:
Revised:
Issue Date:
DOI: https://doi.org/10.1007/BF01217693