Planning using hierarchical constrained Markov decision processes

Abstract

Constrained Markov decision processes offer a principled method to determine policies for sequential stochastic decision problems where multiple costs are concurrently considered. Although they could be very valuable in numerous robotic applications, to date their use has been quite limited. Among the reasons for their limited adoption is their computational complexity, since policy computation requires the solution of constrained linear programs with an extremely large number of variables. To overcome this limitation, we propose a hierarchical method to solve large problem instances. States are clustered into macro states and the parameters defining the dynamic behavior and the costs of the clustered model are determined using a Monte Carlo approach. We show that the algorithm we propose to create clustered states maintains valuable properties of the original model, like the existence of a solution for the problem. Our algorithm is validated in various planning problems in simulation and on a mobile robot platform, and we experimentally show that the clustered approach significantly outperforms the non-hierarchical solution while experiencing only moderate losses in terms of objective functions.

Appendix

Proof of Theorem 1

Definition 2 establishes two conditions for saying that an HCMDP preserves connectivity. The first requires that $\mathcal {X}_H$ is a partition of $\mathcal {X}$. Algorithm 1 never considers a state twice, i.e., once a state has been assigned to a cluster it will not be considered again for assignment (line 4). Moreover, the main loop ensures that all states in $\mathcal {X}$ are assigned to a cluster. Therefore, $\mathcal {X}_H$ is a partition of $\mathcal {X}$.

We next turn to the second condition. Let $z \in \mathcal {X}'$ and $y\in M$ be two states such that $z\leadsto y$. By definition this means that there exists a sequence of states $\mathcal {S}=s_1,s_2,\dots ,s_n$ such that for each $1\le i\le n-1$ $P_{s_i,s_{i+1}}^{u_i}$ for some $u_i\in U(s_i)$ and $s_1=y$ and $s_n = z$. Since $\mathcal {X}_H$ is a partition of $\mathcal {X}$, this sequence of states is associated with a sequence of macrostates $Z_H=S_1\dots S_n=Y_H$ such that $s_i \in S_i$ for each i. Note that in general there could be some repeated elements in the sequence of macrostates. Let $S_1,\dots S_k$ $(k\le n)$ be the sequence obtained removing subsequences of repeated macrostates.⁵ First note that this sequence includes at least two elements. This is true because we started assuming $y\notin M$ while $z\in M$. According to Algorithm 1 all and only the states in M are mapped to an individual macrostate (line 1), so y cannot be in the same macrostate as z. Next, consider two successive elements in the sequence of macrostates, say $S_i$ and $S_{i+1}$. By construction, there exist two successive states in $\mathcal {S}$, say $s_j$ and $s_{j+1}$, such that $s_j \in S_i$ and $s_{j+1} \in S_{i+1}$. Since these two states are part of $\mathcal {S}$, there exists one input $u_j\in U(s_j)$ such that $P_{s_j,s_{j+1}}^{u_j}>0$. As per Eq. (7), this implies that an action $S_{j+1}$ is added to the set of actions $U(S_j)$. Next, consider the method described in Sect. 4.3, and in particular the definition of the boundary B between two macro states. It follows that $s_{j+1} \in B_{S_i,S_{i+1}}$. The algorithm further continues computing the shortest path between each state in $S_i$ and B, where the shortest path is computed over the induced graph G. For $s_i$ the path trivially consists of a single edge to $s_{i+1}$ (or some other vertex in B that is also one hop away from $s_i$.) Next, the algorithm randomly selects one vertex from $S_j$ using a uniform distribution and executes the policy to reach B. Let m be the total number of Monte Carlo samples generated. Then the probability that the estimate of $P_{S_i,S_i+1}^{S_{i+1}}=0$ is bounded from above by

$$\begin{aligned} (1-\gamma )^{k_1}\left( 1-P_{s_j,s_{j+1}}^{u_j}\right) ^{k_2} \end{aligned}$$

where $\gamma = \frac{1}{S_i}$, $k_1$ is the number of times $s_j$ was not sampled and $k_2$ is the number of times $s_j$ was sampled ($k_1+k_2 =m, k_{1,2}\ge 0$). This proves that as the total number of samples m grows, the estimate for $P_{S_i,S_i+1}^{S_{i+1}}$ will be eventually be positive. This reasoning can be repeated for each couple of successive macro states, thus showing that $Z_H\leadsto Y_H$, and this concludes the proof. $\square$

Proof of Theorem 2

We start observing that Algorithm 3 builds and solves a sequence of HCMDPs. Each is a CMDP with a suitable set of parameters and at every iteration the constrained linear program given in Eq. (5) is solved. Theorem 1 guarantees that state $M_H$ is accessible from every macrostate, and therefore there exists at least one policy $\pi '$ for which $c(\pi ')$ is finite. Let us next consider the inequality constraints in Eq. (5). If the linear program is not feasible, then each bound $D_{i,H}$ is increased by $\varDelta D_{i,H}$ (line 6.) By construction, all costs $d_{i,H}(x,u)\ge 0$ for each state/action pair (x, u). Let $n_s = |\mathcal {K}_H'|$ be the number of state/action pairs in the HCMDP, $d_{max} = \max _{(x,u)\in K_H'} \{d_{i,H}(x,u)\}$ the largest among the additional costs, and $D_{min} = \min \{\varDelta D_{i,H}\}$ the smallest among the increments in line 6. Therefore after at most $\lceil \frac{n_sd_{max}}{D_{min}}\rceil$ iterations all inequality constraints become feasible. $\square$