Symbolic algorithms for qualitative analysis of Markov decision processes with Büchi objectives

Abstract

We consider Markov decision processes (MDPs) with Büchi (liveness) objectives. We consider the problem of computing the set of almost-sure winning states from where the objective can be ensured with probability 1. Our contributions are as follows: First, we present the first subquadratic symbolic algorithm to compute the almost-sure winning set for MDPs with Büchi objectives; our algorithm takes $O(n \cdot\sqrt{m})$ symbolic steps as compared to the previous known algorithm that takes O(n ²) symbolic steps, where n is the number of states and m is the number of edges of the MDP. In practice MDPs have constant out-degree, and then our symbolic algorithm takes $O(n \cdot\sqrt{n})$ symbolic steps, as compared to the previous known O(n ²) symbolic steps algorithm. Second, we present a new algorithm, namely win-lose algorithm, with the following two properties: (a) the algorithm iteratively computes subsets of the almost-sure winning set and its complement, as compared to all previous algorithms that discover the almost-sure winning set upon termination; and (b) requires $O(n \cdot\sqrt{K})$ symbolic steps, where K is the maximal number of edges of strongly connected components (scc’s) of the MDP. The win-lose algorithm requires symbolic computation of scc’s. Third, we improve the algorithm for symbolic scc computation; the previous known algorithm takes linear symbolic steps, and our new algorithm improves the constants associated with the linear number of steps. In the worst case the previous known algorithm takes 5⋅n symbolic steps, whereas our new algorithm takes 4⋅n symbolic steps.

Appendix

1.1 8.1 Technical details of improved symbolic scc algorithm

The pseudocode of SCCFind is formally given as Algorithm 4. The correctness analysis and the analysis of the number of symbolic steps is given in [12]. The pseudocode of ImprovedSCCFind is formally given as Algorithm 5. The main changes of ImprovedSCCFind from SCCFind are as follows: (1) instead of SkelFwd the algorithm ImprovedSCCFind calls procedure ImprovedSkelFwd that returns an additional set P and ImprovedSkelFwd is invoked with an additional argument that is U; (2) in line 4 of ImprovedSCCFind the set SCC is initialized to P instead of s. The main difference of ImprovedSkelFwd from SkelFwd is as follows: (1) the set P is computed in line 4 of ImprovedSkelFwd as FWSet∩Q, where Q is the set passed by ImprovedSCCFind as the argument; and (2) in the while loop it is checked if the element popped intersects with P and if yes, then the procedure breaks the while loop. The correctness argument from the correctness of SCCFind is already shown in Sect. 5.1.

Algorithm 4

SCCFind

Algorithm 5

ImprovedSCCFind

Symbolic steps analysis

We now present the detailed symbolic steps analysis of the algorithm. As noted in Sect. 3.2, common symbolic operations on a set of states are Pre, Post and CPre. We note that these operations involve symbolic sets of 2⋅log(n) variables, as compared to symbolic sets of log(n) variables required for operations such as union, intersection and set difference. Thus only Pre, Post and CPre are counted as symbolic steps, as done in [12]. The total number of other symbolic operations is also O(|S|). We note that only lines 5 and 10 of ImprovedSCCFind and lines 3.3 and 7.3 of ImprovedSkelFwd involve Pre and Post operations.

In the following, we charge the costs of these lines to states in order to achieve the 3⋅|S|+N(G) bound for symbolic steps. We define subspine-set as NewSet returned by ImprovedSkelFwd and show the following result.

Lemma 4

For any spine-set U and its end vertex u, T is a subspine-set iff U∖T⊆SCC(u).

Proof

Note that while constructing a subspine-set T, we stop the construction when we find any state v∈FWSet∩U from the spine set. Now clearly since v∈U, there is a path from v to u. Also, since we found this state in FW(u), there is a path from u to v. Hence, v∈SCC(u). Also, each state that we are omitting by stopping construction of T has the property that there is a path from u to that state and a path from that state to v. This implies that all the states we are omitting in construction of T are in SCC(u). □

Note that since we pass NewSet∖SCC in the subsequent call to ImprovedSCCFind, it will actually be a spine set for the reduced problem. In the following lemma we show that any state can be part of subspine-set at most once, as compared to twice in the SCCFind procedure in [12]. This lemma is one of the key points that lead to the improved analysis of symbolic steps required.

Lemma 5

Any state v can be part of subspine-set at most once.

Proof

In [12], the authors show that any state v can be included in spine sets at most twice in SkelFwd. The second time the state v is included is in line 6 of SkelFwd when the SCC(v) of the state is to be found. In contrast, ImprovedSkelFwd checks intersection of the subspine-set being constructed with the set P that contains the states of SCC(v) which are already in a subspine-set. When this happens, it stops the construction of the subspine-set. Now if v is already included in the subspine-set, then it will be part of P and would not be included in subspine-set again. Hence, v can be part of subspine-set at most once. □

Lemma 6

States added in SCC by iteration of line 5 of ImprovedSCCFind are exactly the states which are not part of any subspine-set.

Proof

We see that in line 5 of ImprovedSCCFind, we start from SCC=P and then we find the SCC by backward search. Also, P has all the states from SCC which are part of any subspine-set. Hence, the extra states that are added in SCC are states which are never included in a subspine-set. □

Charging symbolic steps to states

We now consider three cases to charge symbolic steps to states and scc’s.

• 1. Charging states included in subspine-set. First, we see that the number of times the loop of line 3 in ImprovedSkelFwd is executed is equal to the size of the spine set that SkelFwd would have computed. Using Lemma 4, we can charge one symbolic step to each state of the subspine-set and each state of the SCC that is being computed. Now, the number of times line 7.3 of ImprovedSkelFwd is executed equals the size of subspine-set that is computed. Hence, we charge one symbolic step to each state of subspine-set for this line.

  Now we summarize the symbolic steps charged to each state which is part of some subspine-set. First time when a state gets into a subspine-set, it is charged two steps, one for line 3.3 and one for line 7.3 of ImprovedSkelFwd. If its SCC is not found in the same call to ImprovedSCCFind, then it comes into action once again when its SCC is being found. By Lemma 5, it is never again included in a subspine set. Hence in this call to ImprovedSkelFwd, it is only charged one symbolic step for line 3.3 and none for line 7.3 as line 7.3 is charged to states that become part of the newly constructed subspine-set. Also because of Lemma 6, since this state is in a subspine-set, it is not charged anything for line 5 of ImprovedSCCFind. Hence, a state that occurs in any subspine-set is charged at most three symbolic steps.

• 2. Charging states not included in subspine-set. For line 5 of ImprovedSCCFind, the number of times it is executed is the number of states that are added to SCC after initialization to SCC=P. Using Lemma 6, we charge one symbolic step to each state of this SCC that is never a part of any subspine-set. Also, we might have charged one symbolic step to such a state for line 3.3 of ImprovedSkelFwd when we called it. Hence, each such state is charged at most two symbolic steps.

• 3. Charging SCC s. For line 10 of ImprovedSCCFind, we see that it is executed only once in a call to ImprovedSCCFind that computes a SCC. Hence, the total number of times line 10 is executed equals N(G), the number of SCCs of the graph. Hence, we charge each SCC one symbolic step for this line.

The above argument shows that the number of symbolic steps that the algorithm ImprovedSCCFind requires is at most 3⋅|S|+N(G). This completes the formal proof of Theorem 8.

We now present an example that presents a family of graphs with k⋅n states, where the SCCFind algorithm takes almost 5⋅k⋅n symbolic steps, whereas the ImprovedSCCFind algorithm takes at most 3⋅k⋅n+n symbolic steps.

Example 1

Let k,n∈ℕ. Consider a graph with k⋅n states such that the states are numbered from 1 to k⋅n. The edges are as follows: (1) for all states 1≤i≤k⋅n−1, there is an edge (i,i+1) (i.e., the states are all in a line); and (2) for all 1≤i≤n, there is an edge from state k⋅i to state (i−1)⋅k+1. We will show that the SCCFind algorithm requires roughly (5⋅k−1)⋅n symbolic steps on this graph. Note the number of scc’s in this graph is n, and hence by Theorem 8 the ImprovedSCCFind algorithm takes at most 3⋅k⋅n+n symbolic steps.

We now analyze the symbolic steps required by the SCCFind algorithm, and our analysis is in two steps.

• 1. Step 1. In the beginning, starting from the state 1 of the graph, the algorithm performs a forward and backward search to find the spine set. This will have a cost of two symbolic steps per state (one symbolic step while going forward, and one symbolic step while going backward), except for the first vertex which gets charged only one symbolic step (it does not get charged while going backwards). This gives a cost of 2⋅k⋅n−1 symbolic steps. After this, the first scc will be found with an additional cost of only k, and the first discovered scc consists of states {1,2,…,k}. Hence the total symbolic steps required is 2⋅k⋅n−1+k.

• 2. Step 2. After Step 1, the algorithm will start finding scc’s from the end of the spine set. So consider the set of last k states from k⋅(n−1)+1 to n⋅k. The algorithm will pick the last state n⋅k, find a spine set, consisting of the last k states (states k⋅(n−1)+1 to n⋅k). This will have a cost of 2⋅k−1 symbolic steps (k symbolic steps for the forward search, and k−1 symbolic steps for the backward search). After this, the algorithm will find the scc containing the last state n⋅k (i.e., the scc that consists of states {k⋅(n−1)+1,k⋅(n−1)+2,…,n⋅k}), and this takes k symbolic steps as there are k vertices in the scc. Now Step 2 is repeated with state k⋅(n−1), and then repeated for state k⋅(n−2) and so on. So the cost for every scc, except the very first one, is 2k−1+k=3k−1. Since there are n scc’s, the total number of symbolic steps required for Step 2 is at least (3⋅k−1)⋅(n−1).

Hence the total symbolic steps required for the algorithm is at least

$$2\cdot k\cdot n - 1 + k + (3\cdot k -1)\cdot(n-1)= (5\cdot k-1)\cdot n -2\cdot k. $$

Note that with k=n, SCCFind takes at least 5⋅n ²−3⋅n symbolic steps, whereas the ImprovedSCCFind takes at most 3⋅n ²+n symbolic steps.