Horizontal combinations of online and offline approximate dynamic programming for stochastic dynamic vehicle routing

Abstract

Stochastic and dynamic vehicle routing problems gain increasing attention in the research community. In these problems, routing plans are dynamically updated based on realizations of stochastic information. Due to the complexity of the corresponding Markov decision processes (MDPs), the calculation of optimal policies for these problems is usually not possible and researchers draw on heuristical methods of approximate dynamic programming (ADP). These methods use simulation to approximate the value of a state and decision in the MDP. The simulations are either conducted offline or online. Offline methods such as value function approximations (VFAs) generally neglect the full detail of the state space due to aggregation. Online methods such as rollout algorithms (RAs) are often not able to capture decision and transition space sufficiently due to runtime limitations. In this paper, we alleviate this tradeoff by combining two methods of ADP, an online RA and an offline VFA in two ways. In addition to the integration of the VFA as a base policy into the online RA to strengthen the RA’s simulations, we also limit the RA’s simulation horizon, estimating the remaining reward-to-go again via the VFA. For two stochastic dynamic routing problems from the literature, we show how this combination outperforms state-of-the-art solutions while simultaneously reducing the required time for online calculations.

Appendix

In the Appendix, we present describe the routing heuristic and the detailed algorithm for the V-LHRA. We further describe how the customer distributions are generated and finally provide the results of the tuning and runtimes in detail.

1.1 Routing heuristic

The routing is conducted with a cheapest insertion procedure (CI). The tour is initialized and updated regarding the following procedure: In $\delta =1$ and $t=0$, CI starts with a pre-decision tour $\theta _0=(D,D)$ only consisting of the depot. In every decision point k, a pre-decision tour $\theta ^{\delta }_k$ and a set of candidate subsets $\hat{{\mathcal {C}}}^a_k \subseteq {\mathcal {C}}^r_k, i=1,\dots ,2^{|{\mathcal {C}}^r_k|}$, i.e., the power set of new requests is given. For every candidate subset, CI subsequently selects the cheapest request of the subset regarding the insertion time and adds it at the cheapest insertion position in the tour. The same procedure is applied for $\theta ^{\delta +1}_k$ with the postponed requests ${\mathcal {C}}^r_k\backslash \hat{{\mathcal {C}}}^a_k$. The routing is induced by the accepted subset ${{\mathcal {C}}}^a_k$ and the according post-decision tour $\theta ^{\delta ,x}_k$. If the selected routing decision of x is not waiting, the first customer in the tour $C_{\text {next}}=\theta _1$ is the next customer to visit. After the travel to the next customer, the visited customer is removed from $\theta ^{\delta ,x}_k$ resulting in pre-decision tour $\theta ^{\delta }_{k+1}$. Waiting is applied if no customers are left to serve, i.e., $\theta ^{\delta }_k=({\mathcal {P}}_k,{\mathcal {D}})$, but free time budget is left. In this case, the pre-decision tour $\theta {\delta }_{k+1}$ is identical to $\theta ^{\delta ,x}_k$. If no time is left, the vehicle returns to the depot. A candidate subset and the according decision are considered feasible if the resulting candidate post-decision tour does not exceed the time limit. Because the decision space is reduced, CI may reject candidate subsets, which are feasible regarding a different routing approach, e.g., an optimal traveling salesman solution.

1.2 V-LHRA algorithm

Algorithm 1 describes the procedure of decision making for a VFA-based limited horizon post-decision RA. Let a state $S_k$, decisions $x_1,\dots ,x_n$, the reward-function R, the PDSs ${\mathcal {P}}_k=(S_k^{x_1},\dots , S_k^{x_n})$, a set of sampled realizations $\{\omega _1,\dots ,\omega _m\}$, the VFA base policy $\pi _{\text {VFA}}$, and the VFA-values $\tilde{V}$ be given. Then, for every PDS $S_k^x$, the RA simulates the next h decision points of every realization $\omega _i$. In the decision points, the decision $X^{\pi _{\text {VFA}}}(S_{k+j})$ induced by the VFA base policy $\pi _{\text {VFA}}$ is applied and the rewards $R(S_{k+j},X^{\pi _{\text {VFA}}}(S_{k+j}))$ are accumulated. After the h decision points, the value $\tilde{V}(S^x_{k+h})$ is added to the accumulated rewards. The overall reward-to-go of a PDS $\hat{V}(S_k^{x})$ is the average of the single rewards per realization. The RA selects the decision $x^*$ leading to the maximum sum of immediate reward $R(S_k,x^*)$ and expected future rewards $\hat{V}(S_k^{x^*})$.

1.3 Customer distributions

In this section, we describe the spatial customer distributions for small service area size. For the large service area, the quantities are multiplied by 1.5. Given U, a spatial realization (x, y) is defined as $(x,y) \sim U[0,10] \times U[0,10]$. For 2C, the customers are equally distributed to each cluster. The cluster centers are located at $\mu _1=(2.5,2.5), \mu _2=(7.5,7.5)$. The standard deviation within the clusters is $\sigma =0.5$. For 3C, the cluster centers are located at $\mu _1=(2.5,2.5), \mu _2=(2.5,7.5), \mu _3=(7.5,5)$. $50\%$ of the requests are assigned to cluster two, $25\%$ to each other cluster. The standard deviations are set to $\sigma =0.5$.

1.4 Results

In this section, we present the best simulation horizons and the maximal runtimes per decision point per instance setting.

See Tables 4, 5, 6, 7 and 8.

Table 4 Results: improvement compared to Myopic (in %)

Table 5 Results: best horizon, $\delta _{\text {max}}=1$

Table 6 Results: best horizon, $\delta _{\text {max}}=3$

Table 7 Results: maximal runtime per decision point (in s), $\delta _{\text {max}}=1$

Table 8 Results: maximal runtime per decision point (in s), $\delta _{\text {max}}=3$