Hierarchical Trajectory Planning for Narrow-Space Automated Parking with Deep Reinforcement Learning: A Federated Learning Scheme

Abstract

Collision-free trajectory planning in narrow spaces has become one of the most challenging tasks in automated parking scenarios. Previous optimization-based approaches can generate accurate parking trajectories, but these methods cannot compute feasible solutions with extremely complex constraints in a limited time. Recent research uses neural-network-based approaches that can generate time-optimized parking trajectories in linear time. However, the generalization of these neural network models in different parking scenarios has not been considered thoroughly and the risk of privacy compromise exists in the case of centralized training. To address the above issues, this paper proposes a hierarchical trajectory planning method with deep reinforcement learning in the federated learning scheme (HALOES) to rapidly and accurately generate collision-free automated parking trajectories in multiple narrow spaces. HALOES is a federated learning based hierarchical trajectory planning method to fully exert high-level deep reinforcement learning and the low-level optimization-based approach. HALOES further fuse the deep reinforcement learning model parameters to improve the generalization capabilities with a decentralized training scheme. The federated learning scheme in HALOES aims to protect the privacy of the vehicle’s data during model parameter aggregation. Simulation results show that the proposed method can achieve efficient automatic parking in multiple narrow spaces, improve planning time from $12.15\%$ to $66.02\%$ compared to other state-of-the-art methods (e.g., hybrid A*, OBCA) and maintain the same level of trajectory accuracy while having great model generalization.

1. Introduction

Automated parking is a hot issue of interest in academia and industry. Autonomous vehicles (AVs) can improve driving safety, efficiency and convenience through Advanced Driving Assistance Systems (ADAS) [1]. Automated parking is an essential application of ADAS for autonomous vehicles, which has been used by many car manufacturers such as Audi, BMW, Mercedes-Benz and BYD [2]. In recent years, parking space has become scarce in many cities with the increase in vehicles and the relative lag in infrastructure. The narrow and crowded parking environment increases the risk of collision and makes parking more difficult for drivers while also bringing new challenges to automated parking technology [3]. This paper aims to achieve a more efficient collision-free trajectory planning scheme for automated parking in narrow parking spaces (Figure 1).

Trajectory planning is a critical component of autonomous vehicles, enabling the generation of a collision-free, smooth and kinematically feasible curve for the vehicle’s motion. Compared to the trajectory planning of on-road autonomous driving, the trajectory planning of automated parking has more challenges. To be specific: (1) Polynomial-based path planning approaches are not suitable for automated parking because parked path curves can have non-guidable points, whereas polynomial-based methods can only generate smooth, reachable path curves. (2) Obstacle avoidance constraints in automated parking are more complex than those encountered in on-road autonomous driving due to the intricate parking environment. (3) Automated parking requires full use of the vehicle’s steering capability compared to the relatively smooth curve of on-road autonomous driving [4]. Currently, research mainly has three types of trajectory planning for automated parking: the sample-and-search-based method, the optimization-based method and the neural-network-based method.

Sample-and-search-based path planning discretizes the continuous state space into a graph with nodes and searches the graph structure for the optimal path linking the start and goal points. The typical methods of node-based search with sampling methods are RRT and RRT*, which can relatively quickly generate a curve to the goal point. However, these methods have no guarantee of kinetic feasibility, resulting in no reliable trajectories [5]. D. Dolgov et al. proposed the Hybrid A* algorithm, which discretizes and samples the control space to obtain a smooth and kinetic feasible trajectory curve [6]. He et al. proposed a fast A* anchor point-based path planning algorithm based on Hybrid A* for solving reverse parking path planning in narrow spaces [7]. The sample-and-search-based method can be applied to a wide range of scenarios. However, balancing computational accuracy and time is a challenge because low sampling resolution can lead to inaccurate path planning, while high sampling resolution can consume too many computational resources.

The optimization-based trajectory planning algorithms such as Model Predictive Control (MPC) have been applied to unmanned aircraft and autonomous vehicles. In general, the trajectory planning obstacle avoidance problem is NP-hard and the optimization-based methods consider the obstacle avoidance problem as an Optimal Control Problem (OCP), which is solved using the technique of Nonlinear Programming (NP) [8]. This category has two branches, that is, soft constraint-based and hard constraint-based optimizations. On soft constraint-based methods, K Shibata et al. designed an Artificial Potential Field (APF) in the cost function as the soft constraint by considering the obstacle as a mass but cannot guarantee collision avoidance [4]. On hard constraint-based methods, Zhang et al. proposed an Optimization-Based Collision Avoidance (OBCA) algorithm by reformulating a smooth obstacle avoidance constraint containing the vehicle geometry [8]. Li et al. proposed a unified OCP model aimed at unstructured parking spaces using a triangle-area criterion to construct the collision-avoidance constraints [9]. However, when parking space is narrow, the feasible domain will be greatly limited due to collision avoidance constraints, resulting in non-linear programming solvers that cannot compute a feasible solution in a finite time.

The neural-network-based method is proposed to generate parking trajectories in narrow scenarios while keeping the time-optimality of the trajectory and low computational overhead. Few studies use neural-network-based methods in automated parking trajectory planning [10,11,12,13,14]. Zhang et al. [15] used Deep Deterministic Policy Gradient (DDPG), a deep reinforcement learning method, to solve the perpendicular parking problem. However, this work focuses only on perpendicular parking and does not consider speed planning. Song et al. [16] proposed a method to solve the parallel parking problem by fusing Nonlinear Programming (NP) and Monte Carlo Tree Search (MCTS), using NP to generate data and train the policy neural network offline, followed by using the policy network to guide MCTS to complete the trajectory planning. The above neural-network-based method can trade off computational accuracy and computational time. However, these methods need to be more generalized and applied to other scenarios because it is only trained in a single scene.

In the field of Intelligent Transportation Systems (ITSs), Federated Learning (FL) has emerged as a promising approach for collaborative model training without the need for lengthy data transfers or sacrificing user privacy. By leveraging the distributed computing power of participating devices, federated learning enables the collaborative training of a globally shared vehicle AI model, which can be utilized by all participating devices for improved performance and efficiency. In recent years, federated learning has gained significant attention from the research community due to its potential to enhance the effectiveness and privacy of ITS [17,18]. For privacy protection in connected vehicles, federated learning has been applied to several intelligent transportation systems to achieve traffic flow prediction, parking reservation and traffic edge computing without privacy leakage [19,20,21,22]. Moreover, the application of Federated Reinforcement Learning (FRL) has extended beyond ITSs to encompass a variety of industrial Internet of Things (IoT) applications, including robot task scheduling [23] and robot navigation tasks [24]. FRL has demonstrated improved generalization and faster convergence of neural network models, enabling the efficient and effective training of complex reinforcement learning models across distributed devices.

In this paper, we propose a hierarchical trajectory planning method with deep reinforcement learning in the federated learning scheme, which is named HALOES, to rapidly and accurately generate collision-free automated parking trajectories in multiple narrow spaces. The FL method considers the computation time and the vehicle kinematic constraints for automated parking trajectory planning in extremely narrow spaces. The contributions of this paper are as follows:

• A novel hierarchical trajectory planning method with deep reinforcement learning and optimization-based approach integration is proposed to achieve computational accuracy and computational time trade-off. The method has a high-level neural network model for rapid reference trajectory generation and a low-level optimization model to refine the trajectory.
• A decentralized training scheme is introduced in the model training module to improve the generalization of model performance by fusing the model parameters trained in different parking scenarios and the federated learning scheme is used in decentralized deep reinforcement learning to protect the privacy of the vehicle’s data during model parameter fusion.
• Simulation results demonstrate that the proposed HALOES method can enable efficient automated parking in narrow spaces and outperforms other state-of-art methods, such as Hybrid A* and OBCA, in terms of planning time, trajectory accuracy and model generalization.

The rest of the paper is as follows. Section 2 presents the kinematics of vehicles, the formulation of model predictive trajectory planning and the background of deep reinforcement learning. The details of HALOES are shown in Section 3. In Section 4, the federated learning scheme is presented. Simulations on several narrow-space parking and detailed analyses of the simulation results are presented in Section 5. Finally, conclusions are drawn in Section 6.

2. Materials and Methods

This section will introduce the backgrounds of deep reinforcement learning and federated learning.

2.1. Deep Reinforcement Learning

Instead of sampling-based and optimization-based approaches, Deep Reinforcement Learning (DRL) seeks to find the optimal policy function ${\mathit{\pi }}^{\ast }$ with neural network parameter $\theta$ [25]. The DRL approach starkly contrasts the model predictive trajectory planning method, as it can achieve model-free operation. In DRL, the agent operates without relying on a predefined environment model. Instead, the agent of DRL interacts directly with the environment by making decisions based on the observation of the current state and the policy function. The basic DRL problem is modeled as a Markov Decision Process (MDP), which can be defined by a tuple of elements $<\mathcal{S},\mathcal{A},\mathcal{P},r,H,\gamma >$, in which $\mathcal{S}$ is the set of states, $\mathcal{A}$ is the set of actions, $\mathcal{P}:\mathcal{S}\times \mathcal{A}\to \mathcal{S}$ maps an action and the current state to the next state, $r:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$ is the reward function that maps an action and a state to a scalar, H is the horizon and $\gamma \in (0,1]$ is a discount factor of reward. At the timestep t, the agent of DRL uses the policy function to determine which action $a_t\in \mathcal{A}$ to take. The policy function maps the current state to a probability distribution over the action space. Then the environment updates to the next state based on the transition function $s_{t+1}=\mathcal{P}(s_t,a_t)\in \mathcal{S}$ and returns a scalar reward $r_t=r(s_t,a_t)$ to the agent for its action. The discounted reward from the state $s_t$ is $R_t(s_t,a_t)={\sum }_{i=0}^H{\gamma }^{i}r(s_{t+i},a_{t+i})$. The ultimate goal of DRL is to find the optimal policy function that maximizes the expected discounted return $\underset{\theta }{argmax}\mathbb{E}\left[R_1(s_t,a_t)|S_t\sim {\rho }^{{\pi }_{\theta }},a_t\sim {\pi }_{\theta }\right]$, in which ${\rho }^{{\pi }_{\theta }}$ is the state visitation distribution under the policy function ${\pi }_{\theta }$.

2.2. Federated Learning

FL is a promising and innovative approach to distributed collaborative Artificial Intelligence (AI) that differs significantly from traditional methods [26]. Unlike centralized approaches, which often require vast amounts of sensitive data to be centralized and shared among multiple participants, FL allows for collaborative training across multiple participants through a centralized server while preserving the privacy and security of the individual actual data. In doing so, FL provides a promising solution for distributed Deep Reinforcement Learning (DRL) that can protect the security and privacy of the distributed DRL participants.

The general FL is divided into two distinct stages, each of which plays a critical role in the training process. The first stage is the distributed local training and update; before starting the training, the server initializes a new neural network model, such as $W_g^0$, and transmits it to the other participants to start the distributed training; each participant $w_m$ trains a local neural network model using its own dataset and update $w_m$ by minimizing the loss function $argminF\left(w_m\right),m\in M$, in which M is the set of distributed DRL participants and $F\left(w_m\right)$ is different in different FL algorithms.

The second stage is model aggregation and downloading. The server aggregates all models into a new version of the global model after updated model collection from all the FL participants; the aggregate process could be defined as $W_g=(1/|M\left|\right)\sum w_m$, after all the updated models collected from the participants; note that the model aggregation method differs for different FL algorithms. The participants download the new global model for optimizing the local model in the next learning round.

3. Hierarchical Trajectory Planning with Deep Reinforcement Learning

This section will introduce the kinematic vehicle models used by HALOES and the overall scheme of HALOES.

3.1. Kinematic Vehicle Model

This paper employs the Kinematic Single-track (KS) model as the kinematic vehicle model. As shown in Figure 2, in the $x-y$ coordinate system, the reference point of the vehicle is the rear-wheel axle mid-point ${[S_x,S_y]}^T$, and the orientation in the global coordinate system is $\Psi$. v represents the current velocity, $\delta$ represents the steering angle, $l_{WB}$ is the wheelbase of vehicle, $l_R$ is the vehicle rear hang length, $l_F$ is the vehicle front hang length and $l_W$ is the vehicle width. According to the KS model, the state transfer equation of the vehicle is as follows,

$$\left[\begin{array}{c}{S}_x(t+\Delta t)\\ S_y(t+\Delta t)\\ v(t+\Delta t)\\ \Psi (t+\Delta t)\\ \delta (t+\Delta t)\end{array}\right]=\left[\begin{array}{c}{S}_x\left(t\right)\\ S_y\left(t\right)\\ v\left(t\right)\\ \Psi \left(t\right)\\ \delta \left(t\right)\end{array}\right]+\left[\begin{array}{c}v\left(t\right)cos\Psi \left(t\right)\\ v\left(t\right)sin\Psi \left(t\right)\\ a\left(t\right)\\ \frac{v\left(t\right)}{l_{WB}}tan\delta \left(t\right)\\ v_{\delta }\left(t\right)\end{array}\right]\Delta t,$$

(1)

in which t and $\Delta t$ represent the current time and the control interval, respectively. $a\left(t\right)$ and $v_{\delta }\left(t\right)$ are the control profiles, representing the acceleration and angular velocity of steering angles at the current time, respectively.

Traditional optimization-based approaches typically use model predictive trajectory planning. Model predictive trajectory planning can be considered as an OCP, which can be solved via an NP solver. Let $\mathit{x}\in {\mathbb{R}}^{n_x}$ be the vehicle state profile and $\mathit{u}\in {\mathbb{R}}^{n_u}$ the control profile; the details of $\mathit{x}$ and $\mathit{u}$ at time t are as follows:

$$\mathit{x}\left(t\right)=[S_x\left(t\right),S_y\left(t\right),v\left(t\right),\Psi \left(t\right),\delta \left(t\right)].$$

(2)

$$\mathit{u}\left(t\right)=[a\left(t\right),v_{\delta }\left(t\right)].$$

(3)

Let $k\in \{1,\dots ,N\}$ represent the discrete time index with time interval $\Delta t$; the state transfer Equation (1) can be written as $\mathit{x}(k+1)=\mathit{x}\left(k\right)+f\left(\mathit{x}\right(k),\mathit{u}(k\left)\right)·\Delta t$, in which $f\left(\mathit{x}\right(k),\mathit{u}(k\left)\right)$ denotes the differential equation of vehicle state $\mathit{x}$ specified which could be written as follows,

$$f(\mathit{x}\left(k\right),\mathit{u}\left(k\right))=\frac{\mathrm{d}}{\mathrm{d}t}\left[\begin{array}{c}{S}_x\left(k\right)\\ S_y\left(k\right)\\ v\left(k\right)\\ \Psi \left(k\right)\\ \delta \left(k\right)\end{array}\right]=\left[\begin{array}{c}v\left(k\right)cos\Psi \left(k\right)\\ v\left(k\right)sin\Psi \left(k\right)\\ a\left(k\right)\\ \frac{v\left(k\right)}{l_{WB}}tan\delta \left(k\right)\\ v_{\delta }\left(k\right)\end{array}\right]$$

(4)

The finite-horizon OCP with collision avoidance constraint can be written as:

$$\begin{array}{cc}\hfill & \underset{\mathit{x},\mathit{u}}{min}\sum _{k=1}^{N}J(\mathit{x}\left(k\right),\mathit{u}\left(k\right)),\hfill \end{array}$$

(5a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& \mathrm{kinematic}\mathrm{principles}(\mathrm{Equation}\left(1\right))\hfill \end{array}$$

(5b)

$$\begin{array}{cc}\hfill & \mathrm{collision}\text{-}\mathrm{free}\mathrm{conditions}\hfill \end{array}$$

(5c)

$$\begin{array}{cc}\hfill & \mathrm{for}\mathrm{all}\mathrm{k}\in \{1,\dots ,N\}.\hfill \end{array}$$

(5d)

in which N represents the prediction horizon, and J is the cost function to penalize deviations from the target point. Equation (5d) is the obstacle avoidance constraint to achieve the obstacle avoidance trajectory.

3.2. DRL-Based Trajectory Planning for Automated Parking

Automated parking trajectory planning can be modeled as a Markov process. This section describes the formulation of the DRL for automated parking trajectory planning and the training process of the DRL model.

3.2.1. Setup of the DRL-Based Trajectory Planning

In this section, we define the setup of the DRL framework, such as the set of state $\mathcal{S}$, the set of action $\mathcal{A}$ and the reward function r.

Automated parking trajectory planning in a narrow space requires constant consideration of vehicle coordinates, target point coordinates, obstacle location information and vehicle kinematic states such as speed and steering wheel angle. In this paper, we define the observed state at the moment of time step t as:

$$S^t=\{P_e^t,{\dot{P}}_g^t,v_e^t,{\delta }_e^t,{\mathbb{V}}_{obs}^t\},$$

(6)

in which $P_e^t=\{S_{x,e}\left(t\right),S_{y,e}\left(t\right),{\Psi }_e\left(t\right)\}$ presents the heading coordinates of the ego vehicle at the time step t, ${\dot{P}}_g^t=P_g^t-P_e^t=\{S_{x,g}\left(t\right)-S_{x,e}\left(t\right),S_{y,g}\left(t\right)-S_{y,e}\left(t\right),{\Psi }_g\left(t\right)-{\Psi }_e\left(t\right)\}$ denotes the position of the target point relative to the ego vehicle, $S_{x,g}\left(t\right),S_{y,g}\left(t\right)$ and ${\Psi }_g\left(t\right)$ represent the coordinates of the two-dimensional vehicle target point $(x,y)$ and the heading angle, respectively. $v_e^t$ and ${\delta }_e^t$ denote the speed and steering wheel angle of the ego vehicle at time step t, respectively. ${\mathbb{V}}_{obs}^t=V_{obs1}^t,V_{obs2}^t,\dots ,V_{obsN}^t$ is the set of relative positions of the obstacles and the ego vehicle, in which $V_{obsN}^t=\{v_{obs,x}^1-S_{x,e}\left(t\right),v_{obs,y}^1-S_{y,e}\left(t\right),\dots ,v_{obs,x}^{|V_{obsN} |}-S_{x,e}\left(t\right),v_{obs,y}^{|V_{obsN} |}-S_{y,e}\left(t\right)\}$ and $|V_{obsN}|$ is the number of vertices of the Nth obstacle.

Since the vehicle kinetic characteristics, such as maximum steering wheel angle and maximum turning radius, need to be fully utilized in parking, the action output of the neural network needs to conform to the vehicle kinetic constraints. In this paper, in order to conform to the vehicle kinetic characteristics, instead of using the relative change of distance as the vehicle action set, e.g., $A=\{\Delta x,\Delta y,\Delta \Psi \}$, the acceleration, as well as the steering wheel cornering rate, are used as the action and the action space is defined as:

$$\mathcal{A}=\{\dot{a},\dot{v_{\delta }}\},\dot{a}\in [-1,1],\dot{v_{\delta }}\in [-1,1].$$

(7)

Normalization of the action space improves the learning efficiency of the neural network. The control inputs to the vehicle are ${\mathcal{A}}_{input}=\{\dot{a}\ast a_{max},\dot{v_{\delta }}\ast v_{\delta ,max}\}$ where $a_{max}$ and $v_{\delta ,max}$ are the maximum values of acceleration and steering wheel rate, respectively.

The reward function in deep reinforcement learning mixes multiple reward values to guide the convergence direction of the model. In automatic parking trajectory planning, three aspects of reward need to be considered; one aspect is the duration of parking, one is the distance to the target point and the last one is the collision with the obstacle. In addition, we use negative rewards to penalize the agent in order to make it able to converge faster. The whole reward function is:

$$r=-c_t\ast R_t-c_d\ast R_d-c_{\Psi }\ast R_{\Psi }-c_o\ast R_o,$$

(8)

where $c_{\ast }$ is the penalty factor for each item. $R_t$ is the time-term penalty as a fixed value. $R_d$ is the penalty term for distance to the target point, which represents the change in distance of the vehicle from the target point:

$$R_d=d_t-d_{t-1},d_t\leftarrow {(S_x\left(t\right)-S_{x,g}\left(t\right))}^2+{(S_y\left(t\right)-S_{y,g}\left(t\right))}^2.$$

(9)

$R_{\Psi }$ is the penalty term for the orientation angle to the target point, which represents the change in orientation of the vehicle from the target point:

$$R_{\Phi }=\Delta {\Psi }_t-\Delta {\Psi }_{t-1},\Delta {\Psi }_t\leftarrow |{\Psi }_g-{\Psi }_t|.$$

(10)

$R_o$ represents the obstacle collision penalty, which applies a fixed penalty when the vehicle collides with an obstacle. There are two prominent cases where a vehicle collides with an obstacle; one is that at least one vehicle with one vertex of the polygon is located inside the obstacle and the other is that at least one obstacle with a vertex of the polygon is located inside the vehicle. At time t, if neither of the two cases exists, the automatic parking trajectory can be considered collision-free with the obstacle. Note that each obstacle polygon should be convex. If the polygonal obstacle is non-convex, it must first be divided into several convex polygons. As shown in Figure 3, the set of vertices of the obstacle is ${\mathcal{V}}_{obs}=\{O_1,O_2,O_3,O_4\}$ and the set of vertices of the vehicle is ${\mathcal{V}}_{veh}=\{V_1,V_2,V_3,V_4\}$. Let any point $Q_{obs}=(x_{obs},y_{obs})\in {\mathcal{V}}_{obs}$ denote the coordinates of a vertex of the obstacle and any point $Q_{veh}=(x_{veh},y_{veh})\in {\mathcal{V}}_{veh}$ denote the coordinates of a vertex of the vehicle. The collision-free condition can be obtained from the triangular area criterion:

$$\begin{array}{c}\hfill S_{\Delta {\mathrm{Q}}_{\mathrm{veh}} {\mathrm{O}}_4{\mathrm{O}}_1}+\sum _{i=1}^3{S}_{\Delta {\mathrm{Q}}_{\mathrm{veh}} {\mathrm{O}}_{\mathrm{i}}{\mathrm{O}}_{\mathrm{i}+1}}>S_{obs},\end{array}$$

(11a)

$$\begin{array}{c}\hfill S_{\Delta {\mathrm{Q}}_{\mathrm{obs}} {\mathrm{V}}_4{\mathrm{V}}_1}+\sum _{i=1}^3{S}_{\Delta {\mathrm{Q}}_{\mathrm{obs}} {\mathrm{V}}_{\mathrm{i}}{\mathrm{V}}_{\mathrm{i}+1}}>S_{veh},\end{array}$$

(11b)

where $S_{\Delta }$ denotes the triangle area, and $S_{obs}$ and $S_{veh}$ denote the area of the polygonal obstacle and the area of the polygonal vehicle. Here, the area of a triangle and the area of a polygonal obstacle can be calculated using the shoelace theorem; let the set of vertices of a convex polygon be ${\mathcal{V}}_{convex}=\{Q_1,Q_2,\dots ,Q_{N_{con}}\}$ where $Q_i=(x_i,y_i)$ denotes the coordinate, then:

$$\begin{array}{c}\hfill S_{Q_1,Q_2,\dots ,Q_{{N }_{con}}}=\frac{1}{2}|(\sum _{n=1}^{N-1}{x}_n{y}_{n+1}+x_{N_{con}}{y}_1)\\ \hfill -(\sum _{n=1}^{N-1}{x}_{n+1}{y}_n+x_1{y}_{N_{con}})|.\end{array}$$

(12)

Thus $R_o$ is defined as follows:

$$R_o=\left\{\begin{array}{cc}0\hfill & \mathrm{Equation}\left(11\mathrm{a}\right)\mathrm{is}\mathrm{true}\mathrm{and}\mathrm{Equation}\left(11\mathrm{b}\right)\mathrm{is}\mathrm{true}\hfill \\ 1\hfill & \mathrm{others}\hfill \end{array}\right..$$

(13)

3.2.2. Training Process of DRL Model

The training goal of reinforcement learning is to find the optimal policy function ${\pi }^{\ast }$, parameterized by $\theta$, which maximizes the expected discount reward of Equation (14).

$$\underset{\theta }{argmax}\mathbb{E}\left[\sum _{i=0}^{H}r(S_t,a_t)|S_t\sim {\rho }^{{\pi }_{\theta }},a_t\sim {\pi }_{\theta }\right],$$

(14)

where r is calculated via Equation (8). The Deep reinforcement learning is mainly divided into the off-policy method and the on-policy method. The off-policy methods, such as Deep Deterministic Policy Gradient (DDPG) and Soft Actor-Critic (SAC), improve sampling efficiency by maintaining the experience pool, so the model can reuse old data for training. The on-policy methods, such as Proximal Policy Optimization (PPO), optimize the same policy as its own decision network, making the model update more stable by directly optimizing the objective function. The trajectory planning process in complex environments often involves many exploration processes, making it a computationally challenging task. Reinforcement learning (RL) methods utilizing off-policy have shown the potential to enhance the model’s performance and stability. By exploiting these existing data, RL methods using off-policy can make more efficient use of available resources and accelerate the training process. As a result, using off-policy in RL methods can improve the overall performance and robustness of the learned policies, making them well suited for real-world applications. DDPG is an Actor-Critic, model-free algorithm based on the deterministic policy gradient used for learning policies in environments with continuous action spaces. DDPG uses two neural networks, an actor and a critic network, to update the policy and value function, respectively. The actor network is updated using the sampled policy gradient:

$${\nabla }_{{\theta }^{\mu }}J=\frac{1}{N}\sum _i^N{\nabla }_{a}Q\left(s,a\mid {\theta }^Q\right){{|}_{s=s_i,a=\mu \left(s_i\right)}{\nabla }_{{\theta }^{\mu }}\mu \left(s\mid {\theta }^{\mu }\right)|}_{s_i},$$

(15)

where N is the mini-batch size, $Q\left(s,a\mid {\theta }^Q\right)$, and $\mu \left(s\mid {\theta }^{\mu }\right)$ is the critic network and actor network with parameters ${\theta }^Q$ and ${\theta }^{\mu }$. The critic network is updated by minimizing the loss:

$$L=\frac{1}{N}\sum _i^N{\left(r\left(s_t,a_t\right)+\gamma Q^{\prime }\left(s_{t+1},{\mu }^{\prime }\left(s_{t+1}|{\theta }^{{\mu }^{\prime }}\right)\mid {\theta }^{Q^{\prime }}\right)-Q\left(s_t,a_t\mid {\theta }^Q\right)\right)}^2,$$

(16)

where $r\left(s_t,a_t\right)$ is the reward function, $Q^{\prime }\left(s,a\mid {\theta }^{Q^{\prime }}\right)$, and ${\mu }^{\prime }\left(s\mid {\theta }^{{\mu }^{\prime }}\right)$ is the target critic network and target actor network with parameters ${\theta }^{Q^{\prime }}$ and ${\theta }^{{\mu }^{\prime }}$. The overall minimization objective of DDPG is as follows:

In Algorithm 1, we use the prioritized experience to replay a statistical technique that enables the efficient utilization of experience to accelerate the convergence speed of the model. Furthermore, the algorithm leverages the addition of Gaussian noise to the decision-making process, enhancing the model’s exploratory power.

Algorithm 1 DDPG-based trajectory planning.

Input: batch size: B, discount factor $\gamma$, target smoothing coefficient $\tau$, number of training episode: M, timesteps of each episode: T, exponents $\alpha$ and $\beta$ of prioritization sampling, noise variance $\sigma$. Initialize actor network $\mu \left(s\mid {\theta }^{\mu }\right)$ and critic network $Q\left(s,a\mid {\theta }^Q\right)$ with parameters ${\theta }^{\mu }$ and ${\theta }^Q$. Initialize target actor network ${\mu }^{\prime }\left(s\mid {\theta }^{{\mu }^{\prime }}\right)$ and target critic network $Q^{\prime }\left(s,a\mid {\theta }^{Q^{\prime }}\right)$ with parameters ${\theta }^{{\mu }^{\prime }}$ and ${\theta }^{Q^{\prime }}$. Initialize prioritized replay memory $\mathcal{B}=⌀$, $\Delta =0$, $p_1=1$. $\mathbf{for}episode=1:M\mathbf{do}$    Reset the environment and receive the initial observation state $s_1$.    $\mathbf{for}t=1:T\mathbf{do}$      Select action $a_t=\mu \left(s_t\right|{\theta }^{\mu })+\mathcal{N}(0,\sigma \mathbf{I})$ according to the actor network and exploration noise      Execute action $a_t$ and obtain the reward $R_t=r(s_t,a_t)$ and the next observation state $s_{t+1}$      Store transition $(s_t,a_t,R_t,s_{t+1})$ in $\mathcal{B}$ with maximal priority $p_t=ma{x}_{i<t}{p}_i^{\alpha }$      Sample transition $j\sim P\left(j\right)=p_j/{\sum }_i{p}_i$      Compute importance-sampling weight ${\omega }_j={(B\dot{P}(j))}^{-\beta }/ma{x}_i{\omega }_i$      Compute TD-error ${\delta }_j=R_j+{\gamma }_j{Q}^{\prime }(s_j,{\mu }^{\prime }\left(s_j\right|{\theta }^{{\mu }^{\prime }})|{\theta }^{Q^{\prime })}-Q(s_{j-1},a_{j-1}\mid {\theta }^Q)$      Update critic by minimizing the loss (16)      Update actor using the sample policy gradient (15)      Update transition priority $p_j=\left|{\delta }_j\right|$      Accumulate weight-change $\Delta =\Delta +{\omega }_j{\dot{\delta }}_j{\nabla }_{\theta }Q(s_{j-1},a_{j-1}|{\theta }^Q)$      Update the target network:      ${\theta }^{Q^{\prime }}\leftarrow \tau {\theta }^Q+(1-\tau ){\theta }^{Q^{\prime }}$      ${\theta }^{{\mu }^{\prime }}\leftarrow \tau {\theta }^{\mu }+(1-\tau ){\theta }^{{\mu }^{\prime }}$    $\mathbf{end}\mathbf{for}$ $\mathbf{end}\mathbf{for}$

3.3. Optimization-Based Trajectory Planning

A trajectory from the starting point to the target point will be obtained quickly after trajectory planning via the DRL-based trajectory. Still, some trajectories penetrate obstacles due to the DRL-based trajectory, so it needs to be corrected via the optimization-based method. Because the reference point of the hot start is obtained via DRL-Based trajectory planning, optimization-based trajectory planning based on obstacle constraint can be solved quickly. Suppose $\mathbb{T}=\left\{\mathit{x}\right(0),\mathit{x}(1),\mathit{x}(2),\dots ,\mathit{x}(N\left)\right\}$ is the trajectory point obtained via soft constraint path planning. It is sampled before passing hard-constrained trajectory planning, which can be upsampling or downsampling, depending on the target accuracy of trajectory planning. Finally, the sampled trajectory point is obtained as ${\mathbb{T}}_{sampled}^{ref}=\{{\mathit{x}}^{ref}\left(0\right),{\mathit{x}}^{ref}\left(1\right),{\mathit{x}}^{ref}\left(2\right),\dots ,{\mathit{x}}^{ref}\left(M\right)\}$ where M is the number of trajectory points after sampling.

In optimization-based trajectory planning, this paper uses sampled trajectory points to constrain the reference curve of OBCA, which uses the exact vehicle geometry to develop collision avoidance constraints for tight maneuvers in a narrow space and the nonlinear optimization is formulated as follows:

$$\begin{array}{cc}\hfill & \underset{{\mathit{x}}^{\prime },{\mathit{u}}^{\prime }}{min}L({\mathbb{X}}^{\prime },{\mathbb{T}}_{sample}^{ref},{\mathbb{U}}^{\prime }),\hfill \end{array}$$

(17a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& {\mathit{x}}^{\prime }\left(0\right)={\mathit{x}}_{\mathrm{start}},{\mathit{u}}^{\prime }\left(0\right)={\mathit{u}}_{\mathrm{start}},\hfill \end{array}$$

(17b)

$$\begin{array}{c}{\mathit{x}}^{\prime }\left(M\right)={\mathit{x}}_{\mathrm{target}},\hfill \end{array}$$

(17c)

$$\begin{array}{cc}\hfill & {\mathit{x}}^{\prime }(k+1)={\mathit{x}}^{\prime }\left(k\right)+f({\mathit{x}}^{\prime }\left(k\right),{\mathit{u}}^{\prime }\left(k\right))·\Delta t,\hfill \end{array}$$

(17d)

$$\begin{array}{cc}\hfill & {\mathit{x}}_{\min }\le {\mathit{x}}^{\prime }\left(k\right)\le {\mathit{x}}_{\max },\hfill \end{array}$$

(17e)

$$\begin{array}{cc}\hfill & {\mathit{u}}_{\min }\le {\mathit{u}}^{\prime }\left(k\right)\le {\mathit{u}}_{\max },\hfill \end{array}$$

(17f)

$$\begin{array}{cc}\hfill & -g^{\top }{\mu }_k^{\left(l\right)}+{\left(A^{\left(l\right)}t\left({\mathit{x}}^{\prime }\left(k\right)\right)-b^{\left(l\right)}\right)}^{\top }{\lambda }_k^{\left(l\right)}>0,\hfill \end{array}$$

(17g)

$$\begin{array}{cc}\hfill & G^{\top }{\mu }_k^{\left(l\right)}+r{\left({\mathit{x}}^{\prime }\left(k\right)\right)}^{\top }{A}^{{\left(l\right)}^{\top }}{\lambda }_k^{\left(l\right)}=0,\hfill \end{array}$$

(17h)

$$\begin{array}{cc}\hfill & ∥A^{{\left(l\right)}^{\top }}{\lambda }_k^{\left(l\right)}∥\le 1,{\lambda }_k^{\left(l\right)}⩾0,{\mu }_k^{\left(l\right)}⩾0,\hfill \end{array}$$

(17i)

$$\begin{array}{cc}\hfill & \mathrm{for}\mathrm{all}\mathrm{k}\in \{1,\dots ,M\}\mathrm{for}\mathrm{all}\mathrm{l}\in \{1,\dots ,L\},\hfill \end{array}$$

(17j)

where ${\mathbb{X}}^{\prime }=\{{\mathit{x}}^{\prime }\left(0\right),{\mathit{x}}^{\prime }\left(1\right),\dots ,{\mathit{x}}^{\prime }\left(M\right)\}$ and ${\mathbb{U}}^{\prime }=\{{\mathit{u}}^{\prime }\left(0\right),{\mathit{u}}^{\prime }\left(1\right),\dots ,{\mathit{u}}^{\prime }\left(M\right)\}$ are the trajectory and control sequence generated via hard constraint trajectory planning, respectively. $L({\mathbb{X}}^{\prime },{\mathbb{T}}_{sample}^{ref},{\mathbb{U}}^{\prime })=Q{\sum }_{k=0}^M\parallel {\mathit{x}}^{\prime }\left(k\right)-x^{ref}\left(k\right){\parallel }^2+R{\sum }_{k=0}^{M-1}{\parallel {\mathit{u}}^{\prime }\left(k\right)\parallel }^2$ is the penalty term for the deviation distance from the reference trajectory and the size of the control magnitude. Equations (17g)–(17i) are the smooth and exact reformulations using Theorem 2 in [8].

3.4. Overall Scheme of Hierarchical Trajectory Planning with Deep Reinforcement Learning

The framework of Hierarchical Trajectory Planning with Deep Reinforcement Learning (HTP-DRL) is shown in Figure 4, which is mainly divided into two parts DRL-based trajectory planning and optimization-based trajectory planning. After obtaining the parking space information and the vehicle’s parameters, a reinforcement learning model is used to make decisions based on the observed state and a coarse trajectory from the start to the goal point is output. A vehicle dynamics model is used to update the state. Compared to traditional optimization-based online trajectory planning methods or sampling-based trajectory planning methods that require constant collision detection and the computational time overhead caused by non-linear solving, a rough trajectory from start to finish can be obtained quickly with the reinforcement learning model.

After obtaining the rough trajectory via DRL-based trajectory planning, it is necessary to use optimization-based trajectory planning to constrain it at some points to achieve collision-free automatic parking trajectory planning. Here, we refer to the method of OBCA, which can achieve more accurate collision avoidance by using exact vehicle geometry and since DRL-based trajectory planning has obtained the reference trajectory points, the computational speed of the method of OBCA can converge faster so that the results of automatic parking trajectory planning can be solved in a limited time.

4. Federated Learning-Based Model Training

The integration of communication technology and intelligent terminals has formed an intelligent transportation system aimed at improving the safety and efficiency of transportation. In order to realize an intelligent transportation system, the widespread application of artificial intelligence technology benefits from its ability to access the large amount of real-time activity data generated by traffic participants. Therefore, most AI-based intelligent transportation system solutions rely on centralized learning frameworks on data centers. Model training methods based on federated learning can have high privacy and low communication latency. Thus, federated learning plays an important role in handling privacy-protected distributed trajectory planning training based on reinforcement learning.

The proposed method for training a reinforcement learning model based on federated learning is designed to achieve a privacy-preserving centralized reinforcement learning framework, as shown in Figure 1b. Algorithm 2 presents the centralized reinforcement learning method under the federated learning framework in pseudo code form. At the start of the training process, N federated learning participant vehicles download global model parameters, including actor and critic network parameters, from the central server node via base stations. Subsequently, each participant trained their own model based on the data collected from their respective environments. Afterward, each intelligent agent uploads their own model parameters to the central server. The central server node then performs the model aggregate update for all models and distributes the updated models to all participants.

Algorithm 2 The federated learning-based reinforcement learning model training.

Input: number of federated learning participants N, parameter aggregate intervals L, number of training episodes M. Initial global actor network and global critic network parameters $\mathbf{W}\left(0\right)$. Initial participant actor network and participant critic network parameters ${\mathbf{W}}_n\left(0\right),(n=1,2,\dots ,N)$. $\mathbf{for}episodei=1:M\mathbf{do}$    Each DRL participant downloads $\mathbf{W}(i-1)$ from the central server node.    $\mathbf{for}interval=1:L\mathbf{do}$      Each DRL participant updates locally with ${\mathbf{W}}_n\left(i\right)$ on the owner current observation    $\mathbf{end}\mathbf{for}$    Each DRL participant uploads the trained model parameters $W_n\left(i\right)$ to the central server    Central server receives all weights ${\mathbf{W}}_n\left(i\right)$ and performs the federated averaging for $\mathbf{W}\left(i\right)$    Broadcast the averaged model parameters $\mathbf{W}\left(i\right)$ $\mathbf{end}\mathbf{for}$

5. Experiment and Result

5.1. Environment Setup

In our experiments, we use some cases from the dataset https://tpcap.github.io/benchmarks/ (accessed on 14 April 2023) [27] for algorithm validation, as shown in Figure 5, where cases 1–3 are normal cases representing parallel, vertical and oblique parking cases, respectively, where no regular parking position is set and irregularly prevented obstacles are set to constrain. Cases 4 and 5 represent cases with extremely narrow parking spaces. In case 4, if using the sampling-based trajectory planning method, the accuracy required for sampling is particularly largely affected by the curse of dimensionality, which makes the sampling-based path planning method unable to search for a trajectory from the starting point to the target point in a limited time. If the optimization-based method is used directly, a high collocation point density needs to be set, which renders a large-scale mathematical programming problem. The kinematic parameters of the vehicle are listed in

Table 1. Parameters of Vehicle.

Parameter	Parameter Description	Value
$l_{wb}$	The wheelbase of vehicle	2.8 m
$l_R$	The rear hang length of vehicle	0.929 m
$l_f$	The front hang length of vehicle	0.96 m
$l_w$	The width of vehicle	$1.942$
$a_{max}$	The upper bound of acceleration	$1{\mathrm{m}/\mathrm{s}}^2$
$a_{min}$	The downer bound of acceleration	$-1{\mathrm{m}/\mathrm{s}}^2$
$v_{max}$	The upper bound of velocity	$2.5\mathrm{m}/\mathrm{s}$
$v_{min}$	The downer bound of velocity	$-2.5\mathrm{m}/\mathrm{s}$
${\varphi }_{max}$	The maximum steering angle	0.75 rad
$v_{\varphi max}$	The maximum angular velocity of steering angle	0.5 rad/s
$\Delta t$	The control time interval of RL-based method	0.1 s

.

This paper mainly uses three deep reinforcement learning methods for validation, namely DDPG, SAC and PPO, and implements HALOES based on DDPG. The model parameters of the neural network are mainly shown in

Table 2. Parameters of DDPG.

Parameter	Parameter Description	Value
M	The total episodes of training	2500
T	The total timesteps of each episode	800
B	The batch size of training	256
$\gamma$	The reward discounted factor	$0.99$
$\sigma$	The noise variance	$0.01$
$\tau$	The target smoothing coefficient	$0.05$
$l_{actor}$	The learning rate of actor-network	$1\times {10}^{-4}$
$v{l}_{critic}$	The learning rate of critic-network	$1\times {10}^{-3}$
$\alpha$	The parameter of prioritization sampling	$0.6$
$\beta$	The parameter of prioritization sampling	$0.9$

. In addition, ReLU and Adam are used as the activation function and the optimizer of the neural network. The experiments are trained and tested on a platform with RTX A4000 and Intel(R) Xeon(R) Gold 5320 CPU at 2.20 GHz, respectively. All experiments are conducted in Python on a Linux system.

5.2. Result and Discussion

The first part is used to verify the convergence performance of the reinforcement learning part of HALOES. The performances of DDPG, SAC and PPO in trajectory planning are compared. The test environment used is case 4, where 2500 episodes are trained in case 4 to compare the convergence of the final reward values, where the reward value is calculated by summing the reward values of all steps in each episode. As shown in Figure 6, better performance is achieved using DDPG than SAC and PPO because the use of the Actor-Critic network can effectively learn the optimal decision from the historical trajectory. To verify the effect of using relative position relations on reinforcement learning, we modified the relative positions in the observed states to absolute positions named DDPG-withoutRel and SAC-withoutRel. To be more specific, the input state of DDPG-withoutRel and SAC-withoutRel is changed as $S^t=\{P_e^t,P_g^t,v_e^t,{\delta }_e^t,{\mathbb{V}}_{obs}^t\},$ where $P_g^t$ represents the absolute position of the goal point. Compared with DDPG-withoutRel and SAC-withoutRel, the use of the relative position relation, as implemented in the DDPG, has been shown to yield superior results compared to other approaches. By utilizing relative position information, the DDPG effectively reduces the dimensionality of the state space, independent of the agent’s absolute position. This reduction in state space dimensionality has been shown to enhance the generalization capabilities of the model, leading to improved performance across a range of scenarios. Compared to the offline training method of Actor-Critic networks such as DDPG and SAC, the performance of PPO using online training is poor and there is no significant improvement in training because the training method using online reinforcement learning does not record the historical optimal information, resulting in the model being explored locally, which makes the model unable to find the optimal gradient. This is due to the fact that online training does not allow the model to find the optimal gradient because the model keeps exploring locally. Because DDPG outperformed both SAC and PPO in terms of trajectory planning, subsequent experiments were carried out on the basis of DDPG.

The second part validates the convergence of federated reinforcement learning in HALOES compared to original reinforcement learning. The results presented in Figure 7 highlight the comparative performance of Federated Learning-based Deep Deterministic Policy Gradient (FedDDPG) with respect to the original DDPG algorithm in multiple training scenarios. Specifically, the DDPG algorithm was utilized for training a common model on randomly selected cases 1–5, while FedDDPG involved the collaborative training of a model across four participants in multiple environments, followed by the fusion of the trained models. The horizontal axis of the figure indicates the number of testing rounds, with the model being evaluated after every 20 rounds of training. The vertical axis represents the total reward achieved during each evaluation. The results demonstrate that FedDDPG achieves comparable or better performance than the original DDPG algorithm across all scenarios, with the former exhibiting a more stable and consistent learning curve over time. Due to the high level of privacy protection currently being pursued in intelligent transport systems, federated learning allows only model parameters to be shared without causing privacy issues for individual vehicles. It can be seen that the use of federated learning can achieve the same performance as the original DDPG while achieving data privacy protection. In (d,e) in Figure 7, we can see that the fluctuation of the DDPG trajectory planning based on federated learning is more stable after convergence compared with the original DDPG. The experimental results presented in Figure 7b–d demonstrate the susceptibility of federated learning models to perturbations in the early stages of training, particularly in the context of model fusion. Specifically, the observed instability in model performance during testing can be attributed to the sensitivity of the federated learning process to early fluctuations in model weights and updates, which can significantly impact the final model. However, as the number of training sessions increases, the federated learning model gradually becomes more stable, as evidenced by the improved performance observed over time. The above situation is more obvious in Figure 7d, where the model is updated by multiple participants making it too far from the original model, resulting in large performance fluctuations during the training phase, but after the model converges, stable performance can be maintained with less fluctuations compared to DDPG. This can be attributed to the ability of the federated learning process to aggregate increasingly larger and more diverse datasets over time, resulting in a more robust and accurate model that is better able to generalize to new scenarios. Figure 7a shows that FedDDPG has a more stable performance than the original DDPG, which has performance fluctuations after training.

The third part is used to verify whether the proposed HALOES can accomplish the trajectory planning task under narrow space. This paper mainly verifies five scenarios, cases 1–5. The scenarios of cases 1 and 4 are parallel parking spaces and case 4 is narrower compared to case 1. Cases 2 and 5 are vertical parking spaces and case 5 is narrower compared to case 2. Case 3 is an oblique parking scene. The first row of Figure 8 shows the output reference trajectory of the DRL method of the first layer in HALOES. The second row of Figure 8 shows the final planned trajectory of HALOES in cases 1–5. Figure 9 shows the vehicle kinetic parameters of HALOES and Opt+OBCA at any moment under cases 2–5; we mainly consider the following parameters: speed, acceleration, steering wheel angle and steering wheel angle rate. Based on the comparison of the results of the DRL method and HALOES, it can be seen that using the method of DRL can provide a reference trajectory to the target point for automated parking trajectory planning. Since there is no obstacle collision as a constraint in DRL, the trajectory will overlap with the obstacle, but it can be quickly corrected by the optimization-based method in the second layer of HALOES to generate a collision-free trajectory to the target point. In addition, based on the comparison of the kinetic curves of HALOES and Opt+OBCA, it can be seen that the trajectory planning with reference by reinforcement learning and the trajectory planning with an optimization-based reference can be kept within a reasonable kinetic constraint. Cases 2, 3 and 5 can all reach the target position through continuous control and the kinetic curve at the planning is smooth. The trajectory in case 4 needs to include multiple vehicle braking because it is necessary to make multiple round-trip operations when parking in a narrow space. According to case 4 in Figure 8 and the kinetic curves in Figure 9, we can see that the vehicle makes multiple round-trip operations. This is due to the fact that lateral parking in a narrow space requires continuous gear switching, which requires making full use of the vehicle’s steering angle to maintain a wide range. However, finally, the vehicle can eventually stop at the target position in compliance with the kinetic constraints.

To verify the advantages of HALOES in terms of planning time, we compared the average planning times of Hybrid A* as well as Opt+OBCA in cases 1–5. Where Hybrid A* planning results contain only trajectory points and without vehicle control parameters, additional speed planning is necessary to complete the task. As OBCA requires a warm start term to calculate the reference trajectory points, in this paper, we do not use the Hybrid A* approach to calculate the reference trajectory. Instead, we use an optimization-based approach to calculate the reference trajectory, where an artificial potential field is used to define the optimized loss function and the calculated reference trajectory is then processed via OBCA to obtain the trajectory points needed for vehicle execution. The trajectories planned via Hybrid A* and Opt+OBCA in cases 1–5 are shown in the second and third row of Figure 8, respectively, and the trajectory of case 4 could not be solved in a limited time due to the tendency of Hybrid A* to have dimensional collapse. As can be seen from

Table 3. Comparison of Hybrid A*, Opt+OBCA and HALOES in terms of planning time.

	Case 1	Case 2	Case 3	Case 4	Case 5
HybridA*	325.811 s	49.994 s	87.733 s	NULL	26.067 s
Opt+OBCA	42.805 s	42.805 s	40.932 s	75.723 s	35.546 s
HALOES(ours)	27.955 s	14.546 s	17.159 s	36.359 s	31.228 s
Improved over OBCA	$34.69\%$	$66.02\%$	$58.08\%$	$51.98\%$	$12.15\%$

, HALOES achieves significant speedups in cases 1–4. In case 5, the parking space is too narrow, resulting in the computation time of both HALOES and Opt+OBCA exceeding that of Hybrid A*, but HALOES uses a reinforcement learning model as a guide, thus making the computation faster than the Opt+OBCA approach. Since Hybrid A* only completes the trajectory point planning and then needs to perform velocity quadratic planning afterward, Hybrid A* still needs to complete the velocity quadratic planning in case 5, regardless of the short computation time. Therefore it can be considered that HALOES has a clear advantage in terms of planning time.

6. Conclusions

In this paper, to improve automated parking in narrow spaces, HALOES, a novel hierarchical trajectory planning method with deep reinforcement learning and optimization-based approach integration, is proposed to achieve computational accuracy and computational time trade-off. HALOES uses DDPG as a baseline and uses federated learning to train reinforcement learning models for privacy-preserving training of intelligent agent systems in intelligent transportation systems. HALOES proposes a novel hierarchical approach to trajectory planning, where the output of the reinforcement learning model is used for secondary optimization of the trajectory to achieve a trade-off between computational time and accuracy in trajectory planning. Extensive experiments have demonstrated that HALOES generalizes well to a wide range of scenarios and achieves the same accuracy as the reinforcement learning algorithm, and by verifying the planning time in a wide range of cases, it can be seen that HALOES is significantly better than Hybrid A* and OBCA.

The scenarios verified in this paper are all static obstacles, so the proposed method is limited when moving obstacles are present. Automatic parking trajectory planning under the influence of moving obstacles is a more realistic and complex problem. Therefore, in future research work, we will focus on the optimization method of automatic parking trajectory planning in the case of moving obstacles such as other vehicles and pedestrians, existing in narrow parking environments.