MUGI-Net: A Group-Aware Pedestrian Trajectory Prediction Model for Autonomous Vehicles from First-Person View

With the ongoing evolution of autonomous driving technology and intelligent transportation systems (ITS), pedestrian safety has become an important research topic in traffic safety [1]. Especially against the backdrop of the accelerated implementation of advanced driver assistance and autonomous driving functions, how to achieve reliable prediction of pedestrians’ future motion states, enabling vehicles to plan online and adjust motion strategies promptly to reduce potential collision risks, has become a key issue that needs to be broken through [2]. Direct value for improving road traffic safety is not only provided by pedestrian trajectory prediction, but also by fundamental support for refined, intelligent regulation of smart city traffic management and traffic flow [3,4]. In this context, pedestrian trajectory prediction from the first-person view (FPV) has gradually attracted attention: this paradigm models directly based on on-board forward perception information, avoiding to a certain extent the geometric errors and cumulative deviations introduced by view transformation and coordinate mapping in bird’s-eye view (BEV) methods [5,6,7,8].

However, the FPV scenario is characterized by more significant dynamic changes in imaging perspective, occlusion, and scale transformation, making it more difficult to stably characterize the motion-interaction relationships between pedestrians in the scene [9,10]. To improve FPV prediction performance, existing studies have proposed multi-modal prediction frameworks that usually combine historical trajectories and relevant contextual information to generate multiple feasible future trajectories [11,12,13]. Dynamic changes in the FPV scenario are addressed by these methods to some extent. However, they remain insufficient for modeling dynamic interactions at the group level. Individual historical motion and local spatial cues are focused on by most works, making it difficult to fully characterize the time-evolving interaction effects among pedestrians, thereby limiting the generalization ability and the medium- to long-term prediction accuracy in complex traffic environments [14,15].

Overall, existing FPV pedestrian trajectory prediction methods still face the following challenges: (i) The interaction relationships among pedestrians are highly nonlinear and time-varying. Existing models usually focus on a single trajectory of the target pedestrian, lack spatiotemporal feature fusion, and make it difficult to quantify the contribution of individuals to group behavior, thereby limiting explicit modeling capability for complex group interactions and the accurate characterization of social relationships and group interaction laws. (ii) Existing models mostly rely on relatively static pattern assumptions, and traditional temporal modeling structures tend to smooth out segment differences and lose short-term details, making it difficult to depict the rapid switching of pedestrians’ local behavior patterns, resulting in continuous error accumulation in long temporal prediction and the inability to balance the capture of local dynamic patterns and the global temporal dependence.

To address the above problems, this paper proposes a method called Mixture of Universals and Group Interaction Network (MUGI-Net), as shown in Figure 1. The main innovations are summarized as follows:

Experimental results show that the method in this paper achieves excellent performance in the FPV pedestrian trajectory prediction task: on the JAAD dataset, the model achieves MSE_15 = 153 in the 1.5 s prediction task, which is about 5% lower than the current state-of-the-art AANet method (161); on the PIE dataset, the model achieves MSE_10 = 33 in the 1.0 s prediction task, and outperforms most existing methods in indicators such as MSE_15, CMSE and CFMSE, verifying the effectiveness of the proposed group interaction modeling and hybrid temporal encoding structure. The rest of this paper is organized as follows: Section 2 introduces related research; Section 3 elaborates on the proposed pedestrian trajectory prediction model and its key modules; Section 4 presents the experimental settings and analyzes the results; and Section 5 summarizes the paper and outlines future research directions.

This section systematically reviews existing research on pedestrian trajectory prediction and the temporal modeling techniques used to model it. Mainstream methods and the existing limitations of related tasks are outlined.

In recent years, deep learning has advanced significantly in FPV trajectory prediction. For instance, many studies combine CNNs with RNNs to predict future positions using historical trajectory data, capturing spatiotemporal relationships [16,17,18,19,20]. Due to the randomness of pedestrian trajectories, Xu et al. [21] proposed a multi-modal framework combining LSTM and CNN to generate diverse trajectories, applicable to FPV scenarios. As intelligent transportation development advances, FPV prediction also requires contextual information. Niu et al. [22] proposed the AANet model, which processes pedestrian trajectories and ego-vehicle motion using a two-stream architecture to enhance scene understanding by learning about agent interactions.

Modeling pedestrian social and group interactions is another key direction. SHENet correlates individual and group trajectories with environmental information, while BiTraP captures bidirectional group motion correlation [17]. Recent studies integrated collision awareness into the graph Transformer [23], but lack FPV-specific optimizations. Dynamic subclass-balancing contrastive learning addresses long-tail trajectory issues [24]. Federated learning combined with dual-channel destination guidance protects privacy and improves trajectory rationality [25]. The TPPO predictor with a pseudo-Oracle module provides prior information for multi-modal prediction [26]. Despite covering multiple dimensions, previous studies did not sufficiently explore group interaction, and pedestrian analysis was isolated. To address this, pedestrians are grouped by relative direction and distance, and intra- and inter-group interactions are explored to improve FPV prediction performance.

In recent years, the temporal encoding module has become a core component in trajectory prediction models. Early works are mainly based on RNNs (e.g., LSTM, GRU), which encode observed trajectory sequences into hidden states to model long-term dependence between historical and future motions [27,28]. With Transformers, work increasingly uses self-attention to replace/supplement RNN encoding for unified temporal modeling of long-range dependence and multi-agent interaction [29,30]. Yuan et al.’s AgentFormer [27] applies self-attention in temporal and agent dimensions to encode the “socio-temporal” structure and improve prediction performance. To enhance scene adaptability, an adaptive progressive Transformer adjusts attention weights based on scene features [31]. For fast cross-scene adaptation, meta-inverse reinforcement learning [32] and online multi-source transfer learning [33,34] enhance generalization from different perspectives by mining motion reward functions and extracting universal temporal features, respectively.

Traditional temporal encoding has limitations: RNNs are limited in modeling long-term dependencies, Transformers weaken local details, and linear state-space models lack expressiveness. To address this, this paper proposes a MoU-structured temporal encoding method that balances local representation and global temporal dependence while maintaining controllable complexity.

The overall framework of the proposed FPV pedestrian trajectory prediction model is detailed in this section. To address problems of unstable grouping, insufficient temporal feature modeling, and single-modal prediction bias in first-person view scenarios, the model comprises four core components: a group interaction module, a MoU-structured temporal module, and a decoder with intention anchor refinement. Specifically, the group interaction module accurately groups pedestrians and models fine-grained intra- and inter-group interactions in FPV scenes, the temporal module efficiently captures multi-scale temporal dependencies via a hybrid structural design, and the decoder generates diverse and reasonable future trajectories through iterative intention refinement. The design details of each module are introduced sequentially below. The symbols used in this work are shown in Table 1.

Given a historical horizon of T_h, the goal is to predict K possible trajectories of pedestrians in the future T_f time steps, denoted as $$Y_j=\{y_{T_h+1}\mathrm{,}y_{T_h+2}\mathrm{,}...y_{T_h+T_f}\},j\in K$$. As shown in Figure 2, the encoder module processes historical trajectories and inter-group information via positional encoding and a temporal module to produce representations suitable for model computation. In the temporal module, features are extracted using token embeddings and Mamba layers. Then, in the decoder module, convolutional layers and attention mechanisms are used to further capture local features and critical information. Self-attention and cross-attention mechanisms help the model focus on different parts of the input and on relationships between modalities, and the final trajectory prediction results are generated by a feed-forward network. The key modules are elaborated below.

FPV pedestrian trajectory prediction is affected by both pedestrian motion and ego-motion. Thus, traditional group interaction modeling for BEV may lead to unstable grouping, identity confusion, and distorted interactions. To address the above challenges, we propose a group interaction module, as shown in Figure 3. Its overall process is elaborated as follows: (1) the historical trajectories are used for dynamic grouping; (2) the grouping results are used to construct intra-group interaction graphs and inter-group interaction graphs; (3) intra-group interaction features and inter-group interaction features are extracted from the two types of graphs respectively; (4) these two types of interaction features are then fused with historical trajectory features; (5) the fused features are used as the input of the temporal encoding module and enter MoF and MoA; (6) the encoding results are then sent to the decoder to complete the prediction.

The above processes allow the group interaction module to effectively extract group-level social interaction features while preserving individual identity information, providing richer, structured input for the subsequent temporal encoding module and improving the overall performance of FPV pedestrian trajectory prediction. As shown in Step 1 of Figure 3, the set of bounding boxes of all pedestrians at time t within the observation time range T_h is expressed as:

where x_t denotes the bounding box at the time step t, and N_x is its dimension.

In the pedestrian trajectory prediction task, each node must retain its original identity information and describe the dynamic attributes of group behavior in the scene, thereby providing a basis for restoring the original identity information in subsequent steps. Existing studies have failed to adequately address pedestrian behavioral characteristics, especially motion direction.

Group division based solely on Euclidean distance calculations is prone to trajectory crossing or identity confusion [30]. To solve this problem, A method that fuses Euclidean distance and velocity direction is proposed, accounting for the influence of both motion velocity and spatial distance. As shown in Step 2 of Figure 3, because the distance, velocity difference, and direction angle between individuals in the same group may have significant characterizing effects, feature similarity among all pedestrians is calculated using historical trajectories, Euclidean distances, and velocity magnitudes and directions for each pedestrian. Based on the obtained similarity measurement, pedestrian combinations that may belong to the same group can be effectively identified. For this purpose, A pairwise distance matrix D is defined, and a set of group membership indicator variables P is introduced, which are specifically expressed as follows:

where F_Φ is a learnable convolutional layer for learning deep group features. X_i and V_i represent position information and velocity, respectively. Π is a learnable threshold parameter. Pair(∙,∙) represents pedestrians that may belong to the same group.

Subsequently, group indicators are constructed by analyzing the relationships among group members. In this context, the variable k represents the index of the k-th group, which is the union of each paired set (i, j). The key point is that there is no overlap between members across groups. Therefore, the group index G is defined as a set containing all members of a specific group:

To deeply explore the interaction relationships among pedestrians, A graph network is adopted for modeling [31] in this study. In this model, it is necessary to maintain the identity index information of each graph node to ensure that relevant nodes are included and irrelevant nodes are excluded. However, existing methods often use an average aggregation across the entire group, which is inconsistent with the study’s goal. From the first-person view, average aggregation will further amplify the perspective scale difference: the bounding boxes of pedestrians closer to the camera change more drastically, and undifferentiated averaging is likely to lead to the group representation being dominated by “large-scale changes in the near field”.

To achieve this goal, a new method called attentional group pooling and unpooling is proposed. As shown in Step 3 of Figure 3, first, pedestrian nodes are grouped based on common behavior patterns and surrounding information sets. During this grouping process, the features of the corresponding nodes are aggregated into a single representative node for each pedestrian group, guided by attention weights. The aggregated group features are stacked for use in subsequent modules to learn sparse connections in the spatiotemporal graph. Attentional pooling is used to select the most representative features for each pedestrian node. The influence of group members is determined by their velocity, with higher velocity leading to greater influence and being assigned larger weights. Using the above method, we construct a graph with significantly fewer nodes than the input graph. The clustered trajectory features Z are defined as follows:

where K is the total number of groups, W is the attention weight, and X_i is the feature of each member.

To restore the grouped graph structure to the original scale, an unpooling operation is required. This method can predict each agent’s trajectory based on the agent’s output features and fusion information. Since the convolution operation on zero-vector nodes cannot effectively capture group attributes [35,36], this study evenly distributes group features Z to N relevant group-member nodes and reintroduces geometric information related to camera motion in the unpooling stage, so that each member node obtains the same group behavior information. The pedestrian group unpooling operator is defined as follows:

Social interaction is integrated into the sparsely connected spatiotemporal graph network at the group level. As shown in Step 4 of Figure 3, by providing three different graph-structure data types (intra-group interaction, inter-group interaction, and historical trajectory) to the same base model, the model can extract diverse features. The hierarchical structure for mapping intra-group interaction features is at the individual level. The nodes remain pedestrians. The edges only connect members within the same group. The main modeling focuses on the local coordination relationships among group members, including speed consistency, formation maintenance, mutual following, and collision-avoidance constraints. The hierarchical level for constructing interaction features between groups is the group level; the nodes have become “group nodes”. The edges represent the interactions between different groups. The main modeling focuses on the high-level relationships between groups, such as yielding, overlapping, competing for space, and overall avoidance. The pedestrian graph G_ped = (V_ped, E_ped) is defined as a set V_ped = {X_n | n ∈ [1, 2, …, N]} of pedestrian nodes and a set E_ped = {e__ij | i, j ∈ [1, 2, …, N]}of edges representing pairwise social interactions. The intra-group interaction graph is defined as G_member = (V_ped, E_member), which consists of a set V_ped of pedestrian nodes and a set E_member of edges representing pairwise social interactions among group members, where E_member = {e_ij ∣ i, j ∈ [1, 2, …, N],(i, j) ⊂ G_k, k ∈ [1, 2, …, N]}. This graph enables pedestrian nodes to learn avoidance norms within the group while maintaining their formation and motion direction. From the first-person view, intra-group interaction also needs to maintain invariance to “overall formation translation/scaling caused by camera following”: by introducing an ego-motion correction term in the intra-group edge features, the model can focus more on the changes in relative motion (relative velocity/relative direction) among members rather than global view drift.

Inter-group interaction is also crucial for learning social norms between different groups. The inter-group interaction graph is defined as G_group = (V_group, E_group), where nodes V_group = {X_k ∣ k ∈ [1, 2, …, K]} represent the features of each group and edges E_group = {e_p,q ∣ p,q ∈ [1, 2, …, K]} represent the interactions between paired groups. In the first-person scenario, inter-group relationships are affected by the field of view and perspective depth: distant groups appear denser and smaller in the image, while near groups are sparser and exhibit greater scale changes. For this reason, inter-group edge features can encode both “relative orientation/scale change trend” and “ego-motion consistency”, thereby improving the ability to model interactions across distance scales.

In this framework, weights are shared by the model, thus reducing the number of parameters and improving the overall performance. Subsequently, the model’s output features are aggregated into agents, and the group fusion module is used to predict the probability distribution of future trajectories. The predicted trajectory Ŷ generated by the group fusion module F_ψ is expressed as:

where F_ψ and F_θ are learnable parameters, which are randomly initialized at the beginning.

The temporal module is the core component of the Mixture of Universals (MoU) model, by which short- and long-term dependence relationships in temporal data are effectively captured. As shown in Figure 4, unlike the traditional Transformer and Mamba temporal encoding methods, this model combines two key components: Mixture of Feature Extractors (MoF) and Mixture of Architectures (MoA), which are used to extract short-term features and model long-term dependencies, respectively.

First, in the MoF module (Section 3.3.1), the model divides the trajectory sequence into multiple time patches using a sliding window and employs multiple sub-extractors to adaptively extract features across different patches, thereby capturing dynamic change features over a short time range more accurately. Then, in the MoA module (Section 3.3.2), the model performs hierarchical modeling on the extracted temporal features by fusing multiple structures such as Mamba layers, convolutional layers, self-attention layers, and feed-forward networks, to learn both local dependence relationships and global long-term dependence simultaneously, thus improving the model’s expressive ability for complex trajectory patterns. Through the above design, the temporal module can effectively model the multi-scale temporal dependence of pedestrian trajectories while maintaining computational efficiency and providing richer, more stable temporal feature representations for subsequent trajectory prediction.

Traditional time-series patch-embedding methods (such as linear projection) use the same transformation parameters for all patches, ignoring differences in feature affinity across patches due to semantic context. This homogenization process will lead to the loss of part of the context information and limit the accurate representation of short-term details.

Features from each time-series patch are extracted by MoF, which is an adaptive feature extractor. In MoF, the input time series is divided into multiple patches through a sliding window. For each patch, MoF is processed by a set of Sub-Extractors to produce the patch’s feature representation. The operation of MoF can be expressed as follows:

where X_p is the input time series patch. F_i(X_p) is the i-th sub-extractor, which generates the patch’s feature representation. R_i(X_p) is the routing function, which is used to selectively activate the most suitable sub-extractor to ensure computational efficiency and adaptability.

Although Transformers can model global long-term dependencies, they are computationally intensive; Mamba is computationally efficient, but there is a risk of information loss in long-term prediction. Dependence relationships are modeled step by step by MoA, from local to global, through a hierarchical hybrid architecture that balances efficiency and performance.

MoA is used to capture long-term dependence relationships in time series. As shown in Figure 5, the MoA structure includes four levels: Mamba layer, Feed Forward layer, Convolution layer, and Self-Attention layer, each of which captures different aspects of long-term dependence. The calculation of MoA is as follows:

(1) Mamba layer: First, it selectively processes time-varying dependence relationships as follows:

where σ represents the activation function (usually SiLU), Conv1D represents the one-dimensional convolution operation, and Linear represents the linear transformation.

(2) Self-Attention layer: Then, MoA captures global long-term dependence through the self-attention layer:

where the calculation of the self-attention layer is:

where Q is the query vector, K is the key vector, V is the value vector, and d_k is the dimension of the key.

(3) Convolution layer: It is used to expand the receptive field, so that the dependence relationships between different patches can be learned more comprehensively:

where k is the size of the convolution kernel, s is the step size, p is the padding, and c_out is the number of output channels.

(4) Feed-forward layer: It increases non-linearity, improving the expressive ability of the model:

where w₁ and w₂ are parameters.

Finally, the long-term dependence representation $${X}_{rep}^{\prime}$$ output by the MoA module can be obtained as:

Output of the encoder module: The output of the encoder module is the feature representation $${X}_{rep}^{\prime}$$ generated by combining the MoF and MoA modules, which is fed into the subsequent prediction layer. Finally, the prediction result is obtained by the model through a linear projection:

In the decoding stage, the future multi-modal motion distribution is explicitly modeled by the model through the introduction of intention anchors. Intention anchors are a set of learnable query vectors used to characterize potential different motion patterns; each anchor queries the encoded historical trajectory context during the decoding process, and is updated and refined layer by layer in the multi-layer decoder to obtain a more accurate depiction of the multi-solution future trajectory.

To improve the rationality of anchor initialization, the initial positions of intention anchors are set based on the clustering results of the trajectory distribution in the training data, ensuring they cover the main motion pattern clusters. Subsequently, the pattern representation of specific samples is gradually converged to by the anchors during the decoder’s iterative update. The initial embedding of the anchor is as follows:

where A represents the intention anchor parameters, and PE_A is the positional encoding related to the anchor position, used to inject structured position information.

Iterative Refinement Module (IRM): To refine the representation of trajectory intention anchors layer by layer and enhance the model’s ability to model complex spatiotemporal dependence, an Iterative Refinement Module (IRM) is designed in the decoder. Taking learnable intention anchors as Query, and the historical trajectory context output by the encoder as Key and Value, this module gradually refines the semantic representation of anchors through a multi-layer Transformer structure, realizing dynamic alignment from the global modal prior to the sample conditional modal [31]. The iterative refinement module stacks N-layer structures, and each layer includes three stages: Self-Attention, Cross-Attention, and Feed-Forward transformation. The input of the j layer is the output of the M^j−1 layer, and its update process is as follows:

First, it models the interior of the anchors, enabling interactions between different modalities and capturing potential correlation relationships:

where the multi-head attention mechanism is used to model the dependence structure of different subspaces in parallel, and the output of each head is spliced and linearly transformed to fuse into a global representation. The problem of modal independence between anchors is effectively alleviated, and the complementarity of multi-modal features is improved by this step.

The anchor features after self-attention are used as queries to select the spatiotemporal information most relevant to the current modality from the historical context H, realizing cross-modal feature alignment:

Through this interaction, the model can adaptively extract differentiated evidence from historical trajectories under different intentions, thereby refining the current pedestrian’s intention layer by layer.

The anchor update features $$C_{ca}^{j}$$ are further nonlinearly mapped through the feed-forward network to obtain the output of the j layer:

Moreover, the training process is stabilized through residual connection and layer normalization (Add&Norm). The output of each layer, M^j, serves as the input to the next layer, enabling layer-by-layer semantic refinement and intention alignment. To strengthen multi-layer supervision and training stability, a trajectory regression branch Y^j, is connected after each layer to generate multi-modal future trajectories. The final prediction result is obtained by weighted fusion of the output of each layer:

where the weight coefficient is set as the mean value or learnable parameters to balance the contribution of different layers.

To fully verify the innovation and effectiveness of the proposed MUGI-Net, this chapter designs a systematic experimental scheme on two benchmark datasets. First, the experimental datasets, evaluation metrics, and implementation details are unified to ensure fair comparison. The overall performance of MUGI-Net is then compared with state-of-the-art methods to verify its superiority. Finally, ablation studies are conducted to evaluate the core modules.

The proposed method is evaluated on two benchmarking datasets: JAAD [36] and PIE [37]. The JAAD dataset contains 686 pedestrians collected by vehicle front cameras at 30 Hz; the PIE dataset contains 1842 pedestrians with action annotations, also collected at 30 Hz. Video frames, pixel-level pedestrian coordinates and bounding box positions, and action/motion information for pedestrians and ego-vehicles are provided by both datasets. For pedestrian behavior annotation, both datasets use a consistent coding scheme that divides pedestrian actions into “walking” and “standing”. The annotation of ego-vehicle motion state is slightly different in the two datasets: the ego-vehicle actions in JAAD are given in the form of semantic labels such as “deceleration, acceleration, fast movement, slow movement”, while PIE provides continuous velocity and yaw angle measured by on-board sensors (such as gyroscopes). To achieve consistent modeling across datasets, the ego-vehicle states in JAAD are recoded into four discrete states: 0—stop, 1—slow driving, 2—fast driving, and 3—deceleration. Existing works [21,38] are referred to for data division and sequence settings to ensure fair comparisons: a historical sequence with a length of 0.5 s is used for different modal inputs (i.e., under a 30 Hz sampling rate, the length of the historical sequence is set to T_h = 15 frames), and the prediction horizons are set to 0.5 s, 1.0 s and 1.5 s respectively, corresponding to the prediction sequence lengths of 15, 30, and 45 frames.

To accurately measure the prediction error of pedestrian trajectories in FPV (characterized by bounding box scale changes), three quantitative metrics are adopted based on the displacement between predicted and ground-truth trajectories. Based on established works [21,36,37], our method uses: (1) Bounding Box Mean Square Error (MSE) of the top-left and bottom-right coordinates; (2) Center Mean Square Error (CMSE) of the bounding box; (3) Center Final Mean Square Error (CFMSE) of the center point coordinates of the bounding box, as follows:

where $$\hat{Y}_t$$ and Y_t represent the ground-truth and predicted bounding boxes at time step t, $$\hat{c}_t$$ and c_t represent the corresponding ground-truth and predicted center coordinates.

The model is trained on a single RTX 3060 GPU, using the Adam optimizer with an initial learning rate of 0.001, and the learning rate is adjusted based on the validation set loss with a decay factor of 0.2. The dimension of the hidden layer is set to d_h = 256, and the dimension of the input embedding is set to d₀ = 128. The number of layers in the Transformer encoder is N = 3. The feature dimensions of pedestrian trajectory and pedestrian motion are N_x = 4 and N_p = 1, respectively. The feature dimension of ego-vehicle motion differs between the JAAD and PIE datasets: 1 and 2, respectively. In addition, the number of heads in the attention mechanism is set to h = 8, and the performance is evaluated based on the modality with the best results (K = 20). The training is carried out for 50 epochs, with a batch size of 128 and a dropout rate of 0.1.

To verify the overall performance of MUGI-Net (the integrated effect of all core modules), this section compares the model with 7 state-of-the-art baseline methods (SGNet (2022) [18], Adsampler (2024) [39], AMTN (2024 b1) [39], OS-TF-S (2024) [40], SGNetPose (2025) [41], ABC+ (2023) [42], and AANet (2025) [22]) on JAAD and PIE datasets, with the results shown in Table 2 (bold indicates the best performance).

On the JAAD dataset (1.5 s medium-long term prediction, MSE_15), MUGI-Net achieves 153, a 5% reduction compared with the SOTA method AANet (161), and outperforms all other baselines by a significant margin; in CFMSE (343), the model also surpasses AANet (356), verifying that the integrated design of group interaction and MoU temporal encoding effectively reduces the long-term prediction error accumulation caused by insufficient interaction modeling and temporal dependence capture.

On the PIE dataset (1.0 s medium-term prediction, MSE_10), MUGI-Net attains the optimal value of 33 (on par with ENCORE-NR). In MSE_15 (73), CMSE (52), and CFMSE (140), it is significantly superior to AANet and most baselines, confirming that the model exhibits stable, excellent performance across different datasets. The core modules exhibit strong generalization across FPV scenarios.

Even in short-term prediction (0.5 s, MSE_05), MUGI-Net (34 on JAAD, 16 on PIE) is only slightly lower than the optimal method ENCORE-NR and better than all other baselines, indicating that the model does not sacrifice short-term prediction accuracy while optimizing medium and long-term performance, which is the direct effect of the MoU structure’s balanced capture of short-term dynamics and long-term dependence.

The qualitative evaluation results of our model on the JAAD and PIE datasets are shown in Figure 6 and Figure 7. Our method achieves excellent predictive accuracy, especially in scenarios where pedestrians may exhibit multiple behavior patterns. As shown in Figure 6 and Figure 7, the model generates future trajectories across different urban scenarios and accurately captures diverse motion patterns. In the figures, blue represents the observed trajectory, red represents the ground truth trajectory, and green represents the predicted trajectory. For example, in the middle subgraph, the model successfully predicts different future motions, such as crossing the road or continuing forward. This is the joint effect of the group interaction module (capturing social interaction) and the intention anchor decoder (generating multi-modal, reasonable trajectories), verifying that the model can generate trajectories that conform to real traffic social norms rather than just pursuing numerical error reduction.

To verify the design’s effectiveness, a comprehensive ablation experiment is conducted on the group interaction and temporal encoding modules (i.e., the first-stage encoder). For the part without group interaction, only the historical bounding box is input. For the part without group interaction, the only input is the historical bounding box. Table 3 and Table 4 show the results obtained from the JAAD and PIE datasets. In Table 3, the symbol √ indicates the use of Group Interaction, while the symbol × indicates the absence of Group Interaction.

The impact of group information on pedestrian trajectory prediction is illustrated in Table 3. After adding group information, trajectories are predicted by the model based on distance and direction information among pedestrians. Figure 8 and Figure 9 show the prediction results with and without the group interaction module for the JAAD and PIE datasets, where blue represents the observed trajectory, red represents the ground truth trajectory, green represents the predicted trajectory with the group interaction module, and yellow represents the predicted trajectory without it. The superiority of the prediction results with the group interaction module over those without it is shown by the figures. Therefore, the relevant performance indicators on the JAAD/PIE datasets are significantly improved.

The impact of different temporal encoding methods on pedestrian trajectory prediction performance is illustrated in Table 4. Since only long- or short-term dependence is often focused on by Transformer or Mamba encoders, a decline in prediction performance is caused by using them separately. Figure 10 and Figure 11 show the results of different encoding methods in the JAAD and PIE datasets, where blue represents the observed trajectory, red represents the ground truth trajectory, green represents the predicted trajectory using the MoU encoding method, and yellow represents the predicted trajectory using the Transformer encoding method. The fact that superior predictions are yielded by the MoU encoding method compared to the Transformer encoding method is shown by the figures. Therefore, the relevant performance indicators on the JAAD/PIE datasets are significantly improved. This directly verifies that the MoF (adaptive short-term feature extraction via sliding window and sub-extractors) and MoA (hierarchical fusion of Mamba/Convolution/Self-Attention for global-long term modeling) structure effectively balances the capture of short-term dynamic details and long-term temporal dependence, making up for the inherent defects of Transformer (weak local detail capture, high computation) and Mamba (easily lost information in long-term prediction).

The impact of different numbers of predicted trajectories on pedestrian trajectory prediction performance is reported in Table 5. Since fewer predicted trajectories limit the model’s ability to explore diverse plausible motion patterns and capture the inherent uncertainty of pedestrian behavior, prediction performance declines when the number of predicted trajectories is reduced from 20 to 10 or 5. The table presents results for different numbers of predicted trajectories in the JAAD and PIE datasets, where 20 (Ours) denotes our method with 20 predicted trajectories, and 10/5 denotes baseline settings with fewer trajectories. The table shows that superior performance (lower MSE/CMSE/CFMSE values across all time steps) is achieved with 20 predicted trajectories compared to 10 or 5. Therefore, the relevant performance indicators on the JAAD/PIE datasets are significantly improved when using 20 predicted trajectories. This directly verifies that a larger number of predicted trajectories effectively enhances the model’s capacity to capture multi-modal pedestrian motion patterns and reduce prediction error, making up for the limitations of fewer trajectories in exploring the full range of possible future movements and handling uncertainty in pedestrian behavior.

This work proposes MUGI-Net, a FPV pedestrian trajectory prediction model that fuses group interaction and hybrid temporal encoding, addressing deficiencies in existing methods for group interaction modeling and temporal dependence capture. On the JAAD dataset, the model achieves an MSE_15 of 153 for 1.5 s prediction, a 5% reduction compared with the SOTA AANet (161), and outperforms most baselines in CFMSE. On the PIE dataset, it attains an optimal MSE_10 of 33 for 1.0 s prediction and excels in MSE_15, CMSE, and CFMSE compared with mainstream methods such as AANet and SGNet. Ablation experiments further confirm that the group interaction module reduces JAAD’s MSE_15 by 14.04% and PIE’s by 7.59%, while the MoU temporal structure outperforms single Transformer/Mamba encodings by 7.27% and 3.16% in JAAD’s MSE_15, respectively. These results directly validate the innovation and effectiveness of the proposed group pooling mechanism, MoU hybrid temporal encoding, and intention anchor-based iterative refinement: the group interaction module realizes fine-grained modeling of pedestrian social relationships for FPV scenarios, the MoU structure balances short-term dynamic and long-term dependence capture, and the decoder boosts multi-modal prediction stability. MUGI-Net significantly improves the medium- and long-term prediction accuracy of FPV pedestrian trajectories, provides an effective technical solution for autonomous driving in complex mixed traffic scenarios, and enriches the theoretical and methodological framework of FPV-based pedestrian trajectory prediction, with significant research value for advancing the practical application of autonomous driving and intelligent transportation systems.

MUGI-Net’s superior performance on the JAAD and PIE datasets verifies that group interaction modeling and a MoU-based hybrid temporal encoding are critical for FPV pedestrian trajectory prediction. The group interaction module, via dynamic grouping and sparse intra- and inter-group graphs, effectively captures pedestrian social interactions and mitigates FPV perspective drift, reducing medium- to long-term prediction errors by over 7% across key metrics. The MoU structure balances short-term dynamic capture and long-term dependence, outperforming single Transformer/Mamba encodings by avoiding detail loss and computational inefficiency. Intention anchor-based IRM enhances the rationality of multi-modal prediction, adapting to FPV’s motion randomness. However, the model is limited in severe occlusion scenarios and in complex environmental integration, and its inference speed needs to be optimized for real-time autonomous driving. Future research will focus on multi-sensor fusion, environmental factor modeling, and model lightweighting to further improve its practical application value in autonomous driving systems.

During the preparation of this manuscript, the author(s) used Deepseek for grammer polishing. After using this tool/service, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the content of the published article.

This work is supported by General Project of National Natural Science Foundation of China NO.62576052, General Project of Jiangsu Provincial Department of Science and Technology NO.BK20250969, Changzhou City Applied Basic Project No.CJ20240039, and Changzhou Leading Innovative Talent Introduction and Cultivation Project NO.CQ20250044.

Conceptualization, R.N. and B.Y.; Methodology, R.N.; Software, M.D.; Validation, S.Y. and M.D.; Formal Analysis, S.Y.; Investigation, M.D.; Resources, B.Y.; Data Curation, M.D.; Writing—Original Draft Preparation, R.N.; Writing—Review & Editing, B.Y.; Visualization, M.D.; Supervision, B.Y.; Project Administration, B.Y.; Funding Acquisition, R.N. and B.Y.

Not applicable.

Not applicable.

The research data is based on publicly available datasets on the internet.

This research was funded by General Project of National Natural Science Foundation of China under grant number 62576052; Project of Jiangsu Provincial Department of Science and Technology under grant number BK20250969; Changzhou City Applied Basic Project under grant number KYZ24020273; Changzhou Leading Innovative Talent Introduction and Cultivation Project under grant number CQ20250044.

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.