A Survey on XR-Based Drone Simulation: Technologies, Applications, and Future Directions

2.1. XR Technologies XR is an umbrella term encompassing VR, AR, and Mixed Reality (MR). It is situated along Milgram’s “virtuality continuum” [8] (Figure 1). In a fully immersive VR environment, users interact exclusively within a synthetic space that replaces the real world. By contrast, AR overlays the user’s real-world view with digital elements, typically via see-through displays or camera pass-through, so that virtual objects appear within the physical environment. Importantly, AR systems can enable dynamic interactions between virtual and physical elements, such as a virtual drone colliding with a real wall and responding accordingly. MR represents a more advanced point on the AR spectrum, where virtual and real elements are not only co-present but also interact with each other in real time and in a geometrically coherent manner. In MR, users can simultaneously perceive and manipulate both physical and virtual objects. For instance, an MR drone training system might allow a trainee to operate a physical flight controller and see their own hands, while a virtual drone and wind effects are rendered seamlessly into the scene. Achieving this blending requires robust spatial mapping and tracking so that virtual drones respect physical surfaces (e.g., becoming occluded when passing behind real buildings) and remain correctly anchored in space. As a concrete example of MR in a drone context, Figure 2 reproduces the first-person view of a drone in a mixed-reality environment where real-world video is blended with a virtual forest [9]. It is important to note that MR’s definition varies across the literature: some researchers treat it as a distinct category, while others regard it as part of a spectrum between AR and VR. Regardless of these definitional nuances, MR highlights a form of interaction situated on the continuum—closer to AR but extending toward a richer integration of physical and virtual domains. Compared to VR, which offers complete control over the synthetic environment, AR and MR face unique technical challenges such as dynamic lighting, occlusion handling, and registration accuracy. Nevertheless, MR offers distinct benefits by anchoring training in real-world contexts, thus preserving situational awareness while introducing interactive simulation elements. Each modality along the reality–virtuality continuum serves different training needs, from risk-free full immersion in VR to context-rich MR scenarios. Modern XR drone simulators increasingly combine VR, AR, and MR to maximize realism, adaptability, and training value. 2.2. System Components: Hardware, Interfaces, and Simulation Modes XR-based drone simulators integrate multiple technologies to create immersive and responsive training environments. A typical system comprises three core components: a Head-Mounted Display (HMD) for visual immersion; input devices for control; and simulation software for flight dynamics and environmental modeling. Common HMDs used in XR drone simulation include standalone or tethered devices such as the Meta Quest series, Varjo XR headsets, and HTC Vive. These devices provide stereoscopic rendering, head tracking, and (depending on the model) AR capabilities via pass-through cameras or transparent displays. Control inputs vary based on the intended realism and training goals. Some simulators support the standard radio controllers used in real drone piloting, while others employ gamepads or VR hand controllers mapped to emulate drone flight behavior. High-fidelity simulators allow stick-and-throttle control with yaw, pitch, and roll fidelity, accommodating different UAV types such as multirotors and fixed-wing aircraft. Depending on the display mode, XR simulators can operate using either VR or AR. In VR mode, the pilot is placed in a fully virtual 3D environment, often with realistic terrain, obstacles, and weather effects. In AR mode, virtual elements such as drones and waypoints are superimposed onto the user’s real-world view, using the HMD’s cameras or optical see-through displays. For example, Foxtrot SimFlight XR supports both VR and AR configurations [10]. In VR mode, it presents a complete digital flight ground with variable conditions; in AR mode, it overlays a simulated drone onto the user’s physical surroundings, enabling pilots to train as if a drone were flying in their actual space. In both modes, the system aims to replicate drone behavior and physics with high fidelity, ensuring that virtual UAVs respond realistically to user input and environmental variables such as wind and collisions. 2.3. Simulation Engines and Spatial Computing Capabilities Modern drone simulators—whether XR-based or not—rely heavily on simulation engines to deliver realistic and immersive training experiences. These systems are built on game and physics engines that model aerodynamics, weather, sensor behavior, and collision dynamics. Platforms such as Flightmare [11] and AirSim [12] exemplify research-oriented simulators, offering high-fidelity physics modeling, modular scenario generation, and extensible Application Programming Interfaces (APIs) suitable for both autonomous flight algorithm development and XR integration. In contrast, Real Drone Simulator [13] adopts a more game-centric approach, emphasizing customizable terrains, weather conditions, and skill-based progression for manual pilot training. While it lacks far-reaching extensibility, it serves as an accessible platform for entry-level skill-building and familiarization with diverse flight scenarios in a safe virtual environment. Among the most widely used platforms for building XR-enabled drone simulators are Unity and Unreal Engine. Unity is favored for its flexible cross-platform support and ease of integration with XR Software Development Kits (SDKs), making it well-suited for standalone HMDs like the Meta Quest series. Its component-based design streamlines the development of interactive training interfaces and real-time feedback systems. On the other hand, Unreal Engine, with its high-fidelity rendering and built-in physics systems, is often the foundation for simulation environments requiring photorealism and complex dynamics—particularly in research-oriented projects like AirSim, which uses Unreal Engine as its rendering backend. Both engines support integration with external physics libraries and robotics middleware, enabling the development of highly customizable and scalable training scenarios. In addition, spatial computing capabilities—especially relevant in AR and MR settings—are transforming how simulators interact with the real world. Advanced headsets like the Meta Quest and Varjo XR series feature outward-facing cameras and depth sensors for environment scanning. This allows the virtual drone to recognize and interact with physical objects in the user’s space. For example, SimFlight XR [10] uses passthrough AR and spatial mapping to detect real obstacles such as walls or furniture. Trainees can thus practice avoiding real-world hazards, enhancing realism and situational awareness. This mixed-reality blending ensures that training scenarios feel authentic and responsive, even within confined and indoor environments. 2.4. Physics Engines, Flight Dynamics Modeling, and Realism Validation Accurate flight dynamics modeling is a cornerstone of drone simulation, particularly in XR environments where real-time responsiveness and physical plausibility are essential for pilot immersion. The modeling ensures that virtual UAVs replicate the inertia, thrust, torque, and aerodynamic forces experienced by real-world aircraft, whether multirotors or fixed-wing drones. Modern XR simulators typically rely on integrated physics engines to handle these calculations. For example, Unity and Unreal Engine both incorporate robust physics systems: Unity uses NVIDIA’s PhysX while Unreal integrates its own high-performance physics layer. These engines offer native components such as rigidbodies, colliders, and force application methods, which can be customized to simulate drone-specific dynamics. In the case of SimFlight XR [10], the simulator models flight behavior using Unity’s built-in Rigidbody component in combination with the PhysX engine. The simulator computes lift, gravity, and rotational torque by applying physically accurate forces derived from real drone specifications. Key parameters such as weight, frame size, and maximum velocity are tuned to match those of actual UAV models, resulting in a simulation that reacts to user input with realistic feedback. The physical model also incorporates external disturbances. Wind effects are represented in two primary forms: constant wind, modeled as a steady directional force across the drone’s body with Perlin noise added to replicate natural variability; and gusts, modeled as brief randomized bursts of force generated using functions such as Perlin noise to create motion patterns that are both smooth and unpredictable. These gusts are designed to test a trainee’s control precision and adaptability under dynamically changing conditions. By simulating such disturbances and nuanced aerodynamic interactions, XR simulators not only enhance manual control proficiency but also prepare pilots for real-world challenges such as sudden wind shifts or proximity to obstacles. Although they are simplified when compared to full computational fluid dynamics (CFD) modeling, these physics-based approximations provide a practical compromise between computational efficiency and physical realism. Achieving high physical realism, however, requires rigorous validation and tuning of these flight dynamics models. Developers employ multiple benchmarks to ensure that simulator behavior aligns with empirical UAV performance. One key metric is hover and drift behavior: for example, when a virtual drone hovers or decelerates from forward flight, its tendency to drift or overshoot should mirror that of a real drone under similar conditions. If a simulator’s drone stops instantly with no settling time, it indicates that the model has been oversimplified, which may cause it to teach incorrect muscle memory. To prevent this, simulators are tuned so that dynamics like momentum and inertia (e.g., how far a UAV coasts when throttle is cut) closely match empirical flight data. Another critical criterion is trajectory fidelity. Simulators often replay recorded flight paths from real drones, adjusting the aerodynamic parameters until the virtual trajectory overlaps the real one within acceptable error margins (e.g., position errors within a few percent). Discrepancies in turn rate, climb rate, or response times signal that elements such as thrust curves or drag coefficients require refinement. Wind and gust responses are likewise validated: for instance, a 5 m/s crosswind gust should induce a drift angle and stabilization time in the virtual drone that compare with real flight observations under controlled conditions. Meeting these benchmarks typically requires iterative physics tuning, often involving real-world flight logs and sensor data for calibration. Simulators apply techniques such as system identification: a drone may perform controlled test inputs (e.g., step throttle changes, rapid yaw rotations), and designers adjust the simulator’s responses until they match logged accelerations and velocities from real flights. This process extends to sensor simulation as well. For example, VR camera feeds can be degraded with realistic noise, motion blur, or lens artifacts so that trainees do not develop habits based on a “perfect” visual feed unavailable in real-world operations. Ultimately, a validated physics model is one that has been stress-tested against real flight data across a wide range of scenarios, from stable hover to aggressive maneuvers, ensuring that the XR environment provides both high fidelity and appropriate variability. By defining benchmarks such as drift distance, trajectory error, and gust tolerance, researchers iteratively refine simulators until the virtual drone behaves in ways indistinguishable from its physical counterpart within the regime of interest. This rigorous cycle of modeling, validation, and refinement is essential not only for building pilot trust but also for maximizing skill transfer from XR training to live UAV operations. 2.5. Interaction Fidelity A critical aspect of any flight simulator is how closely the user’s interactions match real-life operations. XR-based systems employ various strategies to maximize interaction fidelity. Many simulators support real drone controllers or use replicas to provide familiar hand-feel and control layouts. For example, the Varjo-Inzpire mixed reality UAV trainer integrates an actual Ground Control Station interface and physical controls with the virtual environment [14]. Trainees can manipulate real sticks, switches, or tablets, which they see rendered in their MR headset exactly where they are in reality, blending tactile feedback with virtual visuals. In less elaborate setups, consumer VR drone apps often allow the use of USB gamepads or dedicated RC transmitter hookups so that muscle memory from simulator practice will transfer directly to real drone transmitters [15]. Haptic feedback is another channel for user instruction: while full-motion platforms are rare for drone simulators (given that drones impart less kinesthetic feedback than manned aircraft), some systems simulate cues like vibrations or resistance. For instance, a hand controller might buzz when the virtual drone crashes or experiences turbulence, alerting the pilot through touch. SimFlight XR demonstrates a scalable interaction approach by utilizing the standard Quest 3 controllers, mapping their left and right joysticks to the corresponding sticks of a traditional RC transmitter (e.g., Mode 2). This eliminates the need for specialized hardware, making the simulator highly accessible and deployable on standalone headsets. To prevent interference between vertical (altitude) and horizontal (yaw) inputs on the left joystick, the system dynamically interprets input based on directional intent. If the joystick is pushed along a clear axis or diagonal, the corresponding input is passed through; otherwise, the dominant axis is prioritized and the lesser axis is suppressed, ensuring clean, intuitive control for new and experienced users alike. XR’s ability to merge the real and virtual also aids interaction fidelity. In AR, trainees can reference physical instruments and maps during simulated missions. For example, Inzpire’s system allows drone operators to see their actual hands and notebooks while flying in the virtual battlespace, thus preserving the workflow of checking a map or jotting coordinates as they would on a real mission [14]. Such seamless integration of real tools addresses a common drawback of pure VR (i.e., the isolation from one’s physical surroundings) and has been noted to improve trainees’ sense of realism and situational awareness. James Clarkson, Engineering Lead at Inzpire, emphasized that using a high-fidelity XR headset achieved “seamless integration between the live and the virtual” and increases trainees’ belief in the scenario: “If you’re immersed in an environment, you forget that you’re wearing a headset. You believe you’re in the field and what you’re doing is part of your environment”. This level of immersion can make simulation training more operationally valid, meaning that pilots treat the exercises seriously and develop habits as if in real operations. 2.6. System Deployment The hardware requirements for XR drone simulators are becoming increasingly accessible. High-end military or aviation setups might use top-tier Varjo XR headsets for ultra-realistic visuals and pass-through (Varjo’s XR-3 can even allow fine text to be read in cockpit instruments via focal plane technology). However, many training solutions today use standalone VR headsets like Meta Quest 2/3, which have built-in inside-out tracking and color pass-through cameras for AR. SimFlight XR, for example, runs on Meta Quest devices and can be deployed wirelessly via an app, making it highly portable (i.e., no external PC or cameras are required). This portability is crucial for scaling training programs, as an instructor can bring a set of headsets to a classroom or a field site and start a training session within minutes. The software is typically distributed through enterprise app stores or sideloaded, with cloud connectivity used for multi-user scenarios or logging. Some B2B solutions (like Foxtrot’s) include cloud-based flight log analytics and progress tracking: the simulator records telemetry and controller inputs during each session, enabling post-training debriefs and analysis of pilot performance trends. Such data can highlight, for example, if a trainee consistently struggles with nose-in orientations or tends to fly too low. This information is invaluable for targeted coaching. Edge computing has also emerged as a complementary deployment option. Whereas cloud services enable centralized logging and multi-user connectivity, edge nodes—for example, a local mini-PC or an on-site server—can offload parts of the simulation workload and reduce latency for interactive XR tasks. This is particularly useful in field training scenarios where network bandwidth is limited or unreliable. In summary, XR-based drone simulators fuse advanced 3D simulation software with immersive display and interaction devices to create convincing flight training experiences. VR provides total immersion enabling practice in any imaginable environment (from basic fields to warzones), whereas AR/MR makes it possible to blend the practice drone into real settings and equipment. Table 1 compares several notable XR drone simulation platforms and prototypes, illustrating the range of technologies and features in use. 2.7. Simulator Realism and Fidelity XR-based simulators vary in their level of realism, but many strive for high fidelity in both visuals and physics. Graphically, modern simulators use detailed 3D models of drones and environments; some incorporate photogrammetry-based maps or geospatial data for authentic landscapes [15]. For example, industrial training sims may recreate specific work sites—like oil refineries or farms—so that pilots can practice virtually in a twin of their actual operating environment. Lighting and weather effects (sun glare, fog, wind gusts, etc.) add to visual realism and can be toggled for training under different conditions. On the physics side, realistic flight dynamics are crucial: accurate aerodynamics (lift, drag), battery/performance models, and even sensor noise models (for GPS or altimeter) must all mimic real drone behavior. Some simulators allow users to switch between flight modes like GPS-stabilized vs. ATTI (manual attitude) to train pilots on handling loss of GPS, or high precision flying. SimFlight XR, for instance, supports both modes: in GPS mode, the virtual drone self-stabilizes, whereas in ATTI mode it drifts in wind. The simulator thereby teaches students to manage different control scenarios [10]. Similarly, simulators often include multiple drone models (quadcopter, fixed-wing, etc.), each possessing distinct physics. Zephyr, a training simulator used in schools and agencies, meticulously models a range of popular drone types so that each virtual aircraft “feels” like the real one in terms of mass and responsiveness [15]. The ultimate goal is that when a trainee transitions from the XR simulator to a real drone, the experience is familiar, with control sensitivities, vehicle dynamics, and even visual cues all matching reality closely, shortening the learning curve and improving safety in initial real flights. 2.8. Open Datasets and Tools for XR Drone Simulation To accelerate research and development, the community has made several datasets, simulation frameworks, and scenario libraries publicly available. One foundational tool is Microsoft AirSim (open-sourced in 2017) [12], a photorealistic drone and vehicle simulator built on Unreal Engine. AirSim provides several ready-to-use environments (e.g., urban city blocks, forests, etc.) and an API for retrieving sensor data, enabling researchers to generate custom flight scenarios and collect synthetic training data for computer vision or control algorithms (Figure 3). Another influential platform is Flightmare (UZH, 2020), an open-source quadrotor simulator that couples the Unity graphics engine with a high-fidelity physics engine [11]. Frameworks like Flightmare, RotorS (for ROS/Gazebo), and MIT’s FlightGoggles allow investigators to test new control strategies or reinforcement learning policies in simulation with relative ease, often including example scenarios or baseline models. Examples include RotorS, which offers a library of multirotor models and environments, and the gym-pybullet-drones toolkit, which provides lightweight simulation and benchmark tasks for autonomous drone control. In addition to simulators, there are curated datasets useful for XR drone research. The EuRoC Micro Aerial Vehicle dataset (ETH Zurich) [20] and the Blackbird UAV dataset (MIT) offer real-world flight recordings (including stereo camera feeds, Inertial Measurement Unit (IMU) readings, and ground-truth trajectories) that serve as benchmarks for validating simulation realism and developing AR tracking or Simultaneous Localization and Mapping (SLAM) algorithms. These datasets (with permanent DOIs and documentation) allow XR simulations to be configured under conditions identical to real flights, facilitating direct comparisons. Some repositories host libraries of 3D assets and drone-specific scenarios; for example, DroneRaceX contains standardized race gate models and course layouts for training or competition. Industry and government initiatives have also released reference models: for example, the U.S. Federal Aviation Authority (FAA) has published sample XR Flight Training Device configurations and scenarios for alignment with training standards. In summary, the XR drone simulation community benefits from a growing ecosystem of open tools: from high-fidelity simulators and physics engines, through public datasets of flight data, to shareable scenario and asset libraries. These resources, which are often accompanied by persistent identifiers or GitHub links for reproducibility, greatly aid researchers and practitioners in building upon each other’s work, ensuring that new advances in XR drone training can be compared against common benchmarks and rapidly integrated into improved simulation systems.