One Operator, Many Robots: UAV–UGV teams for search and rescue

September 2026 · Accepted at IEEE SSRR 2026

Imagine a rescue mission in a large, unstructured area: you need to cover ground fast, find anyone who might be lying there, and get help to them quickly. Now imagine doing all of that with a single person giving voice commands to a whole team of drones and one ground robot — no joysticks, no programming, no control room full of specialists.

That's the vision behind our latest work, One Operator, Many Robots: Natural-Language Command of UAV–UGV Teams for Search, Victim Detection, and Rescue, accepted at IEEE SSRR 2026. It builds directly on Swarm-Steward and takes it one crucial step further: connecting what the drones see with what the ground robot does.

Figure 1 shows the end-to-end workflow demonstrated in the field: (a) natural-language mission control, (b) multi-UAV aerial search and victim localization, and (c) UGV deployment toward the confirmed victim location.

Figure 1 — End-to-end SAR workflow demonstrated in the field: (a) natural-language mission control with *Swarm-Steward*, (b) multi-UAV aerial search and victim localization, and (c) UGV deployment toward the confirmed victim location.

From aerial search to ground response

Search-and-rescue operations require rapid area coverage, reliable victim localization, and fast deployment of response assets — but connecting aerial perception with ground-robot action remains difficult for non-expert operators. During autonomous lawnmower-style search, DJI Mini 4 Pro UAVs stream video and telemetry to a perception pipeline that applies a YOLO26l aerial person detector, tracks detections over time, and projects candidate victims to ground coordinates using UAV pose, gimbal state, camera geometry, and terrain elevation.

Figure 2 shows the full victim-detection and cross-UAV association pipeline: detections in sampled frames are geolocated from telemetry and terrain elevation, accumulated into per-UAV tracks, associated across UAVs by the Object Manager, and surfaced to the operator as candidate objects that can be confirmed, rejected, or allowed to expire.

Victim-detection and cross-UAV association pipeline from UAV video to confirmed candidates — Figure 2 — Victim-detection and cross-UAV association pipeline. Detections in sampled frames are geolocated from telemetry and terrain elevation, then accumulated into per-UAV tracks. Persistent tracks are associated across UAVs by the *Object Manager* and surfaced to the operator as candidate objects that can be confirmed, rejected, or allowed to expire.

Accepted tracks are surfaced on the mission map as dynamic entities that the operator can confirm or reject — and once confirmed, they can be referenced in follow-up natural-language commands, for example to send a UAV to monitor the victim or to dispatch the UGV toward a nearby rescue position. Figure 3 shows the extended architecture: the UI layer supports natural-language commands, previews, and confirmations; the LLM-MAS layer coordinates agents and deterministic managers; the bridge layer performs protocol translation, video decoding, detection, and data batching; and the platform layer contains the UAV and UGV robots.

Extended Swarm-Steward architecture: UI, LLM-MAS, bridge and platform layers — Figure 3 — Extended *Swarm-Steward* architecture for aerial search and ground response. The UI layer supports natural-language commands, previews, and confirmations; the LLM-MAS layer coordinates agents and deterministic managers; the bridge layer performs protocol translation, video decoding, detection, and data batching; and the platform layer contains the UAV and UGV robots.

Training the aerial person detector

Because aerial person detection at these altitudes is different from ground-level detection, we trained a custom YOLO26l model on a merged set of four aerial-person datasets — SARD, HERIDAL, LaDD, and NOMAD — summarized in Table 1.

Dataset	Orig. imgs	Used	Val. imgs	Selection / preprocessing
SARD	1,183	1,000	200	Person-like classes collapsed to person; largest avg bbox area preferred
HERIDAL	1,546	~985	197	CSV labels converted to YOLO; largest avg bbox area preferred
LaDD	1,220	1,000	200	COCO labels converted to YOLO; largest avg bbox area preferred
NOMAD	20,752	1,000	200	Altitude-stratified 10/30/50 m sampling with round-robin actor selection

Table 1 — Training data used for the AerialPerson-Mix detector.

Four aerial-person datasets merged and relabeled to a common person class (~4,000 training images).

Table 2 shows the retrospective model sweep on the balanced validation split. YOLO26l reached mAP50 = 0.832 and mAP50–95 = 0.501, the best of the whole sweep while avoiding the larger checkpoint of YOLO26x — see Table 2. Inference runs on the ground station (RTX 5090), so throughput was never the bottleneck in the field.

Model	mAP50	mAP50–95	Inference (ms/image)	Checkpoint (MiB)
YOLO26n	0.774	0.412	1.91	5.1
YOLO26s	0.823	0.467	2.04	19.4
YOLO26m	0.819	0.487	3.92	42.0
YOLO26l (chosen)	0.832	0.501	4.48	50.5
YOLO26x	0.814	0.500	8.95	112.8

Table 2 — YOLO26 model sweep: best-epoch validation mAP, inference time, and checkpoint size.

YOLO26l achieved the best mAP50–95 (0.501) while avoiding the larger checkpoint of YOLO26x. Inference time is the image-weighted mean over the source-validation runs on the ground-station laptop (NVIDIA RTX 5090), excluding preprocessing, video transport, and tracking.

The camera-ready version adds two detector analyses: how the combined model compares against single-dataset specialists, and where the detector's precision–recall operating points land for each model size.

YOLO26l mAP50-95 per validation source: combined model vs single-source specialists — Figure 2b — YOLO26l mAP50–95 per validation source and on the pooled set (Mixed), comparing the model fine-tuned on all four datasets against single-source specialists (outlines mark each specialist’s own source). On the pooled 797-image set the combined model reaches mAP50–95 = 0.502 versus 0.071–0.235 for the specialists, and even outscores each specialist on its own source for HERIDAL, LADD, and NOMAD.

Precision-recall curves of the five YOLO26 sizes on the validation images — Figure 2c — Precision–recall curves of the five YOLO26 sizes on the validation images. Markers show maximum-F₂, maximum-F₁, and P≥0.90 cutoffs; the diamond marks YOLO26l at the 0.60 track gate. A recall-heavy threshold of 0.16 maximizes F₂ (P=0.758, R=0.836) — we favor recall because the operator confirms every candidate: a false alarm costs review time, whereas a missed person is never reviewed.

Twelve outdoor experiments

We evaluated the system in 12 outdoor experiments over an approximately 48,000 m² unstructured area, covering four victim placements, one-, two-, and four-UAV teams, different gimbal angles, and search speeds. Figure 4 shows the search area, the planned lawnmower sweeps, and drone-view examples of the four victim placements (a–d), with the full search area and sweep plan below.

Location A — Figure 4 — Search area, planned lawnmower sweeps, and drone-view examples of the four victim placements used in the real-world experiments. The area spans approximately 199 m × 238 m, corresponding to roughly 48,000 m² (4.8 ha). One UAV flies all sweeps, two UAVs split the area into two groups, and four UAVs each take one sweep region (colored paths S1–4 denote different sweep regions). Locations A, B, and D produced successful localizations within 10 m; location C was the difficult placement where the available runs did not localize the victim within the success threshold.

Location B — Figure 4 — Search area, planned lawnmower sweeps, and drone-view examples of the four victim placements used in the real-world experiments. The area spans approximately 199 m × 238 m, corresponding to roughly 48,000 m² (4.8 ha). One UAV flies all sweeps, two UAVs split the area into two groups, and four UAVs each take one sweep region (colored paths S1–4 denote different sweep regions). Locations A, B, and D produced successful localizations within 10 m; location C was the difficult placement where the available runs did not localize the victim within the success threshold.

Three placements produced successful detections with best mean localization errors of 1.21 m, 2.29 m, and 1.92 m; the fourth placement (location C) remained unresolved — the body position and background were challenging even for the human eye, and the runs lacked a reliable reference for localization scoring. Table 3 reports the representative outcomes per scenario.

Scenario (Loc./N)	Speed (m/s)	Gimbal	Detector	Alt. (m)	Track det.	Mean err. (m)	Mean conf.
A / 1	2	−90°	mini4	35	18	1.21 ± 0.58	0.67 ± 0.18
A / 2	2	−90°	mini2	40	50	1.99 ± 0.78	0.52 ± 0.12
A / 4	2	−90°	mini3	45	15	2.41 ± 0.51	0.73 ± 0.11
B / 1	4	−45°	mini4	35	38	8.80 ± 1.46	0.74 ± 0.14
B / 2	4	−45°	mini2	40	10	7.09 ± 1.25	0.60 ± 0.14
B / 4	2	−90°	mini2	40	34	2.29 ± 0.29	0.67 ± 0.14
C / 4	4	−90°	—	—	—	—	—
C / 4	4	−45°	—	—	—	—	—
D / 1	4	−45°	mini1	35	17	9.33 ± 0.56	0.66 ± 0.13
D / 2	4	−45°	mini2	40	12	3.13 ± 0.76	0.65 ± 0.16
D / 4	4	−45°	mini3	45	9	1.92 ± 0.06	0.47 ± 0.14
D / 1	4	−90°	mini1	80	47	6.37 ± 2.85	0.66 ± 0.13

Table 3 — Representative search-experiment localization outcomes. Scenario is reported as Loc./N, where Loc. is the victim placement and N is the number of UAVs. Dashes indicate not-found cases where visual inspection confirmed that the detector did not observe the victim. The 80 m high-altitude run is listed last.

Locations A, B, D localized within 10 m; location C (difficult placement) was not localized. A / 1 and B / 1 at 35 m used mini4; D / 1 at 80 m used mini1.

From detection to rescue response

Once the operator confirms a candidate, the victim becomes a shared mission target that can be referenced in follow-up natural-language commands — for example, commanding a UAV to orbit the victim for continued monitoring while the UGV is dispatched to a position 5 m north to assist with evacuation without driving directly over the person. Figure 5 shows this post-detection response.

UGV Rover J8 driving toward a confirmed victim location — Figure 5 — Post-detection response after operator confirmation. After *mini2* detects the candidate *person_117* and the operator confirms it, the victim becomes a shared mission target that can be referenced in follow-up natural-language commands. In this example, *UAV 1* is commanded to orbit the victim for continued monitoring while *mini2* continues searching the area, and the UGV *Rover J8* is sent to a position 5 m north of the victim to assist with evacuation without driving directly over the person.

Feedback from a professional firefighter

A professional firefighter from Benalmádena (Málaga) joined the field demonstration and gave direct operational feedback, supporting the relevance of sector-based search, persistent no-go areas, human confirmation, and emergency pause and return commands that bypass the language model for SAR-oriented multi-robot deployment.

One Operator, Many Robots: Natural-Language Command of UAV–UGV Teams for Search, Victim Detection, and Rescue
Jarabo-Peñas, Bravo-Arrabal, Rolland, Christensen · Accepted at IEEE SSRR 2026

One Operator, Many RobotsSSRR 2026 · AcceptedRelated post