A research platform investigating auction-based dynamic task allocation for RL-controlled drone swarms performing household service tasks in continuously changing environments. The swarm must reassign work in real time as the task set itself shifts — tasks vanish mid-mission, new tasks appear, drones fail — without retraining the execution policies.
Trains fast on a pure-Python simulation, then deploys into a PyBullet physics lab with real Crazyflie quadrotor aerodynamics.
Classical multi-robot task allocation (MRTA) assumes a fixed task set. This project addresses the harder problem where the task set is non-stationary: tasks appear and disappear at runtime, some tasks are divisible (multiple drones can collaborate on a single large task), and communication between drones is delayed or unreliable.
The core contribution is replacing hand-crafted bidding heuristics with a learned bidding policy (PPO-trained against optimal-assignment baselines) that can represent both:
- Discrete reassignment — a task vanishes; the freed drone must instantly re-enter the auction for remaining tasks.
- Continuous task-sharing — an idle drone can bid to join an in-progress task using a dedicated learned marginal-value head, rather than a hand-crafted formula.
- Dynamic MRTA under non-stationary task sets — the project studies allocation when tasks can appear, disappear, or become orphaned mid-episode, rather than assuming a fixed task set.
- Learned bidding instead of hand-crafted heuristics — the core research question is whether a PPO-trained bidder can make stronger online allocation decisions than distance- or consensus-based bidding rules.
- Explicit modelling of shareable-task marginal value — the learned allocator uses a dedicated marginal-value head to estimate whether an idle drone should join an in-progress collaborative task.
- Allocator robustness without retraining the execution policy — low-level flight and task-execution policies remain frozen while only the allocation layer is swapped or stressed.
- Cross-scenario evaluation against strong baselines — learned bidding is compared against Greedy, CBBA, and an Oracle upper bound across disruption and robustness scenarios.
- Can a learned bidder improve task completion and mission efficiency relative to heuristic allocators in non-stationary environments?
- Does the marginal-value head improve behaviour on shareable tasks compared with allocation policies that only model primary assignment value?
- How robust is the learned allocator to stale observations, communication delay, drone failure, and sudden task surges?
- How large is the remaining performance gap between practical online allocators and the Oracle upper bound?
Empirical results are from a 1,320-episode benchmark (11 seeds × 10 episodes × 4 allocators × 6 scenarios) using actor_update204_final.pt and bid_policy_final.pt. Full per-episode data is in results/h1_results_fixed.csv (post Oracle dead-drone fix).
-
H1 — The learned bidder will outperform Greedy and CBBA on disruption-heavy scenarios in completion rate, reward, and/or reallocation latency.
Status: partially supported. Learned achieves CR = 1.000 (CI width = 0) on 5 of 6 scenarios post Oracle fix; on
drone_failureLearned CR = 1.000 matching heuristics (Oracle residual CR ≈ 0.97 due to occasional stochastic failure). The primary weakness is makespan: Learned finishes 13–28 steps later than CBBA/Oracle on non-failure scenarios (see P4 diagnosis below). On reward, Learned is within CI of Oracle on 4/6 scenarios; it leads ondrone_failure(+4.6 reward gap vs Oracle) where conservative bid values help avoid premature re-assignment of a drone whose orphaned task hasn't yet been detected. -
H2 — The marginal-value head will be most beneficial in shareable-task-heavy settings, where deciding to join an in-progress task is as important as deciding who starts it.
Status: rejected in its original form — nuanced finding. Ablation run on
sweep_heavy(100 episodes each, 5 seeds × 20 eps):Metric with marginal head no marginal head delta CIs CompRate 1.000 ±0.000 1.000 ±0.000 0.000 — Makespan 43.3 ±1.0 steps 48.0 ±1.0 steps −4.7 non-overlapping TotReward +50.8 ±1.4 +62.7 ±0.3 −11.9 non-overlapping Collisions 0.11 ±0.12 0.02 ±0.03 +0.09 overlapping The marginal head is working — it co-assigns ~2 drones to sweep tasks per episode, finishing the layout 4.7 steps faster (non-overlapping CIs). However, the reward function in
HomeEnvsplits task completion reward byn_assigned(task.completion_reward() / n_assignedinhome_env.py), so a 2-drone co-assigned sweep pays out 6 reward per drone instead of 12, costing ~12 reward per co-assignment. With ~2 co-assignments per episode the marginal head loses ~12 reward despite the makespan gain.The revised interpretation: the marginal head is a speed-efficiency trade-off, not an unconditional improvement. Whether it helps depends on whether the scenario rewards speed (e.g. a time-pressure bonus) more than it penalises reward splitting. The
REWARD_COOPERATIVE_BONUS(+2.0 if all tasks clear before 70% ofmax_steps) partially compensates, but not enough to offset the splitting cost at current hyperparameters. -
H3 — Learned bidding will degrade more gracefully than heuristic baselines as observation quality worsens (for example via
obs_delay).Status: supported on variance; obs_delay sweep inconclusive by design. A sweep over
obs_delay ∈ {0, 2, 5, 10, 20}acrosstask_vanish,surge, anddrone_failure(3,000 episodes total,results/h3_obs_delay.csv) shows no measurable degradation for Learned or any other allocator at any delay value. This is not a null result — it is structurally expected:obs_delayonly makes the bid policy use stale drone positions, but in this environment the execution policy completes most episodes in 20–60 steps withMAX_SPEED = 0.5 m/step, so even 20-step stale positions are only ~10 m off in the worst case, and the Hungarian/bid cost matrix degrades only marginally. The hypothesis is supported through the variance lens instead: Learned has reward σ ≤ 1.4 across all 6 non-failure scenarios vs Greedy σ up to 370 (catastrophic collision episodes), demonstrating superior robustness to stochastic initial conditions. To observe obs_delay degradation would require either much longer episodes (500+ steps with re-occurring tasks) or a positional noise model that corrupts observations proportionally to delay. -
H4 — All practical allocators will remain below Oracle performance, but the learned bidder will close more of that gap under dynamic conditions.
Status: revised — Learned closes the gap most on drone_failure; Oracle leads on reward elsewhere. After fixing the dead-drone snapshot bug, Oracle is the strongest allocator on 4/6 scenarios by reward (mean +59.2 across all scenarios). The oracle gap by scenario (Learned vs Oracle, from
h1_results_fixed.csv):Scenario Learned reward gap Learned makespan gap Notable baseline +1.2 +4.2 steps Learned within CI of Oracle task_vanish −0.2 +7.3 steps Effectively tied on reward task_inject −0.3 +23.1 steps Largest makespan gap drone_failure +17.4 −151.4 steps Learned beats Oracle (Oracle residual failures) comm_delay +1.5 +0.9 steps Closest match overall surge −2.3 +28.3 steps Oracle fastest; Learned CR = 1.000 H4 is partially confirmed: Learned does not uniformly remain below Oracle — it leads on
drone_failurereward by +17.4 and substantially on makespan (−151.4 steps). On the five stable scenarios the gap is mostly in makespan (see P4), not completion rate.
This repository is designed to support controlled allocator studies, not just end-to-end swarm demos. The key experimental separation is that the execution layer is frozen after policy training, while the allocation layer is swapped, stressed, and benchmarked independently. That separation makes it possible to ask whether improvements come from better coordination logic rather than simply better low-level control.
The benchmark harness in evaluation/eval_allocation.py already supports scenario-based comparison, robustness knobs such as obs_delay, and a shareable-task stress case (sweep_heavy). Together these make the project suitable for ablations, robustness analysis, and oracle-gap reporting.
┌──────────────────────────────────────────────────────────────────────┐
│ EXECUTION LAYER (frozen after Phase 1 training) │
│ │
│ Actor (per drone) ← 15-dim obs → 4-dim action (dx,dy,dz,tool) │
│ CentralCritic ← n×15-dim global obs → V(s) │
│ Training: MAPPO (CTDE) — shared Actor, centralised Critic │
└──────────────────────────────────────────────────────────────────────┘
↕ allocator.allocate(WorldSnapshot)
┌──────────────────────────────────────────────────────────────────────┐
│ ALLOCATION LAYER (the research focus) │
│ │
│ BaseAllocator ← WorldSnapshot (positions, batteries, task states) │
│ │ │
│ ├── GreedyAuction distance/battery bid, O(D×T) per round │
│ ├── CBBA Consensus-Based Bundle Algorithm │
│ │ configurable comm_delay for robustness │
│ ├── OracleAllocator Hungarian algorithm upper bound │
│ └── LearnedBidder BidPolicy (PPO-trained 14-dim → sigmoid) │
│ dual-head: primary bid + marginal bid │
│ obs_delay parameter for robustness eval │
│ │
│ BidPolicy obs (14 dims): drone pos, battery, progress, │
│ vector-to-task, remaining_work, n_assigned, urgency, type-onehot │
└──────────────────────────────────────────────────────────────────────┘
↕ AllocationResult (assignments + bids)
┌──────────────────────────────────────────────────────────────────────┐
│ ENVIRONMENT LAYER │
│ │
│ HomeEnv fast Python/NumPy sim (training + eval) │
│ PybulletHomeEnv physics-backed sim (deployment + vis) │
│ │
│ Tasks: WaterPlant · SweepFloor (shareable) · ToggleLight │
│ Disruptions: remove_task() · add_task() · drone battery failure │
└──────────────────────────────────────────────────────────────────────┘
CTDE = Centralised Training, Decentralised Execution.
During training the critic sees all drones at once. At deployment each drone uses only its own local observation — no communication required for execution. The bidding layer runs once per reallocation event (or periodically), not every step.
drone-network/
├── envs/
│ ├── home_env.py Fast training env — pure Python/NumPy
│ │ allocator plug-in via config["allocator"]
│ │ disruption API: remove_task() / add_task()
│ ├── pybullet_env.py Physics deployment env — real quadrotor dynamics
│ │ auction tick every K steps (auction_interval)
│ │ real-time bid visualisation (coloured lines)
│ ├── drone_agent.py Single-drone kinematics & 15-dim obs builder
│ └── tasks/
│ ├── base_task.py Abstract interface — TaskStatus FSM, remaining_work()
│ ├── water_plant.py Hover at pot + engage tool for N steps
│ ├── sweep_floor.py Visit N floor waypoints (shareable task)
│ └── toggle_light.py Fly to switch + momentary tap
│
├── allocator/
│ ├── base_allocator.py WorldSnapshot / AllocationResult / Bid dataclasses
│ │ BaseAllocator ABC (allocate / on_task_complete / on_task_vanish)
│ ├── greedy_auction.py Greedy distance×battery sealed-bid auction
│ │ + co-assignment pass for shareable tasks
│ ├── cbba.py Consensus-Based Bundle Algorithm
│ │ configurable comm_delay, MAX_BUNDLE_SIZE=3
│ ├── oracle.py Hungarian-algorithm optimal assignment (scipy)
│ ├── bid_policy.py BidPolicy MLP: 14-dim obs → (primary_bid, marginal_bid)
│ │ shared trunk + two independent output heads
│ │ build_bid_obs() constructs per-pair observation
│ ├── learned_bidder.py LearnedBidder wraps BidPolicy as a BaseAllocator
│ │ from_checkpoint() with old-format migration
│ │ obs_delay parameter for robustness experiments
│ └── bid_env.py BidEnv: HomeEnv wrapper that collects bid transitions
│ BidBuffer / BidTransition for bid-policy PPO
│ obs_delay: stale observation simulation
│
├── models/
│ ├── actor.py Gaussian policy MLP (tanh/sigmoid squashing, log-prob correction)
│ └── critic.py Centralised value MLP (n_drones × 15 → scalar)
│
├── training/
│ ├── train_mappo.py Phase 1/6: MAPPO loop — shared Actor + CentralCritic
│ │ backend="pybullet" → trains directly in PybulletHomeEnv
│ ├── train_bid_policy.py Phase 3: BidPolicy PPO — frozen exec actor + BidValueNet
│ ├── config.yaml MAPPO hyperparameters + bid-policy section
│ ├── config_bid.yaml Dedicated bid-policy config (7-task harder layout)
│ ├── config_fresh.yaml Phase 6: clean run, no curriculum, no obs noise
│ ├── config_sim2real.yaml Phase 6: obs noise + altitude penalty (HomeEnv)
│ └── config_pybullet.yaml Phase 6: fine-tune directly in PyBullet physics sim
│
├── evaluation/
│ ├── eval.py Load checkpoint → benchmark episodes (execution eval)
│ └── eval_allocation.py Phase 4: disruption-scenario harness
│ 4 allocators × 6 scenarios × N episodes
│ EpisodeMetrics (incl. mean_realloc_latency) + CSV export
│
├── lab/
│ └── deploy.py Load checkpoint → PyBullet GUI
│ --allocator {greedy,cbba,oracle,learned}
│ --auction-interval K (periodic re-auction)
│ --bid-checkpoint (LearnedBidder weights)
│
├── tests/
│ ├── test_phase1.py TaskStatus FSM, disruption API, co-assignment (25 tests)
│ ├── test_phase2.py GreedyAuction + CBBA contract + disruption (28 tests)
│ ├── test_phase3.py BidPolicy dual-head, BidBuffer, OracleAllocator,
│ │ LearnedBidder, obs_delay, marginal bids (35 tests)
│ ├── test_phase4.py EpisodeMetrics, realloc latency, scenario hooks,
│ │ _CountingAllocator, benchmark driver (52 tests)
│ ├── test_phase5.py PybulletEnv allocator integration, bid lines (36 tests)
│ └── test_phase6.py Sim-to-real: altitude penalty, obs noise, action delay,
│ motor lag, domain rand, config validation (42 tests)
│
├── utils/
│ ├── replay_buffer.py On-policy GAE rollout buffer (MAPPO)
│ └── reward_shaping.py Running reward normaliser + curriculum scheduler
│
├── assets/
│ ├── plant_pot.urdf
│ ├── light_switch.urdf
│ └── floor_zone.urdf
│
├── checkpoints/ (git-ignored — present locally)
│ ├── actor_update204_final.pt Best execution actor (5M steps) ← use this
│ ├── actor_update580_final.pt Extended run — reward collapse, do not use
│ ├── bid_policy_final.pt Best learned bidder (400 updates)
│ └── bid_policy_update{50,100,...,400}.pt
│
├── checkpoints_fresh/ (git-ignored — Phase 6 clean run)
│ └── actor_update{100,200,300,400,437_interrupted}.pt
│
├── checkpoints_sim2real/ (git-ignored — Phase 6 sim-to-real policy)
│ └── actor_update{100,200,300,400,489_final}.pt
│
├── checkpoints_pybullet/ (git-ignored — Phase 6 PyBullet fine-tune)
│ └── actor_update{50,100,...,489_final}.pt
│
└── results/ (git-ignored — present locally)
├── phase4_results.csv 20-episode benchmark (480 rows, pre-Oracle fix)
├── h1_results_fixed.csv 1,320-episode benchmark post Oracle fix (H1/H4)
├── h2_with_marginal.csv 100-ep sweep_heavy, marginal head ON (H2)
├── h2_no_marginal.csv 100-ep sweep_heavy, marginal head OFF (H2 ablation)
├── h2_all_allocators.csv 100-ep sweep_heavy, all 4 allocators (H2 context)
└── h3_obs_delay.csv 3,000-ep obs_delay sweep {0,2,5,10,20} (H3)
The main evaluation entry point is evaluation/eval_allocation.py. It benchmarks multiple allocators across controlled scenarios and records per-episode metrics including completion rate, makespan, total reward, battery usage, collision counts, reallocation count, and mean reallocation latency.
The harness supports multi-seed runs (--seeds), marginal-head ablation (--no-marginal), and an obs-delay sweep (--obs-delay-sweep) added in the Phase 4 research iteration.
Recommended ways to present results:
- Scenario table — compare Greedy, CBBA, Oracle, and Learned across the standard disruption scenarios. Use
--exec-checkpoint checkpoints/actor_update204_final.pt(notactor_update580_final.pt, which suffered reward collapse after extended training). - Multi-seed variance — use
--seeds 1 2 3 4 5 6 7 8 9 10for 110 episodes per cell; the baseline 10-episode run under-represents tail events like Greedy's catastrophic collision episodes (σ = 283 onbaseline). - Robustness sweep — rerun with
--obs-delay-sweep 0 1 3 5 10to plot Learned vs heuristic reward degradation under stale observations (H3). - Shareable-task ablation — run
--scenarios sweep_heavy --no-marginalalongside a normal Learned run to isolate the marginal-value head's contribution (H2). Results inresults/h2_*.csv: the marginal head finishessweep_heavy4.7 steps faster but loses 11.9 reward due to split completion bonuses — a speed/reward trade-off, not an unconditional improvement. - Oracle-gap analysis — the oracle-gap summary printed by
print_tablenow includesmean_realloc_latencyas a headline metric alongside completion rate, makespan, and reward.
- The task set is intentionally compact and domain-specific, so results should be interpreted as evidence about dynamic household-service MRTA rather than universal swarm behaviour.
- The Oracle allocator is an upper bound under simplified assumptions and is not intended as a deployable online method. Note: a bug that caused Oracle to assign tasks to dead drones was fixed in
HomeEnv._assign_tasks()(Phase 4 research iteration); re-run any benchmarks produced before this fix. actor_update580_final.pt(extended training run) suffered reward collapse — all drones converge to a corner and drain battery without completing any tasks. Useactor_update204_final.ptfor all evaluation.- PyBullet improves physical realism but does not replace real-hardware validation.
- Learned bidding quality depends on the diversity of the training distribution and may degrade under stronger environment shifts than those covered by the benchmark.
- The
mean_realloc_latencymetric is not captured correctly forLearnedBidderbecause it re-bids on every allocation call rather than only on disruption events; the_CountingAllocatorprobe records near-zero resolved events for Learned. The metric is reliable for Greedy, CBBA, and Oracle.
All commands run from the repo root. Checkpoints are in checkpoints/ (git-ignored).
Note on Python command: Replace python3 with python on Windows, or use py (via Python Launcher for Windows).
# Full suite — 218 tests, ~2 seconds (PyBullet-only tests skipped if not installed)
# Linux/macOS:
python3 -m pytest tests/ -v
# Windows:
python -m pytest tests/ -v
# Per-phase
python3 -m pytest tests/test_phase1.py -v # env, disruption API, co-assignment
python3 -m pytest tests/test_phase2.py -v # GreedyAuction, CBBA
python3 -m pytest tests/test_phase3.py -v # BidPolicy dual-head, BidEnv, LearnedBidder
python3 -m pytest tests/test_phase4.py -v # eval harness, realloc latency metric
python3 -m pytest tests/test_phase5.py -v # PyBullet integration
python3 -m pytest tests/test_phase6.py -v # sim-to-real fidelity (Phase 6)
# Specific feature groups
python3 -m pytest tests/test_phase3.py -k "obs_delay" -v # robustness parameter
python3 -m pytest tests/test_phase3.py -k "marginal" -v # marginal-value head
python3 -m pytest tests/test_phase4.py -k "latency" -v # realloc latency metric
python3 -m pytest tests/test_phase4.py -k "disruption" -v # disruption hooks
python3 -m pytest tests/ -k "shareable" -v # co-assignmentExpected output: 176 passed (pure-Python tests) — PyBullet tests in test_phase5.py and test_phase6.py are skipped automatically when gym-pybullet-drones is not installed.
# Execution actor
python3 -c "
import torch
ckpt = torch.load('checkpoints/actor_update204_final.pt', map_location='cpu', weights_only=False)
print('Actor — update:', ckpt['update'], '| timesteps:', f\"{ckpt['timesteps']:,}\")
"
# Bid policy (includes backward-compat migration for old single-head format)
python3 -c "
from allocator.learned_bidder import LearnedBidder
lb = LearnedBidder.from_checkpoint('checkpoints/bid_policy_final.pt')
import numpy as np
from allocator.bid_policy import build_bid_obs, BID_OBS_DIM
from envs.tasks.base_task import TaskSpec, TaskStatus
from envs.tasks.water_plant import WaterPlantTask
spec = TaskSpec('t', 'water_plant', np.array([3.,3.,1.], dtype=np.float32), engage_steps_required=10)
task = WaterPlantTask(spec); task.status = TaskStatus.ASSIGNED
obs = build_bid_obs(np.zeros(3, dtype=np.float32), 1.0, 0.0, task, 0, 500)
print('primary bid :', lb.policy.bid_numpy(obs))
print('marginal bid:', lb.policy.marginal_bid_numpy(obs))
"# 20 deterministic episodes with the trained actor
python3 -m evaluation.eval \
--checkpoint checkpoints/actor_update204_final.pt \
--episodes 20
# Render ASCII output step-by-step (slow)
python3 -m evaluation.eval \
--checkpoint checkpoints/actor_update204_final.pt \
--episodes 3 \
--renderExpected: task completion ~80–100%, mean reward +40 to +60.
Important: --exec-checkpoint takes the actor file; --checkpoint takes the bid policy file.
# Full run — 4 allocators × 6 scenarios × 20 episodes = 480 episodes (~3 min)
python3 -m evaluation.eval_allocation \
--exec-checkpoint checkpoints/actor_update204_final.pt \
--checkpoint checkpoints/bid_policy_final.pt \
--episodes 20 \
--csv results/phase4_results.csv
# Quick smoke — random actions, no checkpoints needed (~15 seconds)
python3 -m evaluation.eval_allocation --episodes 3
# Single scenario
python3 -m evaluation.eval_allocation \
--exec-checkpoint checkpoints/actor_update204_final.pt \
--checkpoint checkpoints/bid_policy_final.pt \
--scenarios task_vanish \
--episodes 10
# Two allocators head-to-head on all scenarios
python3 -m evaluation.eval_allocation \
--exec-checkpoint checkpoints/actor_update204_final.pt \
--checkpoint checkpoints/bid_policy_final.pt \
--allocators cbba learned \
--episodes 20
# Robustness: CBBA (comm_delay=10) vs Learned (obs_delay has no CLI flag —
# controlled at library level; use the comm_delay scenario for the paper comparison)
python3 -m evaluation.eval_allocation \
--exec-checkpoint checkpoints/actor_update204_final.pt \
--checkpoint checkpoints/bid_policy_final.pt \
--scenarios comm_delay \
--allocators greedy cbba oracle learned \
--comm-delay 10 \
--episodes 20
# Surge scenario only (tests co-assignment + task injection handling)
python3 -m evaluation.eval_allocation \
--exec-checkpoint checkpoints/actor_update204_final.pt \
--checkpoint checkpoints/bid_policy_final.pt \
--scenarios surge \
--episodes 20
# Drone failure scenario only
python3 -m evaluation.eval_allocation \
--exec-checkpoint checkpoints/actor_update204_final.pt \
--checkpoint checkpoints/bid_policy_final.pt \
--scenarios drone_failure \
--episodes 20Metrics in the output table:
| Column | Meaning |
|---|---|
CompRate |
Fraction of eligible tasks completed (excl. vanished) |
Makespan |
Steps until all tasks done, or max_steps if not |
TotReward |
Total episode reward |
BattFinal |
Mean normalised drone battery at episode end |
Reallocs |
Number of allocate() calls |
ReallocLat |
Mean steps from disruption to drone reassignment (−1 = no disruption) |
Collisions |
Pairwise drone collision count |
# 110 episodes per cell (11 seeds × 10 eps) — the full H1 dataset
python3 -m evaluation.eval_allocation \
--exec-checkpoint checkpoints/actor_update204_final.pt \
--checkpoint checkpoints/bid_policy_final.pt \
--episodes 10 --seeds 1 2 3 4 5 6 7 8 9 10 \
--csv results/h1_results.csv# With marginal head (baseline)
python3 -m evaluation.eval_allocation \
--exec-checkpoint checkpoints/actor_update204_final.pt \
--checkpoint checkpoints/bid_policy_final.pt \
--scenarios sweep_heavy --allocators learned \
--episodes 20 --seeds 1 2 3 4 \
--csv results/h2_with_marginal.csv
# Without marginal head (ablation)
python3 -m evaluation.eval_allocation \
--exec-checkpoint checkpoints/actor_update204_final.pt \
--checkpoint checkpoints/bid_policy_final.pt \
--scenarios sweep_heavy --allocators learned \
--no-marginal \
--episodes 20 --seeds 1 2 3 4 \
--csv results/h2_no_marginal.csv
# Compare the two CSVs
python3 - <<'EOF'
import csv, statistics, math
def load(p):
with open(p) as f: return list(csv.DictReader(f))
def agg(rows, field):
vals = [float(r[field]) for r in rows]
m = statistics.mean(vals); s = statistics.pstdev(vals)
ci = 1.96 * s / math.sqrt(len(vals))
return m, ci
with_m, no_m = load("results/h2_with_marginal.csv"), load("results/h2_no_marginal.csv")
print(f"{'Metric':22s} {'with_marginal':18s} {'no_marginal':18s} {'delta':>8s}")
for field, label in [("completion_rate","CompRate"),("makespan","Makespan"),
("total_reward","TotReward"),("collision_events","Collisions")]:
m1,ci1 = agg(with_m, field); m2,ci2 = agg(no_m, field)
sep = "SEPARATE" if abs(m1-m2) > ci1+ci2 else "overlap"
print(f"{label:22s} {m1:7.3f}±{ci1:.3f} {m2:7.3f}±{ci2:.3f} {m1-m2:+8.3f} {sep}")
EOFResult (100 episodes each): marginal head finishes sweep_heavy 4.7 steps faster (non-overlapping CIs) but loses 11.9 reward due to split completion bonuses (task.completion_reward() / n_assigned). The head co-assigns ~2 drones to sweep tasks per episode; each co-assignment halves that task's completion reward. See H2 status in the Hypotheses section above.
sweep_heavy all-allocator context (results/h2_all_allocators.csv, 100 eps each):
| Allocator | CR | Makespan | Reward | Collisions |
|---|---|---|---|---|
| Greedy | 0.998 ±0.004 | 29.2 ±6.1 | +33.8 ±57.1 | 2.88 |
| CBBA | 0.988 ±0.016 | 39.8 ±8.7 | +47.1 ±4.2 | 0.42 |
| Oracle | 0.998 ±0.004 | 28.5 ±6.1 | +62.7 ±0.8 | 0.01 |
| Learned | 1.000 ±0.000 | 44.2 ±1.1 | +50.8 ±1.1 | 0.18 |
Oracle dominates on reward (full completion bonus per drone, no co-assignment splitting). Learned has the tightest variance and the only CR = 1.000 with zero CI, but pays a 15.7-step makespan penalty vs Oracle. Greedy achieves Oracle-level makespan but has extreme reward variance (σ = 291) from catastrophic collision episodes.
# Sweep obs_delay 0→20 for all allocators across disruption scenarios (3,000 eps total)
python3 -m evaluation.eval_allocation \
--exec-checkpoint checkpoints/actor_update204_final.pt \
--checkpoint checkpoints/bid_policy_final.pt \
--obs-delay-sweep 0 2 5 10 20 \
--scenarios task_vanish surge drone_failure \
--episodes 10 --seeds 1 2 3 4 \
--csv results/h3_obs_delay.csvResult: No measurable degradation at any delay for any allocator. This is structurally expected — episodes complete in 20–60 steps at MAX_SPEED = 0.5 m/step, so 20-step stale positions are ≤10 m error in the worst case, which barely changes bid ordering. The hypothesis is instead supported through reward variance: Learned σ ≤ 1.4 vs Greedy σ up to 370 across all non-failure scenarios. See H3 status in the Hypotheses section above.
Learned is 13–28 steps slower than CBBA/Oracle despite equal completion rates. Running 20 episodes of task_inject with instrumented allocators:
| Allocator | Realloc calls | Makespan |
|---|---|---|
| CBBA | 7.4 | 28 steps |
| Oracle | 7.4 | 23 steps |
| Learned | 8.0 | 41 steps |
Learned makes only ~0.6 extra allocate() calls yet finishes 18 steps later, ruling out call overhead. The cause is conservative bid values: the policy produces intermediate bid values (no clear winner per task) that cause the most-contested-first resolver to iterate through sub-optimal orderings across auction rounds. Additionally, marginal_activation_rate() returns -1.0 on all non-shareable layouts — the marginal head is never exercised outside sweep_heavy, confirming the makespan lag is a primary-bid issue. A practical fix would be to train the bid policy with an auxiliary makespan-minimisation loss or to normalise bid values to be more decisive.
# Greedy allocator, real-time GUI (default)
python3 -m lab.deploy \
--checkpoint checkpoints/actor_update204_final.pt
# Learned bidder with trained weights, slow-motion
python3 -m lab.deploy \
--checkpoint checkpoints/actor_update204_final.pt \
--allocator learned \
--bid-checkpoint checkpoints/bid_policy_final.pt \
--time-scale 0.3
# CBBA, 6 drones, slow-motion
python3 -m lab.deploy \
--checkpoint checkpoints/actor_update204_final.pt \
--allocator cbba \
--n-drones 6 \
--time-scale 0.3
# Oracle allocator (Hungarian, upper bound)
python3 -m lab.deploy \
--checkpoint checkpoints/actor_update204_final.pt \
--allocator oracle \
--time-scale 0.3
# Periodic re-auction every 20 steps (bid lines update visibly)
python3 -m lab.deploy \
--checkpoint checkpoints/actor_update204_final.pt \
--allocator greedy \
--auction-interval 20
# Headless benchmark — no GUI, max speed
python3 -m lab.deploy \
--checkpoint checkpoints/actor_update204_final.pt \
--allocator learned \
--bid-checkpoint checkpoints/bid_policy_final.pt \
--no-gui \
--episodes 20Camera controls (click the PyBullet window first):
| Key | Action |
|---|---|
W / S |
Pan forward / back |
A / D |
Pan left / right |
Q / E |
Zoom in / out |
Z / X |
Rotate left / right (yaw) |
R / F |
Tilt up / down (pitch) |
1–5 |
Preset views (isometric / top-down / sides / cinematic) |
0 |
Cycle follow-drone mode |
| Mouse drag | Orbit (left), pan (right), scroll zoom |
Bid visualisation: coloured lines from each drone to its winning task. Red = low bid, yellow = mid, green = high.
Skip these if you already have the checkpoints.
Phase 1 — execution actor (MAPPO):
# Default config (~5M steps, ~4–8h on GPU)
python3 -m training.train_mappo --config training/config.yaml
# With W&B logging
python3 -m training.train_mappo --config training/config.yaml --wandb
# Press Ctrl+C at any time — saves checkpoint cleanly
# Output: checkpoints/actor_update<N>_final.ptPhase 3 — bid policy:
# Uses the harder 7-task layout in config_bid.yaml (~400 updates, ~2–4h on GPU)
python3 -m training.train_bid_policy \
--config training/config_bid.yaml \
--exec-checkpoint checkpoints/actor_update204_final.pt
# With default config (5-task layout, faster)
python3 -m training.train_bid_policy \
--config training/config.yaml \
--exec-checkpoint checkpoints/actor_update204_final.pt
# Press Ctrl+C — saves checkpoint cleanly
# Output: checkpoints/bid_policy_final.ptThree configs are provided in a progressive fidelity ladder. Run them in order, each warm-starting from the previous phase's checkpoint.
Step A — clean baseline (config_fresh.yaml)
Train from scratch with no noise and no curriculum — used to establish a clean performance baseline:
python3 -m training.train_mappo --config training/config_fresh.yaml
# Output: checkpoints_fresh/actor_update<N>_final.pt (~3M steps, ~10 min CPU)Step B — sim-to-real hardening (config_sim2real.yaml)
Adds 5 cm observation noise and an altitude penalty to teach the policy to maintain z ≥ 1 m during lateral movement — critical for real quadrotor dynamics where low-speed PID struggles to hold altitude:
python3 -m training.train_mappo \
--config training/config_sim2real.yaml \
--resume checkpoints_fresh/actor_update<N>_final.pt
# Output: checkpoints_sim2real/actor_update<N>_final.pt (~3M steps, warm-start)Step C — PyBullet fine-tune (config_pybullet.yaml)
Fine-tune the hardened policy directly inside the physics simulator. Enables full aerodynamic effects (PYB_GND_DRAG_DW: ground effect, drag, downwash) plus action-delay FIFO, motor-lag low-pass filter, and per-episode domain randomisation (±10% mass, ±20% drag):
python3 -m training.train_mappo \
--config training/config_pybullet.yaml \
--resume checkpoints_sim2real/actor_update<N>_final.pt
# Output: checkpoints_pybullet/actor_update<N>_final.pt (~1.5M steps, ~6 min GPU)Sim-to-real parameters reference:
| Parameter | Config key | Default | Description |
|---|---|---|---|
| Obs noise | obs_noise_std |
0.05 m | Gaussian noise added to pos/vel obs (matches UWB/OptiTrack ~5 cm) |
| Action delay | action_delay_steps |
1 step | FIFO holding old commands before execution (Crazyflie radio + compute latency) |
| Motor lag | motor_lag |
0.3 | First-order low-pass α on velocity commands (motor spin-up/down, ~30 ms) |
| Domain rand | domain_rand |
false | ±10% mass jitter + ±20% drag jitter each episode |
| Altitude penalty | altitude_penalty_coef |
0.0 | Penalty coefficient for flying below hover_z |
| Hover altitude | hover_z |
1.0 m | Target hover height used by altitude penalty |
| Physics mode | — (hardcoded) | PYB_GND_DRAG_DW |
Ground effect + aerodynamic drag + downwash between drones |
Evaluate the PyBullet-fine-tuned policy:
python3 -m evaluation.eval \
--checkpoint checkpoints_pybullet/actor_update489_final.pt \
--episodes 20
# Or deploy visually in the PyBullet lab:
python3 -m lab.deploy \
--checkpoint checkpoints_pybullet/actor_update489_final.pt \
--allocator greedy \
--time-scale 0.5# Run one episode with any allocator
from envs.home_env import HomeEnv
from allocator.greedy_auction import GreedyAuction
env = HomeEnv({"n_drones": 3, "allocator": GreedyAuction()})
obs, _ = env.reset(seed=42)
done = False
while not done:
actions = {aid: env.action_space.sample() for aid in obs}
obs, rewards, terminated, truncated, infos = env.step(actions)
done = terminated["__all__"] or truncated["__all__"]
# Load the learned bidder from checkpoint
from allocator.learned_bidder import LearnedBidder
lb = LearnedBidder.from_checkpoint("checkpoints/bid_policy_final.pt")
env = HomeEnv({"n_drones": 3, "allocator": lb})
# Load with obs_delay for robustness experiment (5-step stale observations)
lb_delayed = LearnedBidder.from_checkpoint(
"checkpoints/bid_policy_final.pt",
obs_delay=5,
)
# Disruption API
env = HomeEnv({"n_drones": 3, "allocator": GreedyAuction()})
obs, _ = env.reset()
env.remove_task("water_plant_1") # vanish task mid-episode → re-auction
env.add_task("water_plant", [4., 7., 1.]) # inject task mid-episode → re-auction
# BidPolicy dual-head (primary + marginal)
import numpy as np, torch
from allocator.bid_policy import BidPolicy, build_bid_obs, BID_OBS_DIM
policy = BidPolicy() # 14-dim → (primary_logit, marginal_logit)
obs_vec = np.zeros(BID_OBS_DIM, dtype=np.float32)
print(policy.bid_numpy(obs_vec)) # primary bid ∈ (0, 1)
print(policy.marginal_bid_numpy(obs_vec)) # marginal co-assignment bid ∈ (0, 1)Every command that uses trained weights takes two distinct flags:
| Flag | File | Purpose |
|---|---|---|
--exec-checkpoint |
checkpoints/actor_update204_final.pt |
Execution actor — drives drone movement |
--checkpoint |
checkpoints/bid_policy_final.pt |
Bid policy — drives task allocation |
--bid-checkpoint |
checkpoints/bid_policy_final.pt |
Same file, used in lab/deploy.py |
Common mistake: passing the bid policy to --exec-checkpoint (or vice versa) gives a KeyError: 'actor_state_dict' — the keys in each checkpoint file are different.
macOS / Linux
chmod +x install.sh && ./install.shWindows — open Command Prompt or PowerShell in the repo folder, then:
.\install.batPowerShell tip: you must prefix with
.\— typinginstall.batalone gives "not recognized as a cmdlet". Use.\install.bator open a plain Command Prompt instead.
Or directly (any platform):
python install.pyOn macOS with clang 17+ / SDK 15+ (Sequoia / Tahoe) the installer patches PyBullet's source before compiling. On Windows and Linux it uses pre-built wheels.
Verify install:
python3 install.py --check| Index | Meaning |
|---|---|
| 0–2 | Own position (x, y, z) metres |
| 3 | Battery level normalised 0–1 |
| 4 | Task progress 0–1 |
| 5–7 | Vector to task target (dx, dy, dz) — zeros if idle |
| 8–10 | Own velocity (vx, vy, vz) |
| 11 | Tool-engaged flag (0 / 1) |
| 12–14 | Nearest neighbour relative position — zeros if no neighbour |
| Index | Meaning |
|---|---|
| 0–2 | Drone position (x, y, z) |
| 3 | Battery level |
| 4 | Current task progress (0 if idle) |
| 5–7 | Vector to candidate task (dx, dy, dz) |
| 8 | remaining_work() of candidate task ∈ [0, 1] |
| 9 | Number of drones already assigned to candidate task |
| 10 | Step / max_steps (urgency) |
| 11–13 | Task type one-hot [water_plant, sweep_floor, toggle_light] |
| Index | Meaning | Range |
|---|---|---|
| 0–2 | Δx, Δy, Δz movement | −1 to +1 (tanh squashed) |
| 3 | Tool engage signal | 0 to 1 (sigmoid, threshold 0.5) |
| Event | Value |
|---|---|
| Per-step alive penalty | −0.01 |
| Drone–drone collision | −5.0 |
| Battery depleted | −3.0 |
| Water plant completed | +10.0 |
| Sweep floor completed | +12.0 |
| Toggle light completed | +8.0 |
| Cooperative bonus (all done early) | +2.0 per drone |
| Dense proximity shaping | small positive gradient toward target |
| Scenario | Disruption | Hook fires at |
|---|---|---|
baseline |
None — standard 5-task layout | — |
task_vanish |
Task 1 removed (drone mid-transit) | step 50 |
task_inject |
New water_plant injected | step 60 |
drone_failure |
drone_1 battery zeroed, re-auction triggered | step 40 |
comm_delay |
CBBA uses --comm-delay broadcast delay; others unaffected |
structural |
surge |
Two extra tasks injected | steps 30 and 80 |
All allocators implement BaseAllocator:
from allocator.base_allocator import BaseAllocator, WorldSnapshot, AllocationResult, Bid
class MyAllocator(BaseAllocator):
def allocate(self, snapshot: WorldSnapshot) -> AllocationResult:
# snapshot.drone_positions — dict[str, np.ndarray(3,)]
# snapshot.drone_batteries — dict[str, float] ∈ [0, 1]
# snapshot.drone_task_progress — dict[str, float] ∈ [0, 1]
# snapshot.current_assignments — dict[str, int | None]
# snapshot.tasks — list[BaseTask]
# snapshot.step, .max_steps
return AllocationResult(
assignments={"drone_0": 2, "drone_1": None, ...},
bids=[Bid("drone_0", task_idx=2, bid_value=0.87), ...],
)
def on_task_complete(self, task_idx: int, step: int) -> None: ...
def on_task_vanish(self, task_idx: int, step: int) -> None: ...Plug into either environment:
from envs.home_env import HomeEnv
env = HomeEnv({"n_drones": 4, "allocator": MyAllocator()})
from envs.pybullet_env import PybulletHomeEnv
env = PybulletHomeEnv({
"n_drones": 4,
"gui": True,
"allocator": MyAllocator(),
"auction_interval": 20, # re-auction every 20 steps
})| Phase | Steps | Mean Reward | Entropy | Value Loss |
|---|---|---|---|---|
| Random | 0 | −5 to −15 | ~5.7 | ~0.5 |
| Early signal | ~100k | −2 to +5 | 5–8 | 0.5–2.0 |
| Learning | ~500k | +5 to +20 | 6–9 | 0.5–1.5 |
| Competent | ~1–2M | +20 to +40 | 4–7 (falling) | < 0.5 |
| Good | ~3–5M | +40 to +55 | 2–5 | < 0.3 |
Healthy signs: policy loss small and negative (−0.001 to −0.02); entropy falls as policy specialises; eval reward tracks training reward.
Red flags: entropy > 10 sustained → lower lr_actor; policy loss positive → lower lr_actor or n_epochs.
| Update | Makespan | Tasks done | Note |
|---|---|---|---|
| 0 | ~480 | 2–3 / 5 | Random bids |
| 50 | ~350 | 3–4 / 5 | Learns proximity signal |
| 100 | ~280 | 4 / 5 | Co-assignment emerging |
| 200 | ~220 | 4–5 / 5 | Near-greedy quality |
| 400 | ~30–60 | 5 / 5 | Exceeds greedy on all scenarios |
Training automatically advances through 4 stages as eval reward crosses thresholds in training/config.yaml:
| Stage | Threshold | Description |
|---|---|---|
| 0 | 5.0 | Default — all 5 tasks, 3 drones |
| 1 | 15.0 | Policy completing 1–2 tasks reliably |
| 2 | 30.0 | 3–4 tasks per episode |
| 3 | 50.0 | All tasks + cooperative bonus |
- Create
envs/tasks/my_task.pysubclassingBaseTask - Implement
step(drone_position, tool_engaged) → float,completion_reward() → float,remaining_work() → float - Register in
envs/home_env.py→_TASK_REGISTRYand_DEFAULT_TASK_LAYOUTS - Register in
envs/pybullet_env.py→_TASK_REGISTRY - Add a URDF asset to
assets/
Subclass BaseAllocator and pass it via config — see Allocator Interface above.
Change n_drones in training/config.yaml. The Actor is parameter-shared so it generalises to any N without retraining from scratch. The Critic input dim auto-scales.
| Component | Status | Notes |
|---|---|---|
Training (train_mappo.py) |
✅ | Pure Python, no compilation |
Bid policy training (train_bid_policy.py) |
✅ | Pure Python/PyTorch |
| HomeEnv + allocators | ✅ | Pure Python/NumPy |
| Phase 4 eval harness | ✅ | No PyBullet required |
| PyBullet | ✅ | Pre-built wheel, no source build |
| PyBullet GUI | ✅ | Opens normally |
| gym-pybullet-drones | ✅ | Needs Git in PATH: winget install Git.Git |
gymnasium >= 0.29.0
numpy >= 1.24.0
torch >= 2.0.0
pyyaml >= 6.0
scipy OracleAllocator Hungarian algorithm (optional — falls back to greedy)
pybullet 3.2.7 physics lab only
gym-pybullet-drones 2.1.0 physics lab only
wandb optional W&B logging
MIT