eval

Evaluation

⚠️ This codebase is new and not yet battle-tested. Metrics can often have weird corner-cases. If you encounter one, please let us know. The generated videos show all computed metric, including per-timestamp result, making it easy to check if metric results look reasonable.

This module is a refactored version of the KPI service. It

• Reads in ASL logs
• Computes metrics (see src/eval/scorers/__init__.py for list of implemented "Scorers")
• Saves results locally as parquet files
• And generates a video (see image) - stored locally.

Configuration

See schema.py. Video output is controlled by eval.video: use video_layouts to select which layouts to render (e.g. DEFAULT, REASONING_OVERLAY) and reasoning_text_refresh_interval_s for the reasoning overlay layout.

Writing your own metric scorer

A key motivation for this module was to make writing new scorers fast and easy. To do so, we:

• Rely heavily on dataclasses for storing the information parsed from ASL. The information is organised hierarchically, with the root being SimulationResult in data.py. Use ScenarioEvaluator for evaluation and asl_loader.load_scenario_eval_input_from_asl() to load ASL files.
• We don't use indexing by index, but always by timestamp_us, to reduce off-by-one errors.
• We rely on the Trajectory class from AlpaSim, which allows indexing into trajectories by timestamp. We expand this class to RenderableTrajectory in data.py which also contains the bounding box and knows how to render itself onto a video frame.
• Lastly, we also rely heavily on the shapely library, to abstract away complex geometric computations such as distance, contains, project, intersects, etc... The RenderableTrajectory class has helper methods to convert itself to shapely objects.
• We also have a ShapelyMap class, which is primarily used for fast video rendering of maps. For computing map-based metrics, it's probably easiest to use the trajdata.vec_map directly, which is also stored in SimulationResult and allows querying for current lanes, etc..

Running locally

This part of the codebase is managed by uv.

curl -LsSf https://astral.sh/uv/install.sh | sh

Recommended workflow:

1. First run the wizard normally (after installing it with uv tool install -e src/wizard) and generating ASL files.

uv run alpasim_wizard deploy=local topology=1gpu driver=vavam wizard.log_dir=<log_dir>

2. Execute this from src/eval:

uv run alpasim-eval  \
  --asl_search_glob=<log_dir>/rollouts/clipgt-d8cbf4ca-b7ff-44bd-a5be-260f736a02fe/15f2c488-10ad-11f0-b123-0242c0a84004/\*\*/\*.asl \
  --config_path=<log_dir>/eval-config.yaml \
  --trajdata_cache_dir=<path_to_alpasim_repo>/data/trafficsim/unified_data_cache \
  --usdz_glob="<path_to_alpasim_repo>/data/nre-artifacts/all-usdzs/**/*.usdz"

The environment is shared with that of the main project and is automatically managed by uv.

Overview over the codebase, e.g. for writing

Main components of the codebase:

• data.py contains most datastructures. Start exploring from SimulationResult and ScenarioEvalInput
• Parsing ASL logs is done in asl_loader.load_scenario_eval_input_from_asl() in asl_loader.py
• Scorers are implemented in the folder scorers. If you add a new scorer, don't forget to add it to the list in scorers.__init__.py
• Scorers produce metrics per timestamp per rollout. These results are aggregated in eval_aggregation.py. As long as you conform to the existing datastructure, you probably won't need to touch this.
• Lastly, video generation is done in video.py

Building

The eval service is built from the top-level Dockerfile at the repo root (same image as other services). Use the main project build and CI; there is no separate eval-only image or build.sh in this module. The wizard uses the image produced by the top-level build when running evaluations.