Skip to content

COCO export: emit a tracks table and stop duplicating track-level data on every frame #1873

Description

@PaulHax

Problem

export_dive_as_coco (server/dive_utils/serializers/kwcoco.py:582, mirrored in
client/platform/desktop/backend/serializers/coco.ts) emits info, images,
annotations, categories, and conditionally videos. There is no tracks
key on either platform.

Meanwhile each annotation gets four values read from the track, not the
feature — so they are constant across the track and written once per frame:

Field Server Client
track_id kwcoco.py:674 coco.ts:596
prob kwcoco.py:668 coco.ts:599
dive_confidence_pairs kwcoco.py:671 coco.ts:600
dive_track_attributes kwcoco.py:678 coco.ts:602

Three consequences:

  1. There's nowhere to put track-level data. DIVE groups annotations by
    track_id, the original COCO tracking convention — self-consistent, but it
    gives a track no identity of its own. kwcoco made tracks first-class in
    v0.7.0 (2023-08-15)

  2. Track-level data is duplicated per frame — a 500-frame track writes its
    attributes and confidence pairs 500 times. This is a specific instance of
    kwcoco #7
    ("annotations are often duplicated in every frame of a video").

  3. prob is dense over the whole category list — one float per category per
    annotation, for a track that usually has one or two real pairs. Scales as
    categories × detections, so it worsens as hierarchical types grow the
    category count (Integrate Baseline VLM and Hierarchical Models - P6 #1587). Also worth knowing: CocoDataset.union remaps
    category_id and keypoint_category_id but never prob, so unioning
    datasets with different category orders silently misaligns every array.

Proposed changes

  1. Emit a tracks table, unconditionally — every DIVE annotation belongs to a
    Track (a "detection" is a one-feature track), so there is no DIVE dataset
    without tracks. Not gated on emit_video, which keys on whether the source
    was a video, not on whether tracks exist.
  2. Move confidence pairs and track attributes onto the track entry.
  3. Stop writing the dense per-annotation prob. It's optional in the schema and
    category_id/score still gives any reader the primary label.
  4. Read a tracks table on import — currently nothing on either platform reads
    one (load_coco_as_tracks_and_attributes, kwcoco.py:437), so a file from
    another tool loses its track names. This is the round-trip half of
    #1089.

What's standardized

  • The tracks table with {id, name} is standard. No invention.
  • category_scores and attributes on a track are not. kwcoco's TRACK is
    exactly {id, name}, and kwcoco has no track-level classification concept at
    all — categories attach to annotations only. So hoisting these is an
    efficiency fix, not a standardization one; they remain DIVE extensions
    wherever they sit.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions