Realtime Video Detection

Real-time outdoor video object detection and instance segmentation, packaged for edge deployment to the Rockchip RK3576 NPU through an ONNX to RKNN (INT8) pipeline. The detector is a transformer-based, real-time architecture (DETR family).

The repository contains two things:

1. A live, browser-based detection app (images and video) that demonstrates the model running in real time.
2. The full edge pipeline a camera-AI device needs: fine-tuning on custom classes, evaluation, ONNX export, RKNN INT8 conversion for the RK3576, and an on-device inference sample, with reproducible documentation for each step.

Japanese overview and a requirement-by-requirement mapping: README.ja.md.

Capabilities

• Real-time object detection and instance segmentation on images and video.
• Custom-class fine-tuning from a COCO dataset.
• COCO mAP evaluation for the host model and after INT8 quantization.
• ONNX export and ONNX-to-RKNN INT8 conversion targeting RK3576.
• On-device inference sample with latency/FPS measurement.
• A no-black-box workflow: the client can retrain, re-export, and re-deploy.

Repository layout

app.py                     Live detection app (Gradio) - images and video
utils/                     Image/video helpers for the app
videos/                    Sample clips for the app
pipeline/
  common.py                Model registry (nano/small recommended for edge)
  train.py                 Fine-tune on a custom COCO dataset
  evaluate.py              COCO mAP evaluation
  export_onnx.py           Trained checkpoint -> ONNX
  convert_rknn.py          ONNX -> RKNN INT8 for RK3576 (rknn-toolkit2)
  infer_rknn.py            RK3576 on-device inference sample (rknn-toolkit-lite2)
configs/
  classes.yaml             Detection class design
docs/                      Deliverable documentation (see index below)
requirements.txt           App runtime
requirements-pipeline.txt  Host-side training/export pipeline

Live demo

pip install -r requirements.txt
python app.py

Opens a Gradio UI at http://localhost:7860 with Image and Video tabs. Pick a checkpoint, set confidence and resolution, and run. For self-hosting, serve it behind a reverse proxy (Caddy or Nginx) with HTTPS and bind Gradio to 0.0.0.0:7860.

Edge pipeline

Install the host pipeline dependencies (separate from the app runtime):

pip install -r requirements-pipeline.txt

End-to-end flow, host (x86-64 Linux) then board (RK3576):

# 1. Fine-tune on your dataset (COCO layout: train/ valid/ test/)
python pipeline/train.py --dataset-dir dataset --model small --epochs 50

# 2. Evaluate on the validation split (COCO mAP)
python pipeline/evaluate.py --model small \
  --checkpoint output/checkpoint_best_ema.pth --split-dir dataset/valid

# 3. Export the trained checkpoint to ONNX
python pipeline/export_onnx.py --model small \
  --checkpoint output/checkpoint_best_ema.pth --output-dir output/onnx

# 4. Convert to RKNN with INT8 quantization (needs rknn-toolkit2)
python pipeline/convert_rknn.py --onnx output/onnx/inference_model.onnx \
  --output output/rk3576/model.rknn --quant --dataset calib/list.txt

# 5. On the RK3576 board: measure latency and run detection
python pipeline/infer_rknn.py --rknn model.rknn --image sample.jpg --size 512

convert_rknn.py runs on x86-64 Linux only (pip install rknn-toolkit2). infer_rknn.py runs on the board (pip install rknn-toolkit-lite2).

Class design

Four foreground classes (configs/classes.yaml): target_object (client-specific, disclosed under NDA), person, animal, vehicle. Background is implicit in detection; recurring false alarms are handled with hard-negative images rather than a positive class. See docs/dataset-design.md.

Model choice for the NPU

The nano and small detector variants are the recommended edge targets for accuracy at low latency. The base detector is a transformer (DETR-family) model, and some operators can be costly or unsupported under RKNN INT8; YOLOv5/v8 has first-class RKNN model-zoo support. We validate operator support and INT8 accuracy early and keep YOLOv5/v8 as a fallback for the NPU. The conversion and on-device scripts are model-agnostic at the ONNX boundary. See docs/rknn-conversion.md.

Documentation

• Dataset Design
• Annotation Guide
• RKNN Conversion and INT8 Quantization
• RK3576 On-Device Deployment
• Android Integration API Specification
• Retraining and Model Update Guide

References

• Rockchip RKNN Toolkit2: https://github.com/airockchip/rknn-toolkit2
• Rockchip RKNN Model Zoo: https://github.com/airockchip/rknn_model_zoo

License

Apache 2.0. This product builds on open-source components; a full third-party OSS and license list is provided as a delivery artifact.