Skip to content

Commit 209727a

Browse files
BreakthroughBreakthrough
committed
Merge branch 'benchmark-sweep'
2 parents 9581909 + 8e21778 commit 209727a

19 files changed

Lines changed: 3573 additions & 100 deletions

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -102,7 +102,7 @@ See [the documentation](https://www.scenedetect.com/docs/latest/api.html) for mo
102102

103103
**Benchmark**:
104104

105-
We evaluate the performance of different detectors in terms of accuracy and processing speed. See the [benchmark report](benchmark/README.md) for details.
105+
We evaluate the performance of different detectors in terms of accuracy and processing speed. See [www.scenedetect.com/benchmarks](https://www.scenedetect.com/benchmarks/) for results, or the [benchmark report](benchmark/README.md) for details on the datasets and methodology.
106106

107107
## Reference
108108

‎benchmark/README.md‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,31 @@ python -m benchmark --detector detect-content --dataset BBC
2626

2727
Pass `--help` for `--dataset-root`, `--backend`, `--tolerance`, and `--out` options.
2828

29+
### Parameter sweeps
30+
31+
`python -m benchmark.sweep` runs a grid over detector parameters and reports the
32+
top cells by F1 plus the Pareto front across tolerances. One decode is shared by up to
33+
`--workers` parallel detectors via an internal fan-out wrapper, so the cost scales with
34+
`ceil(cells / workers)` decodes per video rather than `cells` decodes.
35+
36+
```bash
37+
python -m benchmark.sweep \
38+
--detector detect-content --dataset BBC \
39+
--params "threshold=15:35:1;min_scene_len=0.0:1.0:0.1" \
40+
--tolerance 0,1 --workers 16 \
41+
--out sweep-content-bbc.json
42+
```
43+
44+
Spec language for `--params`: clauses joined by `;`. Each clause is either
45+
`key=v1,v2,v3` (enumerated values) or `key=start:stop:step` (numeric range, inclusive
46+
when `stop` lands on a step). Omitted keys use the detector's default.
47+
48+
Time-valued kwargs (`min_scene_len`, etc.) accept `TimecodeLike` - integers are
49+
frames, floats are seconds, and strings like `"0.1s"` / `"00:00:00.500"` also work.
50+
Prefer floats so the same sweep is comparable across datasets with different
51+
framerates. Use `--quick N` to limit to the first N samples for iteration; published
52+
numbers should always come from the full corpus.
53+
2954
## Dataset Download
3055

3156
### BBC
@@ -115,6 +140,33 @@ Elapsed is mean wall-clock seconds per video.
115140
| HistogramDetector | 69.67 | 81.99 | 75.33 |
116141
| ThresholdDetector | 5.69 | 99.24 | 10.77 |
117142

143+
## Parameter sweep results
144+
145+
The tables above use each detector's v0.7 defaults. A grid sweep over the key parameters
146+
scored by hard-cut F1 at 1-frame tolerance, averaged across BBC / AutoShot / ClipShots gives the
147+
best single parameter set for this corpus mix:
148+
149+
| Detector | Best mean F1 | Best params | v0.7 default |
150+
|:-----------------:|:------------:|:-----------------------------------------------------------|:---------------------------------------|
151+
| ContentDetector | 73.4 | threshold=31, min_scene_len=0.6s | threshold=27 |
152+
| AdaptiveDetector | 76.3 | adaptive_threshold=3.5, window_width=3, min_scene_len=0.6s | adaptive_threshold=3.0, window_width=2 |
153+
| HashDetector | 69.8 | threshold=0.35, size=8 | threshold=0.395, size=16 |
154+
| HistogramDetector | 66.3 | threshold=0.20, bins=128 | threshold=0.05, bins=256 |
155+
| ThresholdDetector | -- | detects fades, not hard cuts (validation only) | threshold=12 |
156+
157+
Per-dataset optima differ (e.g. HistogramDetector peaks at threshold=0.11 on BBC but keeps climbing
158+
to 0.35 on ClipShots); the "best params" column is the single cell with the highest cross-dataset
159+
mean. Full per-dataset breakdowns - including F1 at 0-frame tolerance and precision/recall - are in
160+
[`SWEEP_REPORT.md`](SWEEP_REPORT.md), regenerated with `python -m benchmark.report_sweep`.
161+
The full grids (all detectors and datasets) are driven by `scripts/benchmark_sweep.sh`.
162+
163+
Notes:
164+
165+
- Every sweep cell requires a full decode pass (shared across up to `--workers` cells); `size` /
166+
`bins` change the per-frame metric itself, so hash / hist cells cannot share metrics either.
167+
- HistogramDetector's v0.7 default (0.05 / 256) scores only 54.3 mean F1 - well below the swept
168+
optimum, making it the most miscalibrated default of the five.
169+
118170
## Citations
119171

120172
### BBC

‎benchmark/SWEEP_REPORT.md‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
# Detector parameter sweep report
2+
3+
Generated by `benchmark/report_sweep.py` from `benchmark/sweep.py` grid results (hash/hist were swept with `min_scene_len` fixed at its default). F1/precision/recall are percentages on hard cuts; tol = frame tolerance.
4+
5+
## detect-content
6+
7+
**Best cell per dataset (by F1@1):**
8+
9+
| Dataset | F1@1 | Prec@1 | Rec@1 | F1@0 | Params |
10+
| --------- | ----- | ------ | ----- | ----- | ------------------------------- |
11+
| BBC | 88.34 | 90.00 | 86.75 | 87.35 | min_scene_len=0.8, threshold=25 |
12+
| AutoShot | 73.44 | 79.54 | 68.21 | 70.54 | min_scene_len=0.4, threshold=29 |
13+
| ClipShots | 66.74 | 58.93 | 76.95 | 66.44 | min_scene_len=0.8, threshold=35 |
14+
15+
**Best params averaged across all datasets (mean F1@1):**
16+
17+
| Mean F1@1 | BBC | AutoShot | ClipShots | Params |
18+
| --------- | ----- | -------- | --------- | ------------------------------- |
19+
| 73.39 | 85.95 | 71.51 | 62.73 | min_scene_len=0.6, threshold=31 |
20+
| 73.37 | 84.27 | 70.95 | 64.89 | min_scene_len=0.6, threshold=33 |
21+
| 73.32 | 83.99 | 72.43 | 63.54 | min_scene_len=0.4, threshold=33 |
22+
| 73.15 | 87.28 | 72.05 | 60.13 | min_scene_len=0.6, threshold=29 |
23+
| 73.09 | 85.18 | 72.97 | 61.10 | min_scene_len=0.4, threshold=31 |
24+
25+
## detect-adaptive
26+
27+
**Best cell per dataset (by F1@1):**
28+
29+
| Dataset | F1@1 | Prec@1 | Rec@1 | F1@0 | Params |
30+
| --------- | ----- | ------ | ----- | ----- | --------------------------------------------------------- |
31+
| BBC | 94.57 | 94.89 | 94.26 | 93.50 | adaptive_threshold=2, min_scene_len=0.6, window_width=2 |
32+
| AutoShot | 77.19 | 80.48 | 74.16 | 75.45 | adaptive_threshold=3.5, min_scene_len=0.4, window_width=3 |
33+
| ClipShots | 65.53 | 60.19 | 71.89 | 65.47 | adaptive_threshold=5.5, min_scene_len=0.6, window_width=3 |
34+
35+
**Best params averaged across all datasets (mean F1@1):**
36+
37+
| Mean F1@1 | BBC | AutoShot | ClipShots | Params |
38+
| --------- | ----- | -------- | --------- | --------------------------------------------------------- |
39+
| 76.34 | 90.41 | 76.27 | 62.32 | adaptive_threshold=3.5, min_scene_len=0.6, window_width=3 |
40+
| 76.21 | 87.87 | 76.17 | 64.58 | adaptive_threshold=4, min_scene_len=0.6, window_width=3 |
41+
| 76.18 | 90.43 | 77.19 | 60.93 | adaptive_threshold=3.5, min_scene_len=0.4, window_width=3 |
42+
| 76.16 | 87.91 | 77.07 | 63.51 | adaptive_threshold=4, min_scene_len=0.4, window_width=3 |
43+
| 75.45 | 85.28 | 75.71 | 65.37 | adaptive_threshold=4.5, min_scene_len=0.6, window_width=3 |
44+
45+
## detect-hash
46+
47+
**Best cell per dataset (by F1@1):**
48+
49+
| Dataset | F1@1 | Prec@1 | Rec@1 | F1@0 | Params |
50+
| --------- | ----- | ------ | ----- | ----- | ------------------------ |
51+
| BBC | 86.91 | 81.59 | 92.96 | 85.81 | size=16, threshold=0.425 |
52+
| AutoShot | 70.17 | 76.06 | 65.12 | 66.89 | size=8, threshold=0.325 |
53+
| ClipShots | 56.38 | 44.46 | 77.03 | 55.66 | size=8, threshold=0.4 |
54+
55+
**Best params averaged across all datasets (mean F1@1):**
56+
57+
| Mean F1@1 | BBC | AutoShot | ClipShots | Params |
58+
| --------- | ----- | -------- | --------- | ----------------------- |
59+
| 69.83 | 86.38 | 68.98 | 54.12 | size=8, threshold=0.35 |
60+
| 69.63 | 86.65 | 65.86 | 56.38 | size=8, threshold=0.4 |
61+
| 69.63 | 86.65 | 65.86 | 56.37 | size=8, threshold=0.375 |
62+
| 68.18 | 84.28 | 70.17 | 50.10 | size=8, threshold=0.325 |
63+
| 67.00 | 83.71 | 61.66 | 55.65 | size=8, threshold=0.425 |
64+
65+
## detect-hist
66+
67+
> Note: thresholds >= 0.21 come from a grid-extension run (`detect-hist-ext-<dataset>.json`) after the initial grid's best cell landed on its 0.20 upper edge.
68+
69+
**Best cell per dataset (by F1@1):**
70+
71+
| Dataset | F1@1 | Prec@1 | Rec@1 | F1@0 | Params |
72+
| --------- | ----- | ------ | ----- | ----- | ------------------------ |
73+
| BBC | 86.58 | 87.32 | 85.86 | 85.42 | bins=128, threshold=0.11 |
74+
| AutoShot | 68.99 | 75.29 | 63.67 | 65.70 | bins=128, threshold=0.2 |
75+
| ClipShots | 53.25 | 46.26 | 62.74 | 52.90 | bins=128, threshold=0.35 |
76+
77+
**Best params averaged across all datasets (mean F1@1):**
78+
79+
| Mean F1@1 | BBC | AutoShot | ClipShots | Params |
80+
| --------- | ----- | -------- | --------- | ------------------------ |
81+
| 66.27 | 82.10 | 68.99 | 47.72 | bins=128, threshold=0.2 |
82+
| 66.23 | 81.47 | 68.87 | 48.36 | bins=128, threshold=0.21 |
83+
| 66.20 | 82.74 | 68.81 | 47.06 | bins=128, threshold=0.19 |
84+
| 66.19 | 80.69 | 68.92 | 48.97 | bins=128, threshold=0.22 |
85+
| 66.17 | 79.53 | 68.88 | 50.10 | bins=128, threshold=0.24 |
86+
87+
## detect-threshold
88+
89+
> Note: `detect-threshold` detects **fades** (fade to/from black), not hard cuts. These datasets' ground truth is hard cuts, so the hard-cut F1 below is expectedly near zero. It is included to validate the sweep pipeline end-to-end, not as a meaningful hard-cut accuracy result.
90+
91+
**Best cell per dataset (by F1@1):**
92+
93+
| Dataset | F1@1 | Prec@1 | Rec@1 | F1@0 | Params |
94+
| --------- | ---- | ------ | ----- | ---- | ------------------------------- |
95+
| BBC | 0.79 | 2.89 | 0.45 | 0.32 | min_scene_len=0.2, threshold=19 |
96+
| AutoShot | 3.98 | 51.09 | 2.07 | 3.14 | min_scene_len=0.4, threshold=20 |
97+
| ClipShots | 1.75 | 6.21 | 1.02 | 0.18 | min_scene_len=0, threshold=10 |
98+
99+
**Best params averaged across all datasets (mean F1@1):**
100+
101+
| Mean F1@1 | BBC | AutoShot | ClipShots | Params |
102+
| --------- | ---- | -------- | --------- | ------------------------------- |
103+
| 2.07 | 0.77 | 3.90 | 1.55 | min_scene_len=0, threshold=19 |
104+
| 2.06 | 0.73 | 3.98 | 1.48 | min_scene_len=0, threshold=20 |
105+
| 2.02 | 0.79 | 3.90 | 1.37 | min_scene_len=0.2, threshold=19 |
106+
| 2.00 | 0.71 | 3.98 | 1.30 | min_scene_len=0.2, threshold=20 |
107+
| 1.96 | 0.69 | 3.90 | 1.28 | min_scene_len=0.4, threshold=19 |

‎benchmark/__main__.py‎

Lines changed: 22 additions & 97 deletions
Original file line numberDiff line numberDiff line change
@@ -20,40 +20,25 @@
2020
from __future__ import annotations
2121

2222
import argparse
23-
import json
24-
import math
2523
import time
2624
from pathlib import Path
27-
from typing import Any
2825

2926
from tqdm import tqdm
3027

31-
from benchmark.dataset import DATASETS, Dataset, resolve_dataset
32-
from benchmark.evaluator import BenchmarkResult, Frames, Prediction, evaluate
33-
from scenedetect import (
34-
AVAILABLE_BACKENDS,
35-
AdaptiveDetector,
36-
ContentDetector,
37-
HashDetector,
38-
HistogramDetector,
39-
ThresholdDetector,
40-
detect,
28+
from benchmark._common import (
29+
DEFAULT_BACKEND,
30+
DETECTORS,
31+
FADE_HEADER,
32+
HARD_HEADER,
33+
fade_row,
34+
hard_row,
35+
parse_tolerances,
36+
render_table,
37+
write_json,
4138
)
42-
43-
_DEFAULT_BACKEND = "opencv"
44-
45-
_DETECTORS: dict[str, type] = {
46-
"detect-adaptive": AdaptiveDetector,
47-
"detect-content": ContentDetector,
48-
"detect-hash": HashDetector,
49-
"detect-hist": HistogramDetector,
50-
"detect-threshold": ThresholdDetector,
51-
}
52-
53-
54-
def _parse_tolerances(spec: str) -> tuple[Frames, ...]:
55-
"""Parse ``"0,1,5"`` into ``(0, 1, 5)``. Blank entries (e.g. trailing comma) are dropped."""
56-
return tuple(int(x.strip()) for x in spec.split(",") if x.strip())
39+
from benchmark.dataset import DATASETS, Dataset, resolve_dataset
40+
from benchmark.evaluator import BenchmarkResult, Prediction, evaluate
41+
from scenedetect import AVAILABLE_BACKENDS, detect
5742

5843

5944
def _run_predictions(
@@ -62,7 +47,7 @@ def _run_predictions(
6247
backend: str,
6348
) -> dict[Path, Prediction]:
6449
"""Detect cuts for every video in ``dataset`` and return predictions keyed by path."""
65-
detector_cls = _DETECTORS[detector_name]
50+
detector_cls = DETECTORS[detector_name]
6651
predictions: dict[Path, Prediction] = {}
6752
for sample in tqdm(dataset, desc=detector_name):
6853
start = time.time()
@@ -76,77 +61,17 @@ def _run_predictions(
7661
return predictions
7762

7863

79-
# --------------------------------------------------------------------- #
80-
# Output formatting
81-
# --------------------------------------------------------------------- #
82-
83-
84-
def _fmt_pct(value: float, count: int) -> str:
85-
"""Percentage, or ``n/a`` when the underlying class has zero events."""
86-
return "n/a" if count == 0 else f"{value * 100:.2f}"
87-
88-
89-
def _fmt_offset(value: float) -> str:
90-
return "n/a" if math.isnan(value) else f"{value:.3f}"
91-
92-
93-
def _render_table(header: list[str], rows: list[list[str]]) -> str:
94-
widths = [max(len(header[i]), *(len(r[i]) for r in rows)) for i in range(len(header))]
95-
sep = "| " + " | ".join("-" * w for w in widths) + " |"
96-
header_line = "| " + " | ".join(h.ljust(w) for h, w in zip(header, widths, strict=True)) + " |"
97-
body = [
98-
"| " + " | ".join(c.ljust(w) for c, w in zip(r, widths, strict=True)) + " |" for r in rows
99-
]
100-
return "\n".join([header_line, sep, *body])
101-
102-
103-
_HARD_HEADER = ["Tolerance", "Precision", "Recall", "F1", "Offset", "Elapsed"]
104-
_FADE_HEADER = ["Tolerance", "Precision", "Recall", "F1"]
105-
106-
107-
def _hard_row(result: BenchmarkResult) -> list[str]:
108-
hard = result.hard_cuts
109-
hard_predictions = hard.matched + hard.false_positives
110-
hard_events = hard.matched + hard.missed
111-
return [
112-
str(result.tolerance),
113-
_fmt_pct(hard.precision, hard_predictions),
114-
_fmt_pct(hard.recall, hard_events),
115-
_fmt_pct(hard.f1, hard_events),
116-
_fmt_offset(result.mean_abs_offset_hard_cuts),
117-
f"{result.elapsed_mean:.2f}",
118-
]
119-
120-
121-
def _fade_row(result: BenchmarkResult) -> list[str]:
122-
fades = result.fades
123-
fade_predictions = fades.matched + fades.false_positives
124-
fade_events = fades.matched + fades.missed
125-
return [
126-
str(result.tolerance),
127-
_fmt_pct(fades.precision, fade_predictions),
128-
_fmt_pct(fades.recall, fade_events),
129-
_fmt_pct(fades.f1, fade_events),
130-
]
131-
132-
13364
def _print_results(
13465
detector: str,
13566
dataset_name: str,
13667
dataset: Dataset,
13768
results: list[BenchmarkResult],
13869
) -> None:
13970
print(f"\n## {detector} on {dataset_name} (hard cuts)\n")
140-
print(_render_table(_HARD_HEADER, [_hard_row(r) for r in results]))
71+
print(render_table(HARD_HEADER, [hard_row(r) for r in results]))
14172
if "fade" in dataset.event_types:
14273
print(f"\n## {detector} on {dataset_name} (fades)\n")
143-
print(_render_table(_FADE_HEADER, [_fade_row(r) for r in results]))
144-
145-
146-
def _write_json(out_path: str, payload: dict[str, Any]) -> None:
147-
with open(out_path, "w") as f:
148-
json.dump(payload, f, indent=2, default=str)
149-
print(f"\nWrote results to {out_path}")
74+
print(render_table(FADE_HEADER, [fade_row(r) for r in results]))
15075

15176

15277
# --------------------------------------------------------------------- #
@@ -167,8 +92,8 @@ def create_parser() -> argparse.ArgumentParser:
16792
"--detector",
16893
type=str,
16994
required=True,
170-
choices=list(_DETECTORS.keys()),
171-
help=f"Detector name. One of: {', '.join(_DETECTORS.keys())}.",
95+
choices=list(DETECTORS.keys()),
96+
help=f"Detector name. One of: {', '.join(DETECTORS.keys())}.",
17297
)
17398
parser.add_argument(
17499
"--dataset-root",
@@ -183,10 +108,10 @@ def create_parser() -> argparse.ArgumentParser:
183108
parser.add_argument(
184109
"--backend",
185110
type=str,
186-
default=_DEFAULT_BACKEND,
111+
default=DEFAULT_BACKEND,
187112
choices=sorted(AVAILABLE_BACKENDS.keys()),
188113
help=(
189-
f"Video decoding backend (default: {_DEFAULT_BACKEND}). Override to compare "
114+
f"Video decoding backend (default: {DEFAULT_BACKEND}). Override to compare "
190115
"detector output across backends, e.g. opencv vs pyav."
191116
),
192117
)
@@ -223,7 +148,7 @@ def create_parser() -> argparse.ArgumentParser:
223148

224149
def main() -> None:
225150
args = create_parser().parse_args()
226-
tolerances = _parse_tolerances(args.tolerance)
151+
tolerances = parse_tolerances(args.tolerance)
227152
dataset = resolve_dataset(args.dataset, args.dataset_root)
228153
if len(dataset) == 0:
229154
raise SystemExit(
@@ -240,7 +165,7 @@ def main() -> None:
240165

241166
_print_results(args.detector, args.dataset, dataset, results)
242167
if args.out:
243-
_write_json(
168+
write_json(
244169
args.out,
245170
{
246171
"detector": args.detector,

0 commit comments

Comments
 (0)