dcase2026_task2_evaluator

The dcase2026_task2_evaluator is a script for calculating the AUC, pAUC, precision, recall, and F1 scores from the anomaly score list for the evaluation dataset in DCASE 2026 Challenge Task 2 "Noise-aware Unsupervised Anomalous Sound Detection for Machine Condition Monitoring"

https://dcase.community/challenge2026/task-first-shot-unsupervised-anomalous-sound-detection-for-machine-condition-monitoring

Description

The dcase2026_task2_evaluator consists of two scripts:

• dcase2026_task2_evaluator.py
  • This script outputs the AUC and pAUC scores by using:
    • Ground truth of the normal and anomaly labels
    • Anomaly scores for each wave file listed in the csv file for each machine type, section, and domain
    • Detection results for each wave file listed in the csv file for each machine type, section, and domain
• 03_evaluation_eval_data.sh
  • This script execute dcase2026_task2_evaluator.py.

Usage

1. Clone repository

Clone this repository from Github.

2. Prepare data

• Anomaly scores
  • Generate csv files anomaly_score_<machine_type>_section_<section_index>_test.csv and decision_result_<machine_type>_section_<section_index>_test.csv or anomaly_score_DCASE2026T2<machine_type>_section_<section>_test_seed<seed><tag>_Eval.csv and decision_result_DCASE2026T2<machine_type>_section_<section>_test_seed<seed><tag>_Eval.csv by using a system for the evaluation dataset. (The format information is described here.)
• Rename the directory containing the csv files to a team name
• Move the directory into ./teams/

3. Check directory structure

• ./dcase2026_task2_evaluator
  • /dcase2026_task2_evaluator.py
  • /03_evaluation_eval_data.sh
  • /ground_truth_attributes
    • ground_truth_BlowerDustCollector_section_00_test.csv
    • ground_truth_Sander_section_00_test.csv
    • ...
  • /ground_truth_data
    • ground_truth_BlowerDustCollector_section_00_test.csv
    • ground_truth_Sander\section_section_00_test.csv
    • ...
  • /ground_truth_domain
    • ground_truth_BlowerDustCollector_section_00_test.csv
    • ground_truth_Sander_section_00_test.csv
    • ...
  • /teams
    • /<team_name_1>
      • /<system_name_1>
        • anomaly_score_BlowerDustCollector_section_00_test.csv
        • anomaly_score_Sander_section_00_test.csv
        • ...
        • decision_result_ToothBrush_section_00_test.csv
        • decision_result_ToyDrone_section_00_test.csv
      • /<system_name_2>
        • anomaly_score_DCASE2026T2BlowerDustCollector_section_00_test_seed<--seed><--tag>_Eval.csv
        • anomaly_score_DCASE2026T2Sander_section_00_test_seed<--seed><--tag>_Eval.csv
        • ...
        • decision_result_DCASE2026T2ToothBrush_section_00_test_seed<--seed><--tag>_Eval.csv
        • decision_result_DCASE2026T2ToyDrone_section_00_test_seed<--seed><--tag>_Eval.csv
    • /<team_name_2>
      • /<system_name_3>
        • anomaly_score_BlowerDustCollector_section_00_test.csv
        • anomaly_score_Sander_section_00\test.csv
        • ...
        • decision_result_ToothBrush_section_00_test.csv
        • decision_result_ToyDrone_section_00_test.csv
    • ...
  • /teams_result
    • <system_name_1>_result.csv
    • <system_name_2>_result.csv
    • <system_name_3>_result.csv
    • ...
  • /teams_additional_result *out_all==True
    • teams_official_score.csv
    • teams_official_score_paper.csv
    • teams_section_00_auc.csv
    • teams_section_00_score.csv
    • /<system_name_1>
      • official_score.csv
      • <system_name_1>_BlowerDustCollector_section_00_anm_score.png
      • ...
      • <system_name_1>_ToyDrone_section_00_anm_score.png
    • /<system_name_2>
      • official_score.csv
      • <system_name_2>_BlowerDustCollector_section_00_anm_score.png
      • ...
      • <system_name_2>_ToyDrone_section_00_anm_score.png
    • /<system_name_3>
      • official_score.csv
      • <system_name_3>_BlowerDustCollector_section_00_anm_score.png
      • ...
      • <system_name_3>_ToyDrone_section_00_anm_score.png
    • ...
  • /tools
    • plot_anm_score.py
    • test_plots.py
  • /README.md

4. Change parameters

The parameters are defined in the script dcase2026_task2_evaluator.py as follows.

• MAX_FPR
  • The FPR threshold for pAUC : default 0.1
• --result_dir
  • The output directory : default ./teams_result/
• --teams_root_dir
  • Directory containing team results. : default ./teams/
• --dir_depth
  • What depth to search --teams_root_dir using glob. : default 2
  • If --dir_depth=2, then glob.glob(<teams_root_dir>/*/*)
• --tag
  • File name tag. : default _id(0_)
  • If using filename is DCASE2026 baseline style, change parameters as necessary.
• --seed
  • Seed used during train. : default 13711
  • If using filename is DCASE2026 baseline style, change parameters as necessary.
• --out_all
  • If this parameter is True, export supplemental data. : default False
• --additional_result_dir
  • The output additional results directory. : default ./teams_additional_result/
  • Used when --out_all==True.

5. Run script

Run the script dcase2026_task2_evaluator.py

$ python dcase2026_task2_evaluator.py

or

$ bash 03_evaluation_eval_data.sh

The script dcase2026_task2_evaluator.py calculates the AUC, pAUC, precision, recall, and F1 scores for each machine type, section, and domain and output the calculated scores into the csv files (<system_name_1>_result.csv, <system_name_2>_result.csv, ...) in --result_dir (default: ./teams_result/). If --out_all=True, each team results are then aggregated into a csv file (teams_official_score.csv, teams_official_score_paper.csv) in --additional_result_dir (default: ./teams_additional_result).

6. Check results

You can check the AUC, pAUC, precision, recall, and F1 scores in the <system_name_N>_result.csv in --result_dir. The AUC, pAUC, precision, recall, and F1 scores for each machine type, section, and domain are listed as follows:

<section_name_N>_result.csv

BlowerDustCollector
section,AUC (all),AUC (source),AUC (target),pAUC,precision (source),precision (target),recall (source),recall (target),F1 score (source),F1 score (target)
00,0.7893000000000001,0.6948000000000001,0.8837999999999999,0.6178947368421053,0.5157894736842106,0.5681818181818182,0.98,1.0,0.6758620689655174,0.7246376811594203
,,AUC,pAUC,precision,recall,F1 score
arithmetic mean,,0.7893,0.6178947368421053,0.5419856459330143,0.99,0.7002498750624688
harmonic mean,,0.7779858608893957,0.6178947368421053,0.5407194879717503,0.98989898989899,0.6994005138452756
source harmonic mean,,0.6948000000000001,0.6178947368421053,0.5157894736842106,0.98,0.6758620689655174
target harmonic mean,,0.8837999999999999,0.6178947368421053,0.5681818181818182,1.0,0.7246376811594204

...

ToyDrone
section,AUC (all),AUC (source),AUC (target),pAUC,precision (source),precision (target),recall (source),recall (target),F1 score (source),F1 score (target)
00,0.5903,0.7126000000000001,0.468,0.5510526315789473,0.5853658536585366,0.524390243902439,0.48,0.86,0.5274725274725274,0.6515151515151516
,,AUC,pAUC,precision,recall,F1 score
arithmetic mean,,0.5903,0.5510526315789473,0.5548780487804879,0.6699999999999999,0.5894938394938395
harmonic mean,,0.5649615449771302,0.5510526315789473,0.5532028946663093,0.6161194029850746,0.582968507272984
source harmonic mean,,0.7126000000000001,0.5510526315789473,0.5853658536585366,0.48,0.5274725274725274
target harmonic mean,,0.468,0.5510526315789473,0.524390243902439,0.86,0.6515151515151516

,,AUC,pAUC,precision,recall,F1 score
"arithmetic mean over all machine types, sections, and domains",,0.6367,0.5713684210526317,0.5332265500685649,0.852,0.6463695738181222
"harmonic mean over all machine types, sections, and domains",,0.6138396081093476,0.568731371472361,0.5314162750580497,0.8108326266667873,0.6420413585067419
"source harmonic mean over all machine types, sections, and domains",,0.6606242657626471,0.568731371472361,0.5371219163581458,0.7326726867597045,0.6198397073185856
"target harmonic mean over all machine types, sections, and domains",,0.5732431794118468,0.568731371472361,0.5258305773466164,0.9076598155637651,0.6658925476064297

official score,,0.5980289616592438
official score ci95,,2.6619662647486664e-05

Aggregated results for each baseline are listed as follows:

System,metric,h-mean,a-mean,ToyDrone,ToothBrush,SewingMachine,BlowerDustCollector,Sander
baseline_MAHALA,AUC (source),0.6085641541440189,0.62908,0.7275999999999999,0.4398,0.6033999999999999,0.6766,0.698
baseline_MAHALA,AUC (target),0.4985220763401413,0.53948,0.5428,0.394,0.43760000000000004,0.872,0.45100000000000007
baseline_MAHALA,"pAUC (source, target)",0.5467370545597742,0.5519999999999999,0.5926315789473684,0.5278947368421053,0.5057894736842106,0.6389473684210526,0.49473684210526314
baseline_MAHALA,TOTAL score,0.5476277174340481,0.57352,,,,,
baseline_MSE,AUC (source),0.6606242657626471,0.6634800000000001,0.7126000000000001,0.5902,0.6484,0.6948000000000001,0.6714
baseline_MSE,AUC (target),0.5732431794118468,0.60992,0.468,0.6434000000000001,0.6142,0.8837999999999999,0.4402
baseline_MSE,"pAUC (source, target)",0.568731371472361,0.5713684210526317,0.5510526315789473,0.5978947368421053,0.5810526315789474,0.6178947368421053,0.5089473684210526
baseline_MSE,TOTAL score,0.5980289616592438,0.614922807017544,,,,,

License

This project is licensed under the terms described in LICENSEv2.1.pdf.

Citation

If you use this system, please cite all the following four papers:

• Tomoya Nishida, Noboru Harada, Daiki Takeuchi, Daisuke Niizumi, Keisuke Imoto, Kota Dohi, Harsh Purohit, Takashi Endo, and Yohei Kawaguchi. Description and discussion on DCASE 2026 challenge task 2: noise-aware unsupervised anomalous sound detection for machine condition monitoring. In arXiv e-prints: 2606.01578, 2026. URL
• Noboru Harada, Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Masahiro Yasuda, and Shoichiro Saito. ToyADMOS2: another dataset of miniature-machine operating sounds for anomalous sound detection under domain shift conditions. In Proceedings of the Detection and Classification of Acoustic Scenes and Events Workshop (DCASE), 1–5. Barcelona, Spain, November 2021. URL
• Kota Dohi, Tomoya Nishida, Harsh Purohit, Ryo Tanabe, Takashi Endo, Masaaki Yamamoto, Yuki Nikaido, and Yohei Kawaguchi. MIMII DG: sound dataset for malfunctioning industrial machine investigation and inspection for domain generalization task. In Proceedings of the 7th Detection and Classification of Acoustic Scenes and Events 2022 Workshop (DCASE2022). Nancy, France, November 2022. URL
• Noboru Harada, Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, and Masahiro Yasuda. First-shot anomaly detection for machine condition monitoring: a domain generalization baseline. Proceedings of 31st European Signal Processing Conference (EUSIPCO), pages 191–195, 2023. URL