Multilingual Speech Evaluation Dataset

Repository: https://github.com/deepeshaiml/multilingual_speech_evaluation

1. Overview

This project downloads, organizes, and prepares multilingual speech data for evaluating speech-processing systems.

The dataset is prepared for the following evaluation tasks:

1. Speech Language Identification, or Speech LID
2. Speaker Diarization
3. Speaker Embedding and Speaker Verification
4. Speaker Gender Classification
5. Speech-to-Text Transcription, Mode A
6. Multilingual Meeting Transcription, Mode B

The original audio is downloaded from public Hugging Face datasets and then reorganized into task-specific evaluation folders.

The current source datasets are:

• google/fleurs
• facebook/omnilingual-asr-corpus

Additional internal source dataset used for speech-tone evaluation:

• synergy-ai-datasets/dataset/multilingual_emotional (MinIO/S3)
• MinIO browser path: https://s3.presight.dev/minio/ui/browser/synergy-ai-datasets/dataset%2Fmultilingual_emotional%2F

1.1 Client-facing ASR delivery pack

For client sharing, use the ASR-only delivery artifacts:

• CLIENT_DELIVERY.md for the human-readable package, summary metrics, and dataset provenance
• delivery_manifest.json for the machine-readable package manifest

Canonical ASR runs in the current client pack:

• Mode A single utterance: 20260805T102821Z
• Mode B multilingual meetings: 20260804T061343Z

Note: this repository contains broader speech-task tooling, but the current client delivery pack is intentionally scoped to ASR.

The preparation pipeline is designed to:

• Avoid downloading entire datasets unnecessarily
• Stream only a bounded number of source rows
• Select metadata-diverse samples
• Preserve source metadata
• Create reproducible evaluation sets
• Support multiple languages and regional dialects
• Generate speaker and segment ground-truth annotations
• Continue processing when an individual language or dataset configuration fails

---

2. Project Scripts

The project uses three main scripts.

2.1 Download script

download_speech_dataset.py

Purpose:

• Connects to Hugging Face
• Streams speech datasets
• Selects bounded and metadata-diverse samples
• Downloads audio files
• Preserves transcripts and source metadata
• Creates one folder for each language track

Typical location:

download_speech_dataset.py

---

2.2 Remaining evaluation-task builder

build_speech_evaluation_tasks.py

Purpose:

• Creates Speech LID evaluation data
• Creates speaker diarization evaluation recordings
• Creates speaker enrollment, test, and verification-trial data
• Creates speaker gender evaluation data
• Creates single-utterance transcription Mode A data

The script preserves an existing speech_transcription/mode_b_multilingual_meeting directory.

---

2.3 Multilingual meeting builder

Example filename:

build_multilingual_meetings.py

Purpose:

• Creates multi-speaker recordings
• Creates multi-turn recordings
• Includes multiple languages in the same recording
• Generates meeting-style speech sequences
• Generates RTTM diarization labels
• Generates segment-level speaker, language, gender, and transcription labels
• Creates transcription Mode B evaluation data

The meeting audio is produced by concatenating downloaded speech clips with controlled silence between turns.

It does not generate new voices using text-to-speech.

---

3. Supported Languages and Dialects

The downloader currently defines the following evaluation tracks.

3.1 South Asian languages

Track	Source dataset	Source configuration
English	google/fleurs	en_us
Bengali	google/fleurs	bn_in
Gujarati	google/fleurs	gu_in
Hindi	google/fleurs	hi_in
Kannada	google/fleurs	kn_in
Kashmiri	facebook/omnilingual-asr-corpus	kas_Arab
Malayalam	google/fleurs	ml_in
Marathi	google/fleurs	mr_in
Nepali	google/fleurs	ne_np
Punjabi	google/fleurs	pa_in
Pakistani Punjabi	facebook/omnilingual-asr-corpus	pnb_Arab
Tamil	google/fleurs	ta_in
Telugu	google/fleurs	te_in
Urdu	google/fleurs	ur_pk
Sindhi	google/fleurs	sd_in
Eastern Balochi	facebook/omnilingual-asr-corpus	bgp_Arab

---

3.2 Pashto language variants

The Pashto track may use multiple configurations.

Variant	Source dataset	Source configuration
Southern Pashto	facebook/omnilingual-asr-corpus	pbt_Arab
Northern Pashto	facebook/omnilingual-asr-corpus	pbu_Arab
Central Pashto	facebook/omnilingual-asr-corpus	pst_Arab
Pashto, Afghanistan	google/fleurs	ps_af

These configurations are combined under the evaluation track:

Pashto

---

3.3 Other languages

Track	Source dataset	Source configuration
Tagalog	google/fleurs	fil_ph
Persian	google/fleurs	fa_ir
Chinese	google/fleurs	cmn_hans_cn
Hebrew	google/fleurs	he_il

---

3.4 Arabic dialects

Track	Source dataset	Source configuration
Gulf-Khaleeji Arabic	facebook/omnilingual-asr-corpus	afb_Arab
Saudi Najdi Arabic	facebook/omnilingual-asr-corpus	ars_Arab
Saudi Hijazi Arabic	facebook/omnilingual-asr-corpus	acw_Arab
Iraqi Arabic	facebook/omnilingual-asr-corpus	acm_Arab
North Iraqi Arabic	facebook/omnilingual-asr-corpus	ayp_Arab
Levantine Arabic	facebook/omnilingual-asr-corpus	apc_Arab
Egyptian Arabic	facebook/omnilingual-asr-corpus	arz_Arab
Omani Arabic	facebook/omnilingual-asr-corpus	acx_Arab
Dhofari-Omani Arabic	facebook/omnilingual-asr-corpus	adf_Arab
Bahrani Arabic	facebook/omnilingual-asr-corpus	abv_Arab
Moroccan Arabic	facebook/omnilingual-asr-corpus	ary_Arab
Algerian Arabic	facebook/omnilingual-asr-corpus	arq_Arab
Tunisian Arabic	facebook/omnilingual-asr-corpus	aeb_Arab
Libyan Arabic	facebook/omnilingual-asr-corpus	ayl_Arab

---

4. Environment Requirements

4.1 Python

Recommended:

Python 3.10 or later

Check the installed Python version:

python3 --version

---

4.2 Python packages

Install the required packages:

python3 -m pip install \
  datasets \
  huggingface_hub \
  fsspec \
  httpx \
  truststore \
  tqdm

Depending on the local environment, additional audio-related dependencies may also be needed by Hugging Face datasets.

---

4.3 FFmpeg

FFmpeg is required for:

• Converting source audio to WAV
• Converting audio to mono
• Resampling audio to 16 kHz
• Trimming long utterances
• Creating diarization recordings
• Creating multilingual meeting recordings

Install FFmpeg on macOS:

brew install ffmpeg

Confirm installation:

ffmpeg -version

---

4.4 Hugging Face authentication

Some datasets or configurations may require authentication.

Set the Hugging Face token:

export HF_TOKEN="hf_your_token_here"

The scripts read the token from:

HF_TOKEN

Do not store a private token directly inside the source code.

---

4.5 Corporate SSL certificates

The downloader uses truststore so that HTTP clients can use certificates trusted by the macOS system Keychain.

This is useful on corporate systems where HTTPS traffic may pass through a company proxy or certificate authority.

---

5. Downloading the Source Data

Run the downloader from the project directory:

cd /Users/deepesh.agrawal/Documents/workdir/sdk_and_ray

Example:

python3 download_speech_dataset.py \
  --samples 100 \
  --scan 500 \
  --output source_speech_data \
  --seed 42

Parameters:

Parameter	Description
--samples	Maximum number of saved samples per language track
--scan	Maximum number of source rows scanned per language track
--output	Output directory
--seed	Random seed used for reproducible selection
--tracks	Optional subset of language tracks

The scan value must be greater than or equal to the sample value.

Example for selected languages:

python3 download_speech_dataset.py \
  --samples 100 \
  --scan 500 \
  --output source_speech_data \
  --tracks \
    English \
    Hindi \
    Bengali \
    "Gulf-Khaleeji Arabic"

Language names containing spaces must be quoted.

---

6. Download Strategy

The downloader uses Hugging Face streaming mode:

streaming=True

This prevents the script from intentionally downloading a complete dataset.

For every language track, the script:

1. Resolves the configured dataset and language configuration.
2. Selects a preferred dataset split.
3. Streams up to the configured scan limit.
4. Keeps rows that contain a usable transcript.
5. Preserves the encoded audio object.
6. Selects metadata-diverse samples.
7. Saves selected audio files.
8. Writes per-language metadata.
9. Updates the global manifest and status file.

For FLEURS, the preferred split order is:

1. validation
2. test
3. train
4. dev

For Omnilingual, the preferred split order is:

1. train
2. validation
3. dev
4. test

---

7. Metadata-Diverse Sample Selection

The downloader attempts to avoid selecting highly repetitive data.

It prefers underrepresented values from available metadata fields such as:

• Speaker ID
• Speaker key
• Gender
• Age
• Age group
• Dialect
• Accent
• Region
• Locale
• Country
• Language
• Source configuration
• ISO language code
• Script code
• Glottocode
• Duration range
• Prompt ID

It also strongly prefers unique transcripts.

The available metadata differs by dataset and configuration. A missing field does not cause the downloader to fail.

Metadata diversity improves evaluation coverage, but it does not guarantee perfectly balanced representation.

---

8. Initial Downloaded Dataset Structure

After downloading, the source directory generally looks like this:

source_speech_data/
├── English/
│   ├── audio/
│   │   ├── 0000_en_us.wav
│   │   ├── 0001_en_us.wav
│   │   └── ...
│   └── metadata.jsonl
│
├── Hindi/
│   ├── audio/
│   └── metadata.jsonl
│
├── Bengali/
│   ├── audio/
│   └── metadata.jsonl
│
├── Gulf-Khaleeji_Arabic/
│   ├── audio/
│   └── metadata.jsonl
│
├── manifest.jsonl
└── status.json

The exact audio extension can differ by source dataset.

Possible extensions include:

• .wav
• .flac
• .mp3
• .ogg
• Other source-supported formats

---

9. Source Metadata Files

9.1 Per-language metadata

Each language folder contains:

metadata.jsonl

JSONL means JSON Lines. Each line is one independent JSON object.

Example:

{
  "track": "Hindi",
  "source_dataset": "google/fleurs",
  "source_config": "hi_in",
  "source_split": "validation",
  "source_stream_index": 12,
  "speaker_id": 104,
  "speaker_key": "google/fleurs/hi_in:104",
  "gender": 0,
  "gender_name": "male",
  "transcript": "Reference transcription",
  "audio": "audio/0000_hi_in.wav",
  "original_audio_path": "source audio location",
  "audio_file_size_bytes": 152430
}

The exact fields vary by source dataset.

---

9.2 Global manifest

The root file:

manifest.jsonl

contains all successfully saved language rows in one file.

This is useful for:

• Global dataset analysis
• Building unified indexes
• Data validation
• Filtering by language
• Filtering by speaker
• Filtering by gender
• Preparing additional tasks

---

9.3 Download status

The root file:

status.json

contains processing information for every language track.

Example information:

• Requested samples
• Candidate count
• Successfully saved samples
• Dataset configurations used
• Missing configurations
• Source errors
• Audio-saving errors
• Completed, partial, or failed state

---

10. Building Evaluation Data

Example command:

python3 build_speech_evaluation_tasks.py \
  --input source_speech_data \
  --output evaluation_dataset \
  --lid-per-language 100 \
  --gender-per-class 100 \
  --stt-per-language 100 \
  --embedding-speakers 100 \
  --embedding-enrollment 1 \
  --embedding-tests 2 \
  --diarization-recordings 100 \
  --diarization-min-speakers 2 \
  --diarization-max-speakers 4 \
  --diarization-turns-min 8 \
  --diarization-turns-max 16 \
  --diarization-min-turns-per-speaker 2 \
  --overwrite

The --overwrite option removes and rebuilds output directories for the selected tasks.

An existing speech_transcription/mode_b_multilingual_meeting directory is not removed by the remaining-task builder.

---

11. Complete Evaluation Dataset Structure

evaluation_dataset/
├── speech_language_identification/
│   ├── audio/
│   │   ├── English/
│   │   ├── Hindi/
│   │   ├── Bengali/
│   │   └── ...
│   └── metadata.jsonl
│
├── speaker_diarization/
│   ├── audio/
│   │   ├── diar_00000.wav
│   │   ├── diar_00001.wav
│   │   └── ...
│   ├── rttm/
│   │   ├── diar_00000.rttm
│   │   └── ...
│   ├── segments/
│   │   ├── diar_00000.json
│   │   └── ...
│   ├── metadata.jsonl
│   ├── all_segments.jsonl
│   └── failures.json
│
├── speaker_verification/
│   ├── enrollment/
│   │   ├── spk_0123456789abcdef/
│   │   └── ...
│   ├── test/
│   │   ├── spk_0123456789abcdef/
│   │   └── ...
│   ├── speakers.jsonl
│   ├── enrollment.jsonl
│   ├── test.jsonl
│   └── trials.jsonl
│
├── speaker_gender_classification/
│   ├── audio/
│   │   ├── male/
│   │   ├── female/
│   │   └── other/
│   └── metadata.jsonl
│
├── speech_transcription/
│   ├── mode_a_single_utterance/
│   │   ├── audio/
│   │   │   ├── English/
│   │   │   ├── Hindi/
│   │   │   └── ...
│   │   └── metadata.jsonl
│   │
│   └── mode_b_multilingual_meeting/
│       ├── audio/
│       │   ├── meeting_00000.wav
│       │   └── ...
│       ├── rttm/
│       │   ├── meeting_00000.rttm
│       │   └── ...
│       ├── segments/
│       │   ├── meeting_00000.json
│       │   └── ...
│       ├── transcripts/
│       │   ├── meeting_00000.txt
│       │   └── ...
│       ├── metadata.jsonl
│       ├── all_segments.jsonl
│       └── status.json
│
└── remaining_tasks_status.json

---

12. Speech Language Identification Dataset

12.1 Purpose

The Speech LID dataset evaluates whether a model can identify the language or dialect spoken in an audio file.

Each audio file contains one utterance and has one reference language label.

---

12.2 Structure

speech_language_identification/
├── audio/
│   ├── English/
│   ├── Hindi/
│   ├── Bengali/
│   ├── Gulf-Khaleeji_Arabic/
│   └── ...
└── metadata.jsonl

---

12.3 Metadata format

Example:

{
  "sample_id": "utt_b4f8c1a62097b927a231",
  "audio": "audio/Hindi/utt_b4f8c1a62097b927a231.wav",
  "language": "Hindi",
  "language_code": "hi_in",
  "text": "Reference transcription",
  "speaker_signature": "spk_9b00b36a177a117e",
  "gender": "male",
  "source_dataset": "google/fleurs",
  "source_config": "hi_in",
  "source_split": "validation",
  "source_stream_index": 12
}

---

12.4 How it is useful

This dataset can be used to measure:

• Language classification accuracy
• Dialect classification accuracy
• Confusion between related languages
• Confusion between Arabic dialects
• Performance by audio duration
• Performance by speaker
• Performance by gender metadata
• Performance by dataset source

Recommended metrics:

• Overall accuracy
• Macro-averaged F1 score
• Per-language precision
• Per-language recall
• Per-language F1 score
• Confusion matrix

Macro-averaged metrics are important when language counts are not perfectly balanced.

---

13. Speaker Diarization Dataset

13.1 Purpose

Speaker diarization answers:

Who spoke when?

The generated diarization recordings contain:

• Multiple speakers
• Multiple turns
• One language per recording
• Silence between turns
• No overlapping speech
• Ground-truth speaker timestamps
• Ground-truth speaker signatures

Keeping these recordings monolingual helps isolate diarization performance from multilingual ASR and LID performance.

Multilingual diarization is covered by transcription Mode B.

---

13.2 Structure

speaker_diarization/
├── audio/
├── rttm/
├── segments/
├── metadata.jsonl
├── all_segments.jsonl
└── failures.json

---

13.3 Audio format

Generated diarization audio normally uses:

• WAV container
• PCM signed 16-bit audio
• Mono channel
• 16,000 Hz sample rate

Example:

audio/diar_00000.wav

---

13.4 RTTM format

Example:

SPEAKER diar_00000 1 0.000000 4.820000 <NA> <NA> spk_9b00b36a177a117e <NA> <NA>
SPEAKER diar_00000 1 5.310000 4.430000 <NA> <NA> spk_c2618c9007f40164 <NA> <NA>

Important RTTM fields:

Position	Meaning
SPEAKER	RTTM record type
diar_00000	Recording ID
1	Channel
0.000000	Segment start time
4.820000	Segment duration
spk_...	Reference speaker signature

---

13.5 Segment JSON

Example:

{
  "recording_id": "diar_00000",
  "audio": "audio/diar_00000.wav",
  "segments": [
    {
      "recording_id": "diar_00000",
      "turn_index": 0,
      "start_seconds": 0.0,
      "end_seconds": 4.82,
      "duration_seconds": 4.82,
      "speaker_signature": "spk_9b00b36a177a117e",
      "language": "Hindi",
      "language_code": "hi_in",
      "gender": "male",
      "text": "Reference transcription"
    }
  ]
}

---

13.6 How it is useful

This dataset can be used to measure:

• Speaker-change detection
• Speaker segmentation
• Speaker clustering
• Speaker-count estimation
• Speaker-attributed timing
• Diarization error rate

Recommended metrics:

• Diarization Error Rate, or DER
• Jaccard Error Rate, or JER
• Missed speech
• False-alarm speech
• Speaker confusion
• Speaker-count accuracy

Because the generated data does not contain overlap, it is primarily suitable for non-overlap diarization evaluation.

It should not be treated as a complete evaluation of overlapping-speech diarization.

---

14. Speaker Embedding and Signature Dataset

14.1 Purpose

This dataset supports speaker embedding and speaker verification evaluation.

It is divided into:

• Enrollment audio
• Test audio
• Speaker metadata
• Positive trials
• Negative trials

---

14.2 Speaker signature meaning

A speaker signature such as:

spk_9b00b36a177a117e

is a deterministic pseudonymous identifier derived from the source speaker key.

It is used as a ground-truth identity label.

It is not:

• A neural-network embedding
• An acoustic feature vector
• A biometric template generated by the model
• A voiceprint computed from the audio

Your speaker-embedding model must process the audio and generate the actual embedding vectors.

---

14.3 Structure

speaker_verification/
├── enrollment/
│   ├── spk_9b00b36a177a117e/
│   └── ...
├── test/
│   ├── spk_9b00b36a177a117e/
│   └── ...
├── speakers.jsonl
├── enrollment.jsonl
├── test.jsonl
└── trials.jsonl

---

14.4 Enrollment metadata

Example:

{
  "speaker_signature": "spk_9b00b36a177a117e",
  "sample_id": "utt_72ac7319ef72817380ac",
  "audio": "enrollment/spk_9b00b36a177a117e/enroll_000_utt_72ac7319ef72817380ac.wav",
  "language": "Hindi",
  "language_code": "hi_in",
  "gender": "male",
  "text": "Reference transcription"
}

---

14.5 Test metadata

Example:

{
  "speaker_signature": "spk_9b00b36a177a117e",
  "sample_id": "utt_93b64edcc6d69a36bc21",
  "audio": "test/spk_9b00b36a177a117e/test_000_utt_93b64edcc6d69a36bc21.wav",
  "language": "Hindi",
  "language_code": "hi_in",
  "gender": "male",
  "text": "Reference transcription"
}

---

14.6 Trial metadata

Positive trial:

{
  "enrollment_speaker_signature": "spk_9b00b36a177a117e",
  "test_audio": "test/spk_9b00b36a177a117e/test_000_utt_93b64edcc6d69a36bc21.wav",
  "test_speaker_signature": "spk_9b00b36a177a117e",
  "target": 1,
  "label": "target"
}

Negative trial:

{
  "enrollment_speaker_signature": "spk_c2618c9007f40164",
  "test_audio": "test/spk_9b00b36a177a117e/test_000_utt_93b64edcc6d69a36bc21.wav",
  "test_speaker_signature": "spk_9b00b36a177a117e",
  "target": 0,
  "label": "nontarget"
}

---

14.7 How it is useful

This dataset can be used for:

• Speaker verification
• Speaker identification
• Speaker similarity scoring
• Speaker retrieval
• Speaker clustering
• Cross-language speaker comparison

Recommended metrics:

• Equal Error Rate, or EER
• Minimum Detection Cost Function
• False Accept Rate
• False Reject Rate
• Receiver Operating Characteristic
• Precision at K for speaker retrieval
• Top-1 and Top-K speaker-identification accuracy

For a stronger evaluation, enrollment and test utterances should ideally differ in text, recording conditions, and session.

The source datasets may not provide full session information, so session independence cannot always be guaranteed.

---

15. Speaker Gender Dataset

15.1 Purpose

This dataset evaluates speaker gender classification when the source dataset contains a usable gender label.

---

15.2 Structure

speaker_gender_classification/
├── audio/
│   ├── male/
│   ├── female/
│   └── other/
└── metadata.jsonl

---

15.3 Metadata format

Example:

{
  "sample_id": "utt_b4f8c1a62097b927a231",
  "audio": "audio/male/utt_b4f8c1a62097b927a231.wav",
  "gender": "male",
  "language": "Hindi",
  "language_code": "hi_in",
  "speaker_signature": "spk_9b00b36a177a117e",
  "text": "Reference transcription"
}

---

15.4 Important limitation

The preparation script does not infer gender from the audio.

It uses only gender or sex metadata supplied by the source dataset.

Rows without a recognized textual gender label are assigned:

unknown

Unknown rows are excluded from the speaker-gender evaluation folder.

Gender labels can be incomplete, inconsistent, or based on limited source categories. They should not be interpreted as a complete representation of gender identity.

---

15.5 How it is useful

This dataset can be used to measure:

• Gender-label classification accuracy
• Per-language classification accuracy
• Performance differences across languages
• Performance differences across audio durations
• Dataset bias and imbalance

Recommended metrics:

• Accuracy
• Macro F1
• Per-class recall
• Per-class precision
• Confusion matrix
• Balanced accuracy

The dataset should also be reviewed for class imbalance before model comparison.

---

16. Transcription Mode A

16.1 Purpose

Mode A evaluates standard single-utterance speech-to-text.

Each audio file contains one utterance with one reference transcript.

The expected output is text only.

The model is not required to produce:

• Speaker labels
• Diarization
• Gender
• Segment-level LID
• Speaker embeddings

---

16.2 Structure

speech_transcription/
└── mode_a_single_utterance/
    ├── audio/
    │   ├── English/
    │   ├── Hindi/
    │   ├── Bengali/
    │   └── ...
    └── metadata.jsonl

---

16.3 Metadata format

Example:

{
  "sample_id": "utt_b4f8c1a62097b927a231",
  "audio": "audio/Hindi/utt_b4f8c1a62097b927a231.wav",
  "text": "Reference transcription",
  "language": "Hindi",
  "language_code": "hi_in",
  "source_dataset": "google/fleurs",
  "source_config": "hi_in",
  "source_split": "validation",
  "source_stream_index": 12
}

---

16.4 How it is useful

Mode A can be used to evaluate:

• Monolingual ASR
• Multilingual ASR
• Language-specific transcription quality
• Script-specific transcription quality
• Performance on low-resource languages
• Performance on dialectal speech

Recommended metrics:

• Word Error Rate, or WER
• Character Error Rate, or CER
• Normalized WER
• Normalized CER
• Sentence accuracy
• Empty-output rate
• Hallucination rate

CER is especially useful for languages where tokenization and word boundaries are difficult or inconsistent.

Text normalization rules should be documented before reporting WER or CER.

---

17. Transcription Mode B

17.1 Purpose

Mode B evaluates a complete meeting or conversation transcription pipeline.

Each recording may contain:

• Multiple speakers
• Multiple turns
• Multiple languages
• Different speaker genders when metadata is available
• Speaker changes
• Silence between turns

The expected system output may include:

• Speech segments
• Speaker labels
• Speaker embedding signatures or speaker clusters
• Language label for each segment
• Gender label for each segment
• Transcription for each segment

---

17.2 Structure

speech_transcription/
└── mode_b_multilingual_meeting/
    ├── audio/
    ├── rttm/
    ├── segments/
    ├── transcripts/
    ├── metadata.jsonl
    ├── all_segments.jsonl
    └── status.json

---

17.3 Segment format

Example:

{
  "meeting_id": "meeting_00000",
  "turn_index": 1,
  "start_seconds": 5.31,
  "end_seconds": 9.74,
  "duration_seconds": 4.43,
  "speaker_signature": "spk_c2618c9007f40164",
  "language": "English",
  "language_code": "en_us",
  "gender": "female",
  "text": "Reference English transcription",
  "source_dataset": "google/fleurs",
  "source_config": "en_us",
  "source_split": "validation",
  "source_stream_index": 21
}

---

17.4 Human-readable transcript

Example:

[00:00:00.000 - 00:00:04.820] spk_9b00b36a177a117e | Hindi | male: Reference Hindi transcription
[00:00:05.310 - 00:00:09.740] spk_c2618c9007f40164 | English | female: Reference English transcription

---

17.5 How it is useful

Mode B can be used to evaluate an end-to-end meeting-processing system.

Possible components include:

1. Voice activity detection
2. Speaker diarization
3. Speaker embedding extraction
4. Speaker clustering
5. Segment-level language identification
6. Segment-level gender classification
7. Speech-to-text transcription
8. Speaker-attributed transcription

Recommended metrics:

• Diarization Error Rate
• Speaker-attributed WER
• cpWER
• Segment-level LID accuracy
• Segment-level gender accuracy
• Speaker-clustering quality
• Timestamp error
• Segment boundary error
• End-to-end meeting transcription accuracy

---

18. Meeting-Data Limitation

The Mode B recordings are meeting-like rather than naturally recorded meetings.

They are created by concatenating unrelated source utterances.

Therefore:

• The turn order is controlled.
• The speakers and languages are controlled.
• The timestamps are accurate.
• The speaker labels are known.
• The utterances may not be semantically related.
• The recording may not sound like a coherent real conversation.
• There is currently no overlapping speech.
• Room reverberation and shared microphone effects are not naturally represented.
• Speakers were originally recorded under potentially different acoustic conditions.

This dataset is useful for controlled system evaluation, but it should be supplemented with natural meeting recordings for production-level validation.

---

19. Audio Materialization Modes

For single-utterance tasks, the preparation script supports:

copy
hardlink
symlink

19.1 Hard link

Default:

--link-mode hardlink

Advantages:

• Does not duplicate audio data when input and output are on the same filesystem
• Evaluation folders behave like normal files
• Saves disk space

If hard linking fails, the script falls back to copying.

---

19.2 Symbolic link

--link-mode symlink

Advantages:

• Minimal disk usage
• Easy to identify original files

Limitation:

• Moving or deleting the source dataset can break the links

---

19.3 Copy

--link-mode copy

Advantages:

• Evaluation data is independent from the source folder
• Easier to archive or transfer

Limitation:

• Requires more disk space

---

20. Reproducibility

The scripts use a configurable random seed.

Example:

--seed 42

Using the same:

• Input files
• Metadata
• Script version
• Parameters
• Random seed

should produce the same or substantially equivalent selection and ordering.

Reproducibility can be affected when:

• Source datasets change
• Hugging Face configurations change
• Input files are added or removed
• File ordering changes
• Metadata fields change
• FFmpeg versions behave differently
• A previous run partially consumed source pools
• Script logic changes

For formal experiments, record:

• Git commit
• Python version
• FFmpeg version
• Package versions
• Command-line arguments
• Dataset build date
• Input manifest checksum
• Random seed

---

21. Recommended Build Record

For every dataset build, save a configuration record such as:

{
  "build_date": "YYYY-MM-DD",
  "download_script": "download_speech_dataset.py",
  "evaluation_script": "build_speech_evaluation_tasks.py",
  "meeting_script": "build_multilingual_meetings.py",
  "input_directory": "source_speech_data",
  "output_directory": "evaluation_dataset",
  "seed": 42,
  "samples_per_language": 100,
  "scan_per_language": 500,
  "lid_per_language": 100,
  "gender_per_class": 100,
  "stt_per_language": 100,
  "embedding_speakers": 100,
  "embedding_enrollment": 1,
  "embedding_tests": 2,
  "diarization_recordings": 100,
  "meeting_recordings": 100,
  "sample_rate": 16000
}

---

22. Data Validation

Before evaluating a model, validate the prepared dataset.

22.1 Check metadata files

Count JSONL rows:

wc -l evaluation_dataset/speech_language_identification/metadata.jsonl
wc -l evaluation_dataset/speaker_gender_classification/metadata.jsonl
wc -l evaluation_dataset/speech_transcription/mode_a_single_utterance/metadata.jsonl

---

22.2 Check audio files

find evaluation_dataset -type f \
  \( -name "*.wav" -o -name "*.flac" -o -name "*.mp3" \) \
  | wc -l

When using symbolic or hard links, the find options may need adjustment depending on how files are materialized.

---

22.3 Check missing paths

A simple Python check:

import json
from pathlib import Path

root = Path("evaluation_dataset/speech_lid")
manifest = root / "metadata.jsonl"

missing = []

with manifest.open("r", encoding="utf-8") as handle:
    for line in handle:
        row = json.loads(line)
        audio_path = root / row["audio"]

        if not audio_path.exists():
            missing.append(str(audio_path))

print("Missing audio files:", len(missing))

for path in missing[:20]:
    print(path)

---

22.4 Inspect audio properties

Example:

ffprobe \
  -hide_banner \
  evaluation_dataset/speaker_diarization/audio/diar_00000.wav

Expected diarization and Mode B properties usually include:

Audio: pcm_s16le
16000 Hz
mono

---

22.5 Validate RTTM timestamps

For every RTTM segment:

start time >= 0
duration > 0
start time + duration <= recording duration

Speaker signatures in the RTTM file should also exist in the corresponding segment JSON.

---

23. Recommended Evaluation Splitting

The downloaded data is prepared as an evaluation collection, but a strict development/test split may need to be created separately.

For speaker-sensitive tasks, split by speaker rather than by utterance.

Recommended:

development speakers != test speakers

This avoids speaker leakage.

For language identification and transcription, consider splitting by:

• Speaker
• Source dataset
• Source configuration
• Region
• Dialect
• Recording duration

For speaker verification:

• Do not place the same audio clip in enrollment and test.
• Prefer different sessions when session metadata is available.
• Avoid evaluating only same-language trials.
• Include cross-language trials when speakers have multilingual data.

---

24. Evaluation Leakage Risks

Potential leakage can occur when:

• The same audio appears in training and evaluation
• The same speaker appears in development and test unintentionally
• Duplicate transcripts appear across splits
• The evaluation source was included in the model's training data
• The same source recording is copied into more than one task
• Enrollment and test use near-duplicate audio
• File names expose the reference label

The current folder names intentionally expose labels for convenience.

For blind evaluation, create a separate package where:

• Audio files use anonymous IDs
• Reference metadata is stored separately
• The model sees only the audio and task input
• Labels are hidden until scoring

---

25. Suggested Blind Evaluation Layout

Example:

blind_eval/
├── input/
│   ├── audio/
│   │   ├── item_000001.wav
│   │   ├── item_000002.wav
│   │   └── ...
│   └── request.jsonl
│
└── reference/
    └── ground_truth.jsonl

The system receives only:

blind_eval/input

The scoring process uses:

blind_eval/reference

---

26. Error and Status Files

26.1 Download status

source_speech_data/status.json

Tracks download and source errors.

---

26.2 Remaining-task status

evaluation_dataset/remaining_tasks_status.json

Tracks completion for:

• Speech LID
• Diarization
• Speaker embedding/signature
• Speaker gender
• Transcription Mode A

---

26.3 Diarization failures

evaluation_dataset/speaker_diarization/failures.json

Contains errors such as:

• FFmpeg conversion failure
• Invalid audio file
• Audio too short after conversion
• Unexpected WAV properties

---

26.4 Mode B status

evaluation_dataset/speech_transcription/mode_b_multilingual_meeting/status.json

Contains:

• Requested meeting count
• Completed meeting count
• Input statistics
• Source language counts
• Configuration
• Failures
• Stop reason

---

27. Important Dataset Limitations

27.1 Public-dataset dependency

The evaluation data is derived from external public datasets.

Dataset availability, configurations, and metadata may change.

---

27.2 Metadata consistency

Not every source includes:

• Speaker ID
• Gender
• Age
• Region
• Accent
• Dialect
• Session ID

Tasks that require missing metadata may contain fewer samples.

---

27.3 Speaker-label consistency

Speaker IDs are scoped using dataset and configuration identifiers.

This reduces collisions, but it does not prove that two identifiers from different source configurations represent different physical people.

---

27.4 Gender labels

Gender labels are inherited from source metadata.

They are not inferred from audio and may contain limited categories.

---

27.5 Synthetic meeting construction

Diarization and Mode B recordings are produced from concatenated utterances.

They do not fully reproduce:

• Natural meeting acoustics
• Cross-talk
• Overlapping speech
• Background noise
• Shared microphones
• Room impulse responses
• Natural conversational context
• Interruptions
• Backchannels
• Speaker movement

---

27.6 Language labels

Language and dialect labels are based on the configured dataset track.

Some closely related dialects may be difficult to separate acoustically.

Some speakers may also code-switch inside an utterance even when the dataset provides one primary language label.

---

27.7 Transcript normalization

Source transcripts may differ in:

• Punctuation
• Number formatting
• Casing
• Unicode normalization
• Diacritics
• Script conventions
• English loanwords
• Hesitation markers

The evaluation pipeline should define normalization rules before calculating WER or CER.

---

28. Data Licensing and Usage

Before distributing or using the prepared dataset, review the dataset cards and licenses for:

google/fleurs
facebook/omnilingual-asr-corpus

The preparation scripts do not change the original licensing terms.

The generated folders contain reorganized or derived copies of source data and should be handled according to the original dataset terms.

Do not assume that all source configurations have identical redistribution conditions.

Recommended actions:

1. Record the source dataset name for every sample.
2. Preserve source metadata.
3. Preserve original attribution.
4. Review commercial-use restrictions.
5. Review redistribution restrictions.
6. Review personal-data and voice-data requirements.
7. Avoid publishing private Hugging Face access tokens.
8. Confirm legal and organizational approval before external release.

---

29. Privacy and Responsible Use

Speech data may be considered personal or biometric data in some jurisdictions and organizational policies.

Speaker embeddings can be used for speaker recognition and identity comparison.

Recommended safeguards:

• Use pseudonymous speaker signatures
• Do not attempt to recover personal identities
• Restrict access to speaker-verification data
• Store generated embeddings securely
• Avoid using the dataset for surveillance
• Follow applicable privacy and biometric-data policies
• Document retention and deletion procedures
• Avoid making sensitive inferences from voice recordings

---

30. Example End-to-End Workflow

Step 1: Enter the project directory

cd /Users/deepesh.agrawal/Documents/workdir/sdk_and_ray

Step 2: Download source audio

python3 download_speech_dataset.py \
  --samples 100 \
  --scan 500 \
  --output source_speech_data \
  --seed 42

Step 3: Build the five standard tasks

python3 build_speech_evaluation_tasks.py \
  --input source_speech_data \
  --output evaluation_dataset \
  --lid-per-language 100 \
  --gender-per-class 100 \
  --stt-per-language 100 \
  --embedding-speakers 100 \
  --embedding-enrollment 1 \
  --embedding-tests 2 \
  --diarization-recordings 100 \
  --diarization-min-speakers 2 \
  --diarization-max-speakers 4 \
  --diarization-turns-min 8 \
  --diarization-turns-max 16 \
  --overwrite

Step 4: Build multilingual Mode B meetings

python3 build_multilingual_meetings.py \
  --input source_speech_data \
  --output evaluation_dataset \
  --meetings 100 \
  --min-speakers 2 \
  --max-speakers 4 \
  --min-languages 2 \
  --max-languages 4 \
  --turns-min 10 \
  --turns-max 18 \
  --min-turns-per-speaker 2 \
  --gap-min-ms 200 \
  --gap-max-ms 900 \
  --overwrite

Step 5: Review build status

cat source_speech_data/status.json

cat evaluation_dataset/remaining_tasks_status.json

cat evaluation_dataset/speech_transcription/mode_b_multilingual_meeting/status.json

Step 6: Validate files

find evaluation_dataset -type f | sort

---

31. Summary of Evaluation Tasks

Task	Input	Expected output	Main reference file
Speech LID	Single audio utterance	Language or dialect label	speech_language_identification/metadata.jsonl
Diarization	Multi-speaker recording	Speaker timestamps and clusters	RTTM and segment JSON
Speaker embedding	Enrollment and test audio	Similarity score or embedding	trials.jsonl
Speaker gender	Single audio utterance	Gender metadata class	speaker_gender_classification/metadata.jsonl
Transcribe Mode A	Single audio utterance	Transcript	speech_transcription/mode_a_single_utterance/metadata.jsonl
Transcribe Mode B	Multilingual meeting audio	Segments, speaker, language, gender, and transcript	RTTM, segment JSON, and transcript

---

32. Final Notes

This evaluation dataset is intended to provide a controlled and reproducible framework for testing multilingual speech systems.

It is especially useful for:

• Initial model benchmarking
• Regression testing
• Comparing model versions
• Measuring language-specific performance
• Testing integrated speech pipelines
• Identifying low-resource-language failures
• Evaluating speaker and language segmentation
• Creating automated quality gates

For final production validation, this dataset should be supplemented with:

• Natural meetings
• Real telephone calls
• Noisy recordings
• Far-field microphones
• Overlapping speech
• Code-switching
• Different acoustic environments
• Unseen speakers
• Unseen domains
• Human-reviewed transcripts and annotations