MTBench_finance_aligned_pairs_long/short has no json files for evaluations

[lirannoc]

The folders MTBench_finance_aligned_pairs_short and MTBench_finance_aligned_pairs_long don't seem to contain json files, which makes it incompatible with evaluation/finance/trend_classification etc.
It is solvable with using different formatting, could you make those fixes?

[itouchz]

I think you may use this script.

"""
Convert Finance Aligned Pairs from Parquet to JSON Format

The official MTBench download provides finance aligned_pairs in Parquet format,
but our implementation expects JSON files. This script converts them.
"""

import pandas as pd
import json
from pathlib import Path

def convert_parquet_to_json(parquet_file: str, output_dir: str):
    """Convert a parquet file to individual JSON files."""
    print(f"Reading {parquet_file}...")
    df = pd.read_parquet(parquet_file)
    
    print(f"Found {len(df)} samples")
    print(f"Columns: {list(df.columns)}")
    
    output_path = Path(output_dir)
    output_path.mkdir(parents=True, exist_ok=True)
    
    # Convert each row to a JSON file
    for idx, row in df.iterrows():
        # Create filename from index or use a unique identifier if available
        # Assuming there's an 'index' or 'id' column, otherwise use row number
        if 'index' in df.columns:
            filename = f"{row['index']}.json"
        elif 'id' in df.columns:
            filename = f"{row['id']}.json"
        else:
            filename = f"{idx}.json"
        
        # Convert row to dict
        data = row.to_dict()
        
        # Convert any numpy types to Python types
        data = json.loads(json.dumps(data, default=str))
        
        # Write JSON file
        json_path = output_path / filename
        with open(json_path, 'w') as f:
            json.dump(data, f, indent=2)
        
        if (idx + 1) % 100 == 0:
            print(f"  Processed {idx + 1}/{len(df)} samples...")
    
    print(f"✅ Converted {len(df)} samples to {output_dir}")
    return len(df)

def main():
    """Convert all finance aligned pairs datasets."""
    base_dir = Path("datasets/MTBench/data/processed/finance")
    
    datasets = [
        {
            "parquet": base_dir / "aligned_in7days_out1days" / "data" / "train-00000-of-00001.parquet",
            "output": base_dir / "aligned_in7days_out1days",
            "name": "Finance Short (7 days in, 1 day out)",
            "expected": 750  # From HuggingFace
        },
        {
            "parquet": base_dir / "aligned_in30days_out7days" / "data" / "train-00000-of-00001.parquet",
            "output": base_dir / "aligned_in30days_out7days",
            "name": "Finance Long (30 days in, 7 days out)",
            "expected": 525  # From HuggingFace
        }
    ]
    
    print("=" * 80)
    print("Converting Finance Aligned Pairs: Parquet → JSON")
    print("=" * 80)
    
    total_converted = 0
    
    for ds in datasets:
        print(f"\n[Converting] {ds['name']}")
        print(f"  From: {ds['parquet']}")
        print(f"  To: {ds['output']}")
        print(f"  Expected: {ds['expected']} samples")
        
        if not ds['parquet'].exists():
            print(f"  ❌ Parquet file not found!")
            continue
        
        count = convert_parquet_to_json(str(ds['parquet']), str(ds['output']))
        total_converted += count
        
        if count == ds['expected']:
            print(f"  ✅ Count matches expected: {count}")
        else:
            print(f"  ⚠️ Count mismatch: got {count}, expected {ds['expected']}")
    
    print("\n" + "=" * 80)
    print(f"Conversion Complete!")
    print(f"Total converted: {total_converted} samples")
    print("=" * 80)
    
    print("\nNext steps:")
    print("1. Verify JSON files:")
    print("   ls -la datasets/MTBench/data/processed/finance/aligned_in7days_out1days/*.json | wc -l")
    print("2. Test with our implementation:")
    print("   bash run_ts_debate_finance_trend.sh")

if __name__ == "__main__":
    main()

[lirannoc]

thanks for the response!

In the end, I changed the code to accept the datasets directly from Hugging Face, it's just more convenient.
I extract the relevant fields and thus construct extracted_data = {...}.
Do you think my process should be consistent with the current construction with JSONs?