Hi LongBench team — I maintain EvalPort, an open, framework-agnostic JSON spec for portable LLM eval datasets (Apache 2.0, TestCase/Suite schema with a real JSON Schema validator). It has a benchmarks/ directory converting well-known public benchmarks (GSM8K, ARC, HumanEval, SQuAD, etc.) into validated EvalPort suites, each with attribution and license preserved, so someone using a different eval harness can still run the same dataset without a bespoke converter.
I read your README's own "Data Format" section rather than guessing at fields:
{
"_id": "Unique identifier for each piece of data",
"domain": "...", "sub_domain": "...",
"difficulty": "easy | hard",
"length": "short | medium | long",
"question": "...",
"choice_A": "...", "choice_B": "...", "choice_C": "...", "choice_D": "...",
"answer": "A | B | C | D",
"context": "..."
}This maps cleanly onto an EvalPort TestCase: question + context → input, choice_A..D → the multiple-choice options, answer → expected_output, and domain/sub_domain/difficulty/length → metadata (which is exactly what your own result.py breaks accuracy down by — easy/hard, short/medium/long).
Would your maintainers be open to an EvalPort-format export of LongBench v2 living in EvalPort's benchmarks/ directory, with attribution to this repo/paper and the original license preserved? I'd do the conversion work and validate it against EvalPort's real schema — this would just be a heads-up issue first, not a PR out of nowhere. Also happy to link back to LongBench from EvalPort's benchmark listing either way. Let me know if this is of interest, or not a priority right now — no worries either way.
Hi LongBench team — I maintain EvalPort, an open, framework-agnostic JSON spec for portable LLM eval datasets (Apache 2.0,
TestCase/Suiteschema with a real JSON Schema validator). It has abenchmarks/directory converting well-known public benchmarks (GSM8K, ARC, HumanEval, SQuAD, etc.) into validated EvalPort suites, each with attribution and license preserved, so someone using a different eval harness can still run the same dataset without a bespoke converter.I read your README's own "Data Format" section rather than guessing at fields:
{ "_id": "Unique identifier for each piece of data", "domain": "...", "sub_domain": "...", "difficulty": "easy | hard", "length": "short | medium | long", "question": "...", "choice_A": "...", "choice_B": "...", "choice_C": "...", "choice_D": "...", "answer": "A | B | C | D", "context": "..." }This maps cleanly onto an EvalPort
TestCase:question+context→input,choice_A..D→ the multiple-choice options,answer→expected_output, anddomain/sub_domain/difficulty/length→metadata(which is exactly what your ownresult.pybreaks accuracy down by — easy/hard, short/medium/long).Would your maintainers be open to an EvalPort-format export of LongBench v2 living in EvalPort's
benchmarks/directory, with attribution to this repo/paper and the original license preserved? I'd do the conversion work and validate it against EvalPort's real schema — this would just be a heads-up issue first, not a PR out of nowhere. Also happy to link back to LongBench from EvalPort's benchmark listing either way. Let me know if this is of interest, or not a priority right now — no worries either way.