Evaluate the benchmark against other hosted LLM API's, not just OpenAI GPT-4o (eg. Claude, Gemini, etc)

[0xdevalias]

(Note: This was originally created as an issue on the benchmark website repo, but it feels like it is probably better placed here: jsdeobf/jsdeobf.github.io#10)

---

Looking at the current leaderboard / paper, it seems like the only cloud hosted API that was evaluated was OpenAI's GPT-4o. It would be interesting to see how some of the other providers (eg. Claude, Gemini, etc) frontier models rank in this benchmark.

See Also

• Evaluate the benchmark against other hosted LLM API's, not just OpenAI GPT-4o (eg. Claude, Gemini, etc) jsdeobf/jsdeobf.github.io#10
• Evaluate the benchmark against other open-source LLM's (eg. gpt-oss, etc) #9
• Evaluate the benchmark against other open-source LLM's (eg. gpt-oss, etc) jsdeobf/jsdeobf.github.io#11
• Include LLM evaluation costs in the leaderboard jsdeobf/jsdeobf.github.io#9

[0xdevalias]

For example it would be interesting to see how the newer OpenAI GPT-5 models compare / rank:

• https://openai.com/index/introducing-gpt-5/

  • Introducing GPT-5

  • https://platform.openai.com/docs/models/gpt-5
  • https://platform.openai.com/docs/models/gpt-5-mini
  • https://platform.openai.com/docs/models/gpt-5-nano

Even the older OpenAI GPT-4.1 model would be interesting still:

• https://openai.com/index/gpt-4-1/

  • Introducing GPT-4.1 in the API

  • https://platform.openai.com/docs/models/gpt-4.1

Note that for OpenAI's open-weight models like gpt-oss, I have included them in a separate issue, though realistically they could also be evaluated using the hosted API versions:

• Evaluate the benchmark against other open-source LLM's (eg. gpt-oss, etc) #9 (comment)

As OpenAI doesn't currently provide a proper hosted API version of them:

• https://openai.com/index/introducing-gpt-oss/

  • We’re continuing to listen closely to developer feedback and may consider API support for gpt-oss in the future.

    If you want to try the models, head over to our open model playground⁠(opens in a new window). To learn more about how to use the models using different ecosystem providers or how to fine-tune the models, check out our guides⁠(opens in a new window).

---

For models from Anthropic / Claude:

• https://docs.anthropic.com/en/docs/about-claude/models/overview
• https://www.anthropic.com/news/1m-context

  • Claude Sonnet 4 now supports 1M tokens of context (13 Aug 2025)

  • Claude Sonnet 4 now supports up to 1 million tokens of context on the Anthropic API—a 5x increase that lets you process entire codebases with over 75,000 lines of code or dozens of research papers in a single request.

  • Long context support for Sonnet 4 is now in public beta on the Anthropic API, Amazon Bedrock, and Google Cloud’s Vertex AI.

• https://www.anthropic.com/news/claude-opus-4-1

  • Claude Opus 4.1 (6 Aug 2025)

  • Today we're releasing Claude Opus 4.1, an upgrade to Claude Opus 4 on agentic tasks, real-world coding, and reasoning. We plan to release substantially larger improvements to our models in the coming weeks.

• https://www.anthropic.com/news/claude-4

  • Introducing Claude 4 (23 May 2025)

  • Today, we’re introducing the next generation of Claude models: Claude Opus 4 and Claude Sonnet 4, setting new standards for coding, advanced reasoning, and AI agents.

  • Claude Opus 4 is the world’s best coding model, with sustained performance on complex, long-running tasks and agent workflows. Claude Sonnet 4 is a significant upgrade to Claude Sonnet 3.7, delivering superior coding and reasoning while responding more precisely to your instructions.

• https://www.anthropic.com/news/claude-3-7-sonnet

  • Claude 3.7 Sonnet and Claude Code (25 Feb 2025)

  • Today, we’re announcing Claude 3.7 Sonnet, our most intelligent model to date and the first hybrid reasoning model on the market. Claude 3.7 Sonnet can produce near-instant responses or extended, step-by-step thinking that is made visible to the user. API users also have fine-grained control over how long the model can think for.

  • Claude 3.7 Sonnet shows particularly strong improvements in coding and front-end web development. Along with the model, we’re also introducing a command line tool for agentic coding, Claude Code. Claude Code is available as a limited research preview, and enables developers to delegate substantial engineering tasks to Claude directly from their terminal.

• https://www.anthropic.com/news/claude-3-5-sonnet

  • Claude 3.5 Sonnet (21 Jun 2024)

  • Today, we’re launching Claude 3.5 Sonnet—our first release in the forthcoming Claude 3.5 model family. Claude 3.5 Sonnet raises the industry bar for intelligence, outperforming competitor models and Claude 3 Opus on a wide range of evaluations, with the speed and cost of our mid-tier model, Claude 3 Sonnet.

• https://www.anthropic.com/news/claude-3-family

  • Introducing the next generation of Claude (4 Mar 2024)

  • Today, we're announcing the Claude 3 model family, which sets new industry benchmarks across a wide range of cognitive tasks. The family includes three state-of-the-art models in ascending order of capability: Claude 3 Haiku, Claude 3 Sonnet, and Claude 3 Opus. Each successive model offers increasingly powerful performance, allowing users to select the optimal balance of intelligence, speed, and cost for their specific application.