Issues

• integrate violation probability scoring code

  blocked
  mlcommons/modelbench#1699 · Filter by author bkorycki opened on Oct 15, 2026

• Increase output token limit 3k->10k for security 1.1 benchmark (and later)

  mlcommons/modelbench#1696 · Filter by author bollacker opened on Oct 9, 2026

• Implement and test SLA measurements

  mlcommons/modelbench#1695 · Filter by author bollacker opened on Oct 9, 2026

• Practice tests return annotations

  mlcommons/modelbench#1692 · Filter by author bollacker opened on Oct 8, 2026

• Run e2e test less often

  chore
  mlcommons/modelbench#1686 · Filter by author rogthefrog opened on Sep 29, 2026
  

• Fix dynamic openai suts

  mlcommons/modelbench#1682 · Filter by author bkorycki opened on Sep 25, 2026

• run modelbench jobs off the modelrunner machine (cloud run)

  chore
  mlcommons/modelbench#1680 · Filter by author rogthefrog opened on Sep 25, 2026
  

• figure out huggingface billing and endpoint health

  mlcommons/modelbench#1679 · Filter by author rogthefrog opened on Sep 25, 2026
  

• Improve benchmark object design

  chore
  mlcommons/modelbench#1676 · Filter by author bkorycki opened on Sep 23, 2026

• Prism collects Candela run data

  mlcommons/modelbench#1674 · Filter by author bollacker opened on Sep 23, 2026
  2

• Support benchmark SKUs/UIDs for initialization in Candela

  mlcommons/modelbench#1673 · Filter by author bollacker opened on Sep 23, 2026
  2

  

• Stand up security 1.1 benchmark.

  mlcommons/modelbench#1672 · Filter by author bollacker opened on Sep 23, 2026

• Optional EvalPort interop for HazardScore/BenchmarkScore results

  mlcommons/modelbench#1669 · Filter by author adhabnr-ux opened on Sep 22, 2026

• Why can't we run any NVIDIA NIM suts

  mlcommons/modelbench#1660 · Filter by author bkorycki opened on Sep 14, 2026

• Fix mistral SUT model id formatting

  mlcommons/modelbench#1659 · Filter by author bkorycki opened on Sep 14, 2026
  1

• Misc google SUT failures

  mlcommons/modelbench#1658 · Filter by author bkorycki opened on Sep 14, 2026
  1

• Some Google SUTs don't return real content

  mlcommons/modelbench#1657 · Filter by author bkorycki opened on Sep 14, 2026
  2

• Update William's tool to only check models we care about

  mlcommons/modelbench#1656 · Filter by author bkorycki opened on Sep 14, 2026

• Decide if we should support Google's Interactions API

  mlcommons/modelbench#1655 · Filter by author bkorycki opened on Sep 14, 2026

• Fix AWS SUTs

  mlcommons/modelbench#1654 · Filter by author bkorycki opened on Sep 14, 2026

• Anthropic SUT sometimes doesn't return a TextBlock

  mlcommons/modelbench#1653 · Filter by author bkorycki opened on Sep 14, 2026
  1

• Create openrouter adapter

  mlcommons/modelbench#1648 · Filter by author bollacker opened on Sep 11, 2026
  1

• Add default model lists to dynamic SUT providers that don't auto-discover

  mlcommons/modelbench#1646 · Filter by author wpietri opened on Sep 9, 2026

• Investigate dynamic SUT run failures

  mlcommons/modelbench#1645 · Filter by author wpietri opened on Sep 9, 2026

• Investigate dynamic SUT listing failures

  mlcommons/modelbench#1644 · Filter by author wpietri opened on Sep 9, 2026
  1