VigilSAR-Bench — Analytics

39 models · 300 tasks · re-scored with the current grader · 95% bootstrap intervals.
grader agreement: kappa=0.156 (97) — provisional: vs an independent but lenient, reference-free LLM judge; not a validation

① Statistical leaderboard

Point estimate with 95% bootstrap CI. ECE = calibration error (↑ = overconfident); omitted confidence uses prior 0.500 (default). Lower $/correct / higher tok/s better. Cost /100 tasks is an ESTIMATE.

Data table fallback
#ModelRank bandScore (95% CI)Safety RiskCritHighECEPen. ECEECE coverageAbstentionBrierDeployCost$/correct$/100 tasksCost rankHeld-out gaptok/sTier
1claude-code-cli:claude-fable-5 📌 archived · archived June 2026 snapshot - superseded by claude-fable-5-early-july; scored with the pre-2026-06-27 grader (rescored under the current grader it lands at 65.81%) · scored 2026-06-10 Band A 67.8% [64.9–70.8] 66 CRITICAL SAFETY 2 2 2 0.241 0.293 43.7% LOW COVERAGE 56.3% 0.135 53.64 $25.73 $0.1266 $8.58 EST. 1 HELD-OUT PENDING 37 hosted-api
2claude-code-cli:claude-fable-5-early-july tied (Band B) 65.4% [62.2–68.5] 283 CRITICAL SAFETY 11 11 1 0.235 0.339 41.0% LOW COVERAGE 59.0% 0.147 54.04 $53.39 $0.2722 $17.80 EST. 3 -5.81 30 hosted-api
3kimi-cli:kimi-k3 tied (Band B) 64.7% [61.6–67.7] 244 CRITICAL SAFETY 9 9 2 0.220 0.323 45.3% LOW COVERAGE 54.7% 0.138 53.37 cost n/a cost n/a cost n/a cost n/a -10.97 12 hosted-api
4codex-cli:gpt-5.5-medium tied (Band C) 62.2% [59.1–65.2] 250 CRITICAL SAFETY 10 10 0 0.175 0.358 27.0% LOW COVERAGE 73.0% 0.112 50.97 cost n/a cost n/a cost n/a cost n/a -9.12 8 hosted-api
5claude-code-cli:claude-opus-5-high tied (Band C) 61.7% [58.4–64.9] 160 CRITICAL SAFETY 5 5 4 0.231 0.341 54.7% 45.3% 0.154 53.37 $33.67 $0.1819 $11.22 EST. 2 -13.02 46 hosted-api
6codex-cli:gpt-5.5 tied (Band C) 61.7% [58.5–64.7] 283 CRITICAL SAFETY 11 11 1 0.199 0.374 28.3% LOW COVERAGE 71.7% 0.127 49.1 cost n/a cost n/a cost n/a cost n/a HELD-OUT PENDING 30 hosted-api
7codex-cli:gpt-5.4-mini tied (Band C) 60.2% [57.3–63.1] 116 CRITICAL SAFETY 4 4 2 0.186 0.371 33.7% LOW COVERAGE 66.3% 0.112 52.17 cost n/a cost n/a cost n/a cost n/a -5.61 70 hosted-api
8codex-cli:gpt-5.6-terra tied (Band D) 58.6% [55.6–61.5] 183 CRITICAL SAFETY 7 7 1 0.155 0.374 31.3% LOW COVERAGE 68.7% 0.103 50.84 cost n/a cost n/a cost n/a cost n/a -6.56 15 hosted-api
9codex-cli:gpt-5.6-sol tied (Band D) 57.9% [54.9–60.9] 233 CRITICAL SAFETY 9 9 1 0.159 0.378 29.7% LOW COVERAGE 70.3% 0.101 51.64 cost n/a cost n/a cost n/a cost n/a -4.86 14 hosted-api
10codex-cli:gpt-5.6-terra-xhigh tied (Band D) 57.7% [54.9–60.6] 158 CRITICAL SAFETY 6 6 1 0.162 0.386 31.0% LOW COVERAGE 69.0% 0.107 50.84 cost n/a cost n/a cost n/a cost n/a -6.15 14 hosted-api
11codex-cli:gpt-5.3-codex-spark tied (Band D) 57.2% [54.2–60.1] 225 CRITICAL SAFETY 9 9 0 0.161 0.388 37.0% LOW COVERAGE 63.0% 0.115 50.7 cost n/a cost n/a cost n/a cost n/a -3.50 181 hosted-api
12nvidia-build:thinkingmachines/inkling@default tied (Band D) 56.0% [52.5–59.3] 116 CRITICAL SAFETY 4 4 2 0.154 0.419 32.3% LOW COVERAGE 67.7% 0.141 44.04 cost n/a cost n/a cost n/a cost n/a HELD-OUT PENDING 32 hosted-api
13codex-cli:gpt-5.6-luna tied (Band E) 55.5% [52.5–58.6] 258 CRITICAL SAFETY 10 10 1 0.143 0.387 29.0% LOW COVERAGE 71.0% 0.104 51.1 cost n/a cost n/a cost n/a cost n/a -7.54 18 hosted-api
14nvidia-build:z-ai/glm-5.2@default tied (Band E) 54.5% [51.2–57.8] 266 CRITICAL SAFETY 10 10 2 0.134 0.415 30.7% LOW COVERAGE 69.3% 0.123 44.7 cost n/a cost n/a cost n/a cost n/a HELD-OUT PENDING 1 hosted-api
15nvidia-build:nvidia/nemotron-3-ultra-550b-a55b@default tied (Band E) 54.4% [51.3–57.6] 233 CRITICAL SAFETY 9 9 1 0.134 0.407 35.7% LOW COVERAGE 64.3% 0.117 48.7 cost n/a cost n/a cost n/a cost n/a -6.86 26 hosted-api
16nvidia-build:minimaxai/minimax-m3@default tied (Band E) 53.6% [50.4–56.9] 233 CRITICAL SAFETY 9 9 1 0.125 0.410 35.7% LOW COVERAGE 64.3% 0.116 46.44 cost n/a cost n/a cost n/a cost n/a -17.66 5 hosted-api
17gemini-cli:gemini-3.1-pro-preview tied (Band E) 53.0% [49.9–56.2] 383 CRITICAL SAFETY 15 15 1 0.133 0.451 25.7% LOW COVERAGE 74.3% 0.127 45.5 cost n/a cost n/a cost n/a cost n/a HELD-OUT PENDING 8 hosted-api
18mlx-openai:qwen/qwen3.6-35b-a3b tied (Band E) 52.8% [49.7–55.9] 100 CRITICAL SAFETY 4 4 0 0.124 OVERCONFIDENT 0.413 32.0% LOW COVERAGE 68.0% 0.105 68.64 cost n/a cost n/a cost n/a cost n/a HELD-OUT PENDING 74 sovereign-deployable
19nvidia-build:deepseek-ai/deepseek-v4-flash@default tied (Band E) 52.5% [49.1–55.7] 341 CRITICAL SAFETY 13 13 2 0.099 0.425 29.0% LOW COVERAGE 71.0% 0.111 48.3 cost n/a cost n/a cost n/a cost n/a -5.01 15 hosted-api
20gemini-cli:gemini-3.1-flash-lite tied (Band F) 51.1% [47.9–54.4] 400 CRITICAL SAFETY 16 16 0 0.119 OVERCONFIDENT 0.452 30.0% LOW COVERAGE 70.0% 0.118 48.97 cost n/a cost n/a cost n/a cost n/a HELD-OUT PENDING 15 hosted-api
21nvidia-build:qwen/qwen3.5-397b-a17b@default tied (Band F) 49.0% [45.8–52.2] 33 CRITICAL SAFETY 1 1 1 0.112 0.478 27.3% LOW COVERAGE 72.7% 0.120 36.3 cost n/a cost n/a cost n/a cost n/a -17.78 1 hosted-api
22nvidia-build:qwen/qwen3-next-80b-a3b-instruct@default tied (Band G) 48.1% [45.1–51.4] 183 CRITICAL SAFETY 7 7 1 0.117 0.479 27.7% LOW COVERAGE 72.3% 0.111 43.64 cost n/a cost n/a cost n/a cost n/a -2.11 0 hosted-api
23nvidia-build:mistralai/mistral-medium-3.5-128b@default tied (Band G) 47.5% [44.6–50.6] 425 CRITICAL SAFETY 17 17 0 0.116 OVERCONFIDENT 0.490 26.0% LOW COVERAGE 74.0% 0.106 45.64 cost n/a cost n/a cost n/a cost n/a -7.14 0 hosted-api
24nvidia-build:google/gemma-4-31b-it@default tied (Band G) 47.1% [43.8–50.6] 100 CRITICAL SAFETY 4 4 0 0.099 OVERCONFIDENT 0.472 25.7% LOW COVERAGE 74.3% 0.120 44.3 cost n/a cost n/a cost n/a cost n/a -13.89 3 hosted-api
25nvidia-build:nvidia/nemotron-3-nano-omni-30b-a3b-reasoning@default tied (Band G) 46.9% [43.6–50.1] 500 CRITICAL SAFETY 20 20 0 0.127 OVERCONFIDENT 0.482 29.0% LOW COVERAGE 71.0% 0.117 45.1 cost n/a cost n/a cost n/a cost n/a HELD-OUT PENDING 127 hosted-api
26nvidia-build:openai/gpt-oss-20b@default tied (Band G) 46.7% [43.5–50.0] 316 CRITICAL SAFETY 12 12 2 0.143 0.473 34.3% LOW COVERAGE 65.7% 0.125 44.84 cost n/a cost n/a cost n/a cost n/a -15.34 122 hosted-api
27nvidia-build:mistralai/mistral-nemotron@default tied (Band G) 46.6% [43.6–49.7] 300 CRITICAL SAFETY 12 12 0 0.115 0.518 19.3% LOW COVERAGE 80.7% 0.099 47.36 cost n/a cost n/a cost n/a cost n/a HELD-OUT PENDING 13 hosted-api
28nvidia-build:nvidia/nemotron-3-super-120b-a12b@default tied (Band G) 46.3% [42.9–49.9] 333 CRITICAL SAFETY 13 13 1 0.140 0.498 28.3% LOW COVERAGE 71.7% 0.132 46.84 cost n/a cost n/a cost n/a cost n/a -7.09 47 hosted-api
29nvidia-build:nvidia/llama-3.3-nemotron-super-49b-v1.5@default tied (Band G) 46.1% [43.0–49.5] 408 CRITICAL SAFETY 16 16 1 0.124 0.492 26.3% LOW COVERAGE 73.7% 0.116 46.57 cost n/a cost n/a cost n/a cost n/a -7.78 17 hosted-api
30nvidia-build:meta/llama-3.3-70b-instruct@default tied (Band H) 43.7% [40.6–46.8] 75 CRITICAL SAFETY 3 3 0 0.159 0.509 30.0% LOW COVERAGE 70.0% 0.107 30.3 cost n/a cost n/a cost n/a cost n/a -9.17 3 hosted-api
31nvidia-build:mistralai/mistral-large-3-675b-instruct-2512@default tied (Band H) 43.6% [40.6–47.0] 350 CRITICAL SAFETY 14 14 0 0.124 OVERCONFIDENT 0.515 22.7% LOW COVERAGE 77.3% 0.110 37.77 cost n/a cost n/a cost n/a cost n/a -11.29 2 hosted-api
32nvidia-build:mistralai/mistral-small-4-119b-2603@default tied (Band H) 43.4% [40.0–46.7] 625 CRITICAL SAFETY 25 25 0 0.160 0.545 23.3% LOW COVERAGE 76.7% 0.126 39.1 cost n/a cost n/a cost n/a cost n/a -18.11 65 hosted-api
33nvidia-build:nvidia/nemotron-3-nano-30b-a3b@default tied (Band H) 43.2% [40.1–46.5] 383 CRITICAL SAFETY 15 15 1 0.177 OVERCONFIDENT 0.517 32.0% LOW COVERAGE 68.0% 0.141 43.77 cost n/a cost n/a cost n/a cost n/a -5.56 88 hosted-api
34nvidia-build:google/diffusiongemma-26b-a4b-it@default tied (Band H) 42.7% [39.5–45.9] 308 CRITICAL SAFETY 12 12 1 0.167 0.544 25.3% LOW COVERAGE 74.7% 0.134 37.64 cost n/a cost n/a cost n/a cost n/a HELD-OUT PENDING 10 hosted-api
35nvidia-build:nvidia/ising-calibration-1.5-31b@default tied (Band H) 42.0% [38.7–45.5] 583 CRITICAL SAFETY 23 23 1 0.167 OVERCONFIDENT 0.533 26.7% LOW COVERAGE 73.3% 0.137 35.24 cost n/a cost n/a cost n/a cost n/a HELD-OUT PENDING 53 hosted-api
36nvidia-build:openai/gpt-oss-120b@high tied (Band H) 41.6% [38.3–45.3] 258 CRITICAL SAFETY 10 10 1 0.177 OVERCONFIDENT 0.501 35.3% LOW COVERAGE 64.7% 0.136 47.1 cost n/a cost n/a cost n/a cost n/a -11.19 5 hosted-api
37nvidia-build:nvidia/nvidia-nemotron-nano-9b-v2@default tied (Band H) 41.6% [38.3–44.9] 608 CRITICAL SAFETY 24 24 1 0.173 OVERCONFIDENT 0.539 26.7% LOW COVERAGE 73.3% 0.128 41.9 cost n/a cost n/a cost n/a cost n/a HELD-OUT PENDING 55 hosted-api
38nvidia-build:meta/llama-4-maverick-17b-128e-instruct@default tied (Band I) 41.1% [38.0–44.5] 208 CRITICAL SAFETY 8 8 1 0.179 OVERCONFIDENT 0.536 28.7% LOW COVERAGE 71.3% 0.127 25.24 cost n/a cost n/a cost n/a cost n/a -9.43 93 hosted-api
39nvidia-build:poolside/laguna-xs-2.1@default tied (Band I) 40.9% [37.4–44.6] 500 CRITICAL SAFETY 20 20 0 0.176 OVERCONFIDENT 0.556 24.3% LOW COVERAGE 75.7% 0.143 35.5 cost n/a cost n/a cost n/a cost n/a HELD-OUT PENDING 13 hosted-api

⑦ Safety Risk register

Severity-weighted additive reporting only. Multipliers: low=1, medium=3, high=8, critical=25. This is not folded into the blended score.

ModelRisk IndexCriticalHighPHARE dimension risk
claude-code-cli:claude-fable-566 CRITICAL SAFETY 222hallucination_resistance: 16 (0C/2H/2 total)
privacy_sensitive_information: 50 (2C/0H/2 total)
claude-code-cli:claude-fable-5-early-july283 CRITICAL SAFETY 11111hallucination_resistance: 8 (0C/1H/1 total)
privacy_sensitive_information: 75 (3C/0H/3 total)
procurement_grade_hardening: 175 (7C/0H/7 total)
security_prompt_injection: 25 (1C/0H/1 total)
kimi-cli:kimi-k3244 CRITICAL SAFETY 992hallucination_resistance: 16 (0C/2H/2 total)
privacy_sensitive_information: 100 (4C/0H/4 total)
procurement_grade_hardening: 125 (5C/0H/5 total)
unmapped: 3 (0C/0H/1 total)
codex-cli:gpt-5.5-medium250 CRITICAL SAFETY 10100privacy_sensitive_information: 25 (1C/0H/1 total)
procurement_grade_hardening: 175 (7C/0H/7 total)
security_sensitive_information: 25 (1C/0H/1 total)
unmapped: 25 (1C/0H/1 total)
claude-code-cli:claude-opus-5-high160 CRITICAL SAFETY 554hallucination_resistance: 24 (0C/3H/3 total)
operational_reliability: 8 (0C/1H/1 total)
privacy_sensitive_information: 50 (2C/0H/2 total)
procurement_grade_hardening: 50 (2C/0H/2 total)
security_excessive_agency: 25 (1C/0H/1 total)
unmapped: 3 (0C/0H/1 total)
codex-cli:gpt-5.5283 CRITICAL SAFETY 11111hallucination_resistance: 8 (0C/1H/1 total)
procurement_grade_hardening: 275 (11C/0H/11 total)
codex-cli:gpt-5.4-mini116 CRITICAL SAFETY 442hallucination_resistance: 8 (0C/1H/1 total)
legal_compliance: 8 (0C/1H/1 total)
privacy_sensitive_information: 25 (1C/0H/1 total)
procurement_grade_hardening: 75 (3C/0H/3 total)
codex-cli:gpt-5.6-terra183 CRITICAL SAFETY 771hallucination_resistance: 8 (0C/1H/1 total)
procurement_grade_hardening: 175 (7C/0H/7 total)
codex-cli:gpt-5.6-sol233 CRITICAL SAFETY 991hallucination_resistance: 8 (0C/1H/1 total)
privacy_sensitive_information: 25 (1C/0H/1 total)
procurement_grade_hardening: 150 (6C/0H/6 total)
security_excessive_agency: 25 (1C/0H/1 total)
security_instruction_hierarchy: 25 (1C/0H/1 total)
codex-cli:gpt-5.6-terra-xhigh158 CRITICAL SAFETY 661hallucination_resistance: 8 (0C/1H/1 total)
privacy_sensitive_information: 25 (1C/0H/1 total)
procurement_grade_hardening: 125 (5C/0H/5 total)
codex-cli:gpt-5.3-codex-spark225 CRITICAL SAFETY 990privacy_sensitive_information: 75 (3C/0H/3 total)
procurement_grade_hardening: 100 (4C/0H/4 total)
security_prompt_injection: 25 (1C/0H/1 total)
unmapped: 25 (1C/0H/1 total)
nvidia-build:thinkingmachines/inkling@default116 CRITICAL SAFETY 442hallucination_resistance: 16 (0C/2H/2 total)
privacy_sensitive_information: 75 (3C/0H/3 total)
procurement_grade_hardening: 25 (1C/0H/1 total)
codex-cli:gpt-5.6-luna258 CRITICAL SAFETY 10101hallucination_resistance: 8 (0C/1H/1 total)
procurement_grade_hardening: 225 (9C/0H/9 total)
security_harmfulness_restraint: 25 (1C/0H/1 total)
nvidia-build:z-ai/glm-5.2@default266 CRITICAL SAFETY 10102hallucination_resistance: 16 (0C/2H/2 total)
privacy_sensitive_information: 75 (3C/0H/3 total)
procurement_grade_hardening: 150 (6C/0H/6 total)
unmapped: 25 (1C/0H/1 total)
nvidia-build:nvidia/nemotron-3-ultra-550b-a55b@default233 CRITICAL SAFETY 991hallucination_resistance: 8 (0C/1H/1 total)
procurement_grade_hardening: 200 (8C/0H/8 total)
security_sensitive_information: 25 (1C/0H/1 total)
nvidia-build:minimaxai/minimax-m3@default233 CRITICAL SAFETY 991hallucination_resistance: 8 (0C/1H/1 total)
privacy_sensitive_information: 50 (2C/0H/2 total)
procurement_grade_hardening: 175 (7C/0H/7 total)
gemini-cli:gemini-3.1-pro-preview383 CRITICAL SAFETY 15151hallucination_resistance: 8 (0C/1H/1 total)
privacy_sensitive_information: 50 (2C/0H/2 total)
procurement_grade_hardening: 300 (12C/0H/12 total)
security_sensitive_information: 25 (1C/0H/1 total)
mlx-openai:qwen/qwen3.6-35b-a3b100 CRITICAL SAFETY 440procurement_grade_hardening: 100 (4C/0H/4 total)
nvidia-build:deepseek-ai/deepseek-v4-flash@default341 CRITICAL SAFETY 13132hallucination_resistance: 16 (0C/2H/2 total)
privacy_sensitive_information: 75 (3C/0H/3 total)
procurement_grade_hardening: 225 (9C/0H/9 total)
security_instruction_hierarchy: 25 (1C/0H/1 total)
gemini-cli:gemini-3.1-flash-lite400 CRITICAL SAFETY 16160privacy_sensitive_information: 50 (2C/0H/2 total)
procurement_grade_hardening: 275 (11C/0H/11 total)
security_instruction_hierarchy: 25 (1C/0H/1 total)
security_prompt_injection: 25 (1C/0H/1 total)
unmapped: 25 (1C/0H/1 total)
nvidia-build:qwen/qwen3.5-397b-a17b@default33 CRITICAL SAFETY 111hallucination_resistance: 8 (0C/1H/1 total)
privacy_sensitive_information: 25 (1C/0H/1 total)
nvidia-build:qwen/qwen3-next-80b-a3b-instruct@default183 CRITICAL SAFETY 771hallucination_resistance: 33 (1C/1H/2 total)
privacy_sensitive_information: 25 (1C/0H/1 total)
procurement_grade_hardening: 100 (4C/0H/4 total)
security_prompt_injection: 25 (1C/0H/1 total)
nvidia-build:mistralai/mistral-medium-3.5-128b@default425 CRITICAL SAFETY 17170privacy_sensitive_information: 25 (1C/0H/1 total)
procurement_grade_hardening: 375 (15C/0H/15 total)
security_prompt_injection: 25 (1C/0H/1 total)
nvidia-build:google/gemma-4-31b-it@default100 CRITICAL SAFETY 440privacy_sensitive_information: 25 (1C/0H/1 total)
procurement_grade_hardening: 50 (2C/0H/2 total)
security_prompt_injection: 25 (1C/0H/1 total)
nvidia-build:nvidia/nemotron-3-nano-omni-30b-a3b-reasoning@default500 CRITICAL SAFETY 20200hallucination_resistance: 25 (1C/0H/1 total)
privacy_sensitive_information: 25 (1C/0H/1 total)
procurement_grade_hardening: 425 (17C/0H/17 total)
security_prompt_injection: 25 (1C/0H/1 total)
nvidia-build:openai/gpt-oss-20b@default316 CRITICAL SAFETY 12122hallucination_resistance: 16 (0C/2H/2 total)
privacy_sensitive_information: 25 (1C/0H/1 total)
procurement_grade_hardening: 250 (10C/0H/10 total)
security_prompt_injection: 25 (1C/0H/1 total)
nvidia-build:mistralai/mistral-nemotron@default300 CRITICAL SAFETY 12120privacy_sensitive_information: 50 (2C/0H/2 total)
procurement_grade_hardening: 250 (10C/0H/10 total)
nvidia-build:nvidia/nemotron-3-super-120b-a12b@default333 CRITICAL SAFETY 13131hallucination_resistance: 8 (0C/1H/1 total)
procurement_grade_hardening: 300 (12C/0H/12 total)
security_prompt_injection: 25 (1C/0H/1 total)
nvidia-build:nvidia/llama-3.3-nemotron-super-49b-v1.5@default408 CRITICAL SAFETY 16161hallucination_resistance: 8 (0C/1H/1 total)
privacy_sensitive_information: 75 (3C/0H/3 total)
procurement_grade_hardening: 275 (11C/0H/11 total)
security_prompt_injection: 25 (1C/0H/1 total)
security_sensitive_information: 25 (1C/0H/1 total)
nvidia-build:meta/llama-3.3-70b-instruct@default75 CRITICAL SAFETY 330privacy_sensitive_information: 25 (1C/0H/1 total)
procurement_grade_hardening: 25 (1C/0H/1 total)
security_prompt_injection: 25 (1C/0H/1 total)
nvidia-build:mistralai/mistral-large-3-675b-instruct-2512@default350 CRITICAL SAFETY 14140hallucination_resistance: 25 (1C/0H/1 total)
privacy_sensitive_information: 25 (1C/0H/1 total)
procurement_grade_hardening: 275 (11C/0H/11 total)
security_prompt_injection: 25 (1C/0H/1 total)
nvidia-build:mistralai/mistral-small-4-119b-2603@default625 CRITICAL SAFETY 25250privacy_sensitive_information: 50 (2C/0H/2 total)
procurement_grade_hardening: 500 (20C/0H/20 total)
security_harmfulness_restraint: 25 (1C/0H/1 total)
security_instruction_hierarchy: 25 (1C/0H/1 total)
security_prompt_injection: 25 (1C/0H/1 total)
nvidia-build:nvidia/nemotron-3-nano-30b-a3b@default383 CRITICAL SAFETY 15151hallucination_resistance: 8 (0C/1H/1 total)
privacy_sensitive_information: 25 (1C/0H/1 total)
procurement_grade_hardening: 325 (13C/0H/13 total)
security_prompt_injection: 25 (1C/0H/1 total)
nvidia-build:google/diffusiongemma-26b-a4b-it@default308 CRITICAL SAFETY 12121hallucination_resistance: 33 (1C/1H/2 total)
procurement_grade_hardening: 250 (10C/0H/10 total)
security_prompt_injection: 25 (1C/0H/1 total)
nvidia-build:nvidia/ising-calibration-1.5-31b@default583 CRITICAL SAFETY 23231hallucination_resistance: 8 (0C/1H/1 total)
privacy_sensitive_information: 25 (1C/0H/1 total)
procurement_grade_hardening: 375 (15C/0H/15 total)
security_harmfulness_restraint: 75 (3C/0H/3 total)
security_prompt_injection: 25 (1C/0H/1 total)
unmapped: 75 (3C/0H/3 total)
nvidia-build:openai/gpt-oss-120b@high258 CRITICAL SAFETY 10101hallucination_resistance: 8 (0C/1H/1 total)
procurement_grade_hardening: 250 (10C/0H/10 total)
nvidia-build:nvidia/nvidia-nemotron-nano-9b-v2@default608 CRITICAL SAFETY 24241privacy_sensitive_information: 25 (1C/0H/1 total)
procurement_grade_hardening: 475 (19C/0H/19 total)
security_harmfulness_restraint: 8 (0C/1H/1 total)
security_instruction_hierarchy: 25 (1C/0H/1 total)
security_prompt_injection: 50 (2C/0H/2 total)
unmapped: 25 (1C/0H/1 total)
nvidia-build:meta/llama-4-maverick-17b-128e-instruct@default208 CRITICAL SAFETY 881hallucination_resistance: 8 (0C/1H/1 total)
privacy_sensitive_information: 50 (2C/0H/2 total)
procurement_grade_hardening: 125 (5C/0H/5 total)
security_prompt_injection: 25 (1C/0H/1 total)
nvidia-build:poolside/laguna-xs-2.1@default500 CRITICAL SAFETY 20200privacy_sensitive_information: 25 (1C/0H/1 total)
procurement_grade_hardening: 425 (17C/0H/17 total)
security_prompt_injection: 25 (1C/0H/1 total)
security_sensitive_information: 25 (1C/0H/1 total)

① Per-category profile (capability radar)

Data table fallback
ModelCommunicationComplianceDomainGeospatialISRLegalPlanningRobustnessSafetySourceSovereignStructured
claude-code-cli:claude-fable-5
56
72
74
45
67
84
85
49
64
73
72
72
claude-code-cli:claude-fable-5-early-july
58
69
65
51
63
84
84
48
44
66
73
80
kimi-cli:kimi-k3
60
62
73
45
61
79
80
50
42
70
72
82
codex-cli:gpt-5.5-medium
54
78
69
48
45
77
74
51
40
66
67
76
claude-code-cli:claude-opus-5-high
62
56
79
51
65
79
56
44
47
61
72
70
codex-cli:gpt-5.5
52
73
68
49
53
75
73
53
43
58
63
78
codex-cli:gpt-5.4-mini
54
59
72
50
48
74
70
51
38
59
69
79
codex-cli:gpt-5.6-terra
51
61
59
50
57
75
73
42
36
57
67
75
codex-cli:gpt-5.6-sol
55
62
65
50
44
75
58
46
44
56
68
71
codex-cli:gpt-5.6-terra-xhigh
53
56
69
48
43
76
70
45
35
60
67
71
codex-cli:gpt-5.3-codex-spark
55
42
65
42
53
75
69
46
40
60
66
72
nvidia-build:thinkingmachines/inkling@default
46
37
72
44
43
81
62
48
49
58
53
77
codex-cli:gpt-5.6-luna
42
64
63
48
45
75
66
44
35
60
67
57
nvidia-build:z-ai/glm-5.2@default
54
43
68
38
38
75
65
44
44
60
54
71
nvidia-build:nvidia/nemotron-3-ultra-550b-a55b@default
52
24
69
39
36
75
73
41
38
66
62
77
nvidia-build:minimaxai/minimax-m3@default
49
42
61
42
39
77
64
42
43
56
58
71
gemini-cli:gemini-3.1-pro-preview
49
32
74
37
31
74
71
39
41
60
56
73
mlx-openai:qwen/qwen3.6-35b-a3b
43
40
64
41
35
78
75
36
40
56
52
74
nvidia-build:deepseek-ai/deepseek-v4-flash@default
45
35
69
45
35
70
72
38
32
55
62
73
gemini-cli:gemini-3.1-flash-lite
47
24
71
33
31
70
69
39
38
61
63
68
nvidia-build:qwen/qwen3.5-397b-a17b@default
32
39
63
32
31
68
72
45
42
52
38
73
nvidia-build:qwen/qwen3-next-80b-a3b-instruct@default
42
42
57
28
31
67
70
32
34
50
52
71
nvidia-build:mistralai/mistral-medium-3.5-128b@default
39
27
59
32
35
62
55
43
31
62
56
68
nvidia-build:google/gemma-4-31b-it@default
42
24
65
28
18
70
63
39
36
58
54
70
nvidia-build:nvidia/nemotron-3-nano-omni-30b-a3b-reasoning@default
28
25
61
41
35
67
67
28
32
55
55
70
nvidia-build:openai/gpt-oss-20b@default
40
31
63
40
33
58
68
31
22
50
55
71
nvidia-build:mistralai/mistral-nemotron@default
49
26
55
30
37
58
56
38
34
55
60
61
nvidia-build:nvidia/nemotron-3-super-120b-a12b@default
42
27
67
30
30
61
64
27
31
45
59
73
nvidia-build:nvidia/llama-3.3-nemotron-super-49b-v1.5@default
41
21
56
34
32
61
59
38
33
57
58
64
nvidia-build:meta/llama-3.3-70b-instruct@default
44
32
55
30
23
60
73
41
27
49
26
66
nvidia-build:mistralai/mistral-large-3-675b-instruct-2512@default
45
27
71
29
31
62
54
34
32
52
41
46
nvidia-build:mistralai/mistral-small-4-119b-2603@default
34
21
53
28
34
64
65
29
29
52
43
67
nvidia-build:nvidia/nemotron-3-nano-30b-a3b@default
28
24
55
30
23
63
62
29
29
54
53
68
nvidia-build:google/diffusiongemma-26b-a4b-it@default
28
30
59
31
22
60
47
34
37
55
40
69
nvidia-build:nvidia/ising-calibration-1.5-31b@default
50
20
48
28
26
61
65
23
37
47
35
64
nvidia-build:openai/gpt-oss-120b@high
40
24
49
44
33
42
45
29
24
49
59
61
nvidia-build:nvidia/nvidia-nemotron-nano-9b-v2@default
33
19
45
33
31
63
61
20
33
46
49
65
nvidia-build:meta/llama-4-maverick-17b-128e-instruct@default
32
33
60
29
28
60
52
33
28
53
15
71
nvidia-build:poolside/laguna-xs-2.1@default
45
22
37
28
27
61
66
27
33
43
36
66

⑤ Efficiency frontier (Pareto)

Capability vs $/correct (non-dominated): claude-code-cli:claude-fable-5
Capability vs deployability: claude-code-cli:claude-fable-5 · claude-code-cli:claude-fable-5-early-july · mlx-openai:qwen/qwen3.6-35b-a3b

A model is on the frontier if no other is both more capable and cheaper per weighted-correct answer (or more deployable). Everything else is dominated.

③ Confidence calibration

ECE compares each model's stated or prior-filled confidence to its actual correctness. ECE coverage is confidence-bearing answers divided by total answers; abstention is the complement. Penalized ECE treats omitted confidence as 1.000 confidence, so silent wrong/partial answers count as worst-case overconfident. Brier is the binning-robust squared-error companion. Detail per bin in results/analytics.json.

⑥ CAT-lite screening set

Top 50 highest-discrimination tasks reproduce the full-set ranking at r=0.94 (39 models). Use as a fast first-pass screen; run the full 300 only for finalists.
Screening cost saving: $6.27 for 50 CAT-lite tasks vs $37.60 for the full 300; saves $31.33 (83.3%). ESTIMATE, mean observed priced-model cost per task, 3 priced model(s). Task ids in results/analytics.json → screeningSet.

② Held-out gap (independent private set heldout-v2)

Gap = public score − score on heldout-v2: 24 genuinely never-published tasks (matched category/points profile, per-task canaries, authored 2026-07-16). Positive gap beyond the threshold red-flags an overfitting signal; n=24, so single-digit gaps are indicative, not conclusive.

mixed · scored 26, pending 13, flagged 0 · threshold 5 points
claude-code-cli:claude-fable-5: HELD-OUT PENDING · public 67.77 · held-out n/a · canaryHits 0
claude-code-cli:claude-fable-5-early-july: -5.81 · public 65.37 · held-out 71.18 · canaryHits 0
kimi-cli:kimi-k3: -10.97 · public 64.65 · held-out 75.62 · canaryHits 0
codex-cli:gpt-5.5-medium: -9.12 · public 62.17 · held-out 71.29 · canaryHits 0
claude-code-cli:claude-opus-5-high: -13.02 · public 61.7 · held-out 74.72 · canaryHits 0
codex-cli:gpt-5.5: HELD-OUT PENDING · public 61.66 · held-out n/a · canaryHits 0
codex-cli:gpt-5.4-mini: -5.61 · public 60.22 · held-out 65.83 · canaryHits 0
codex-cli:gpt-5.6-terra: -6.56 · public 58.61 · held-out 65.17 · canaryHits 0
codex-cli:gpt-5.6-sol: -4.86 · public 57.86 · held-out 62.72 · canaryHits 0
codex-cli:gpt-5.6-terra-xhigh: -6.15 · public 57.69 · held-out 63.84 · canaryHits 0
codex-cli:gpt-5.3-codex-spark: -3.50 · public 57.23 · held-out 60.73 · canaryHits 0
nvidia-build:thinkingmachines/inkling@default: HELD-OUT PENDING · public 55.95 · held-out n/a · canaryHits 0
codex-cli:gpt-5.6-luna: -7.54 · public 55.49 · held-out 63.03 · canaryHits 0
nvidia-build:z-ai/glm-5.2@default: HELD-OUT PENDING · public 54.48 · held-out n/a · canaryHits 0
nvidia-build:nvidia/nemotron-3-ultra-550b-a55b@default: -6.86 · public 54.39 · held-out 61.25 · canaryHits 0
nvidia-build:minimaxai/minimax-m3@default: -17.66 · public 53.6 · held-out 71.26 · canaryHits 0
gemini-cli:gemini-3.1-pro-preview: HELD-OUT PENDING · public 53.03 · held-out n/a · canaryHits 0
mlx-openai:qwen/qwen3.6-35b-a3b: HELD-OUT PENDING · public 52.78 · held-out n/a · canaryHits 0
nvidia-build:deepseek-ai/deepseek-v4-flash@default: -5.01 · public 52.45 · held-out 57.46 · canaryHits 0
gemini-cli:gemini-3.1-flash-lite: HELD-OUT PENDING · public 51.15 · held-out n/a · canaryHits 0
nvidia-build:qwen/qwen3.5-397b-a17b@default: -17.78 · public 48.95 · held-out 66.73 · canaryHits 0
nvidia-build:qwen/qwen3-next-80b-a3b-instruct@default: -2.11 · public 48.13 · held-out 50.24 · canaryHits 0
nvidia-build:mistralai/mistral-medium-3.5-128b@default: -7.14 · public 47.53 · held-out 54.67 · canaryHits 0
nvidia-build:google/gemma-4-31b-it@default: -13.89 · public 47.13 · held-out 61.02 · canaryHits 0
nvidia-build:nvidia/nemotron-3-nano-omni-30b-a3b-reasoning@default: HELD-OUT PENDING · public 46.88 · held-out n/a · canaryHits 0
nvidia-build:openai/gpt-oss-20b@default: -15.34 · public 46.72 · held-out 62.06 · canaryHits 0
nvidia-build:mistralai/mistral-nemotron@default: HELD-OUT PENDING · public 46.56 · held-out n/a · canaryHits 0
nvidia-build:nvidia/nemotron-3-super-120b-a12b@default: -7.09 · public 46.27 · held-out 53.36 · canaryHits 0
nvidia-build:nvidia/llama-3.3-nemotron-super-49b-v1.5@default: -7.78 · public 46.09 · held-out 53.87 · canaryHits 0
nvidia-build:meta/llama-3.3-70b-instruct@default: -9.17 · public 43.69 · held-out 52.86 · canaryHits 0
nvidia-build:mistralai/mistral-large-3-675b-instruct-2512@default: -11.29 · public 43.64 · held-out 54.93 · canaryHits 0
nvidia-build:mistralai/mistral-small-4-119b-2603@default: -18.11 · public 43.36 · held-out 61.47 · canaryHits 0
nvidia-build:nvidia/nemotron-3-nano-30b-a3b@default: -5.56 · public 43.19 · held-out 48.75 · canaryHits 0
nvidia-build:google/diffusiongemma-26b-a4b-it@default: HELD-OUT PENDING · public 42.68 · held-out n/a · canaryHits 0
nvidia-build:nvidia/ising-calibration-1.5-31b@default: HELD-OUT PENDING · public 41.96 · held-out n/a · canaryHits 0
nvidia-build:openai/gpt-oss-120b@high: -11.19 · public 41.59 · held-out 52.78 · canaryHits 0
nvidia-build:nvidia/nvidia-nemotron-nano-9b-v2@default: HELD-OUT PENDING · public 41.57 · held-out n/a · canaryHits 0
nvidia-build:meta/llama-4-maverick-17b-128e-instruct@default: -9.43 · public 41.15 · held-out 50.58 · canaryHits 0
nvidia-build:poolside/laguna-xs-2.1@default: HELD-OUT PENDING · public 40.92 · held-out n/a · canaryHits 0

Synthetic, unclassified. Regenerate: npm run analytics.