{"generated_utc":"2026-09-28T00:39:48+00:00","edition":{"id":"v0.4","label":"Decision Index 0.4.1","date":"2026-10-21","panel_id":"decision-index-0.3","jev_version":"jev-1.14.0","jev_edition":"Lab run, 2026-10-14"},"suite":{"requests":110201,"scoreable":109759,"excluded":442,"source_cases":110313,"fields":536776,"static_benchmarks":37,"jev_static_benchmarks":42,"reproduction_benchmarks":37,"seed":20260919,"selection":"Deterministic hash sample without replacement; ESCI proportional locale × relevance; request-budget datasets retain complete linked groups; no outcome-based selection.","corpus_sha256":"b2b56d6fb636837ca469e689087bdbf373dda8de7638aa2da6793e6eda0792d5","created_utc":"2026-09-22T20:05:14.197492+00:00","jev_version":"jev-1.14.0","jev_edition":"Lab run, 2026-10-14","jev_api_cost_usd":12.2567,"jev_billed_tokens":{"input_tokens":291826668,"output_tokens":44483219},"hardware":{"label":"1 x NVIDIA RTX PRO 6000","reproductions":"1 x NVIDIA RTX PRO 6000 (Blackwell Server Edition, 96 GB), one GPU per run, isolated Python environment per engine","jev":"TypeSafe hosted API (jev-1.14.0) over HTTPS; the frozen suite's request payloads still name jev-1.13.0, the model they were built with"},"release_notes":["Original 500-case pilot remains frozen separately.","Request caps count planned logical evaluations. Transport retries or context splits are logged separately and may add HTTP attempts.","Cached successful pilot responses reused only for identical full request SHA256; original latency and provenance retained.","Game protocols and budgets are pinned separately.","Release v2 keeps every v1 row except a stratified subset cut of ToolRet (1,000 of 7,961 queries incl. excluded ones) and BRIGHT (550 of 1,384 queries); all other benchmarks, question order, options and scoring are unchanged. Kept ToolRet and BRIGHT queries retain their complete candidate lists.","ToolRet and BRIGHT rows of questions listed in excluded-questions.json were carried over unchanged (they are never scored); the scored subsets are 1,000 and 550 query groups, see scored_cases."],"jev_summary_notes":["Metrics exclude failed/incomplete cases; coverage is explicit. No silent replacement of failures.","Different primary metrics are not averaged across benchmarks.","Latency includes HTTPS and concurrent service scheduling; not isolated model inference.","Full raw HTTP responses/payloads and all retry timings are in http-attempts.jsonl.","BPoMP/RouterBench/GSM8K and chunked tasks retain all linked requests for sampled cases."],"archive_limitations":["Context fit not certified under Jev tokenizer.","No inference, accuracy, or latency measured.","Source archive counts are not active evaluation counts: see dataset-evaluation-counts.csv. Only ESCI is capped at 2000.","Some downloaded environments still need executable adapters.","Previously used benchmarks are exposed development evidence.","Adapted scores are not automatically comparable with native leaderboards.","Source duplicates are retained and flagged by structural audit."],"source_pinning":"Git/HF commit where available plus local SHA256. Direct university/GitHub raw releases pinned by content hash.","origins":{"Original-task benchmarks":"published questions, answer spaces and scoring retained; extraction of a clearly labeled supported subset or random sampling does not by itself create a new task.","Jev-adapted benchmarks":"published source data or environment, with a decision interface, answer construction, candidate retrieval, partner protocol or evaluated objective adapted for Jev.","Purpose-built benchmarks":"new local scenarios and evaluation logic; currently the executable home-appliance benchmark."},"adaptation_classes":{"A_direct":"Original finite choices, class labels, or probabilities map directly; only parsing, serialization and label mapping are needed.","B_mechanical":"Extra deterministic processing changes the evaluated view: filtering, extraction, search, enumeration, pairing, simulation or numeric encoding. No per-item semantic option authoring.","C_semantic_options":"The adaptation relies on authored semantic option spaces/scenarios or model-generated candidate answers absent from the original native answer format.","not_applicable":"Removed, skipped, or unspecified; no active adaptation classification."},"licence_caveat":"Licences are read from the archived source repository or dataset card and describe that artefact, not necessarily every text inside it; public availability is not a blanket grant to republish the underlying benchmark content.","edition":"release-v4","label":"Decision Index 0.4.1","base_edition":"release-v2.1","note":null},"excluded_questions":{"purpose":"Questions dropped from scoring for every engine so no model is graded on a case another model was structurally unable to attempt.","frozen_corpus":"The frozen corpus and its sha256 are unchanged; exclusion is applied at scoring time only.","criterion":"A row is excluded for reach only when nearly the whole field cannot attempt it. A limit that one model has and others do not is that model's result, not a defect in the suite, and is never grounds for removing the row.","provenance":"Defects verified against the upstream sources; cais/mmlu ships the duplicate choices verbatim and the builder applies no normalisation to MMLU.","rules":[{"reason":"duplicate_options","n":17,"rule":"Two or more options are byte-identical. A distinct-options contract (Solomon) refuses these outright, so scoring them rewards engines with looser input validation."},{"reason":"gold_duplicated","n":6,"rule":"The correct answer is itself one of the identical strings, so a textually correct choice is scored wrong at chance. Unanswerable."},{"reason":"whitespace_duplicate_options","n":3,"rule":"MuSR ships \"pantry\" and \"pantry \" as separate locations; the trailing space carries no task meaning."},{"reason":"gold_whitespace_duplicated","n":1,"rule":"As above, and the gold is the space-variant."},{"reason":"beyond_nearly_every_context_window","n":380,"rule":"Rows longer than almost every entrant can accept. Measured across the 27 completed sweeps, 14 answer none of them and only 4 answer all; the rest manage a fifth. They cannot separate models, and the group-completeness rule already drops them for most engines, so excluding them uniformly removes an incomparability rather than a result.","datasets":{"ToolRet":293,"BRIGHT":87},"engines_answering_all":4,"engines_answering_none":14,"engines_measured":27},{"reason":"sibling_chunk_of_an_excluded_group","n":35,"rule":"ToolRet partitions a query across chunks. Dropping only the over-long chunks would leave the query scored against a reduced candidate set, which is an easier question than the benchmark asks, so the surviving chunks of a partly excluded query are removed with it.","datasets":{"ToolRet":35}}],"counts":{"rows":442,"all_single_question":true,"by_dataset":{"HLE":12,"MMLU":9,"MuSR":4,"GPQA Diamond":2},"share_of_choice_questions":3.48e-05,"by_reason":{"defective":27,"beyond_reach":380,"sibling_chunks":35}},"deliberately_not_excluded":{"CRUXEval":"40 questions whose options differ only by case or whitespace are legitimate: they are Python values, and the case and spacing are exactly what the task tests.","MMLU/ARC/GPQA case collisions":"genotype notation (BB/Bb/bb), logic predicates (Cs/sC) and variable names (l/L) are semantic; only a case-blind comparison makes them look duplicated."},"by_dataset_and_reason":[{"dataset":"ToolRet","catalog_id":2,"reason":"beyond_nearly_every_context_window","n":293},{"dataset":"BRIGHT","catalog_id":36,"reason":"beyond_nearly_every_context_window","n":87},{"dataset":"ToolRet","catalog_id":2,"reason":"sibling_chunk_of_an_excluded_group","n":35},{"dataset":"HLE","catalog_id":45,"reason":"duplicate_options","n":10},{"dataset":"MMLU","catalog_id":24,"reason":"duplicate_options","n":5},{"dataset":"MMLU","catalog_id":24,"reason":"gold_duplicated","n":4},{"dataset":"MuSR","catalog_id":32,"reason":"whitespace_duplicate_options","n":3},{"dataset":"GPQA Diamond","catalog_id":25,"reason":"duplicate_options","n":2},{"dataset":"HLE","catalog_id":45,"reason":"gold_duplicated","n":2},{"dataset":"MuSR","catalog_id":32,"reason":"gold_whitespace_duplicated","n":1}]},"benchmarks":[{"id":1,"dataset":"BFCL","short":null,"kind":"static","area":"tools","area_label":"Tools & Automation","area_color":"#b77a38","frozen_panel_area":"tools","in_panel":true,"in_frozen_panel":true,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"case exact accuracy","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"gold":1.2,"explainer":"Says yes or no for each supplied function: call it for this request? All must be right.","comparability_note":null,"references":[{"kind":"paper","url":"https://arxiv.org/abs/2305.15334"},{"kind":"code","url":"https://github.com/gorilla-llm/gorilla"}],"cases":1694,"requests":1694,"fields":4768,"source_cases":1694,"unit":"case","scoring_metric":"tool-name exact-set accuracy","subset_rule":"All 1,694 prepared cases retained.","source_filtering":null,"note":"Published candidate schemas projected to tool-name decisions. Arguments/order/execution are not evaluated. Source-to-label gold audit required in addition to schema checks. Release v1: All prepared cases retained.","suite_category":"Tool use, routing and automation","subcategory":"Tool-name selection","origin":"Jev-adapted benchmarks","adaptation":{"class":"B_mechanical","class_description":"Extra deterministic processing changes the evaluated view: filtering, extraction, search, enumeration, pairing, simulation or numeric encoding. No per-item semantic option authoring.","reason":"Projects structured calls to tool-name decisions; source argument/execution task is reduced.","candidate_origin":"Published tool schemas"},"random_baseline":{"value":0.2592,"label":"Random whole-case guesses","kind":"random_exact_case","method":"Exact expectation","explanation":"Mean probability that every field in a case is correct under independent uniform guesses; multiply field probabilities within each case, then average cases. This is not the per-field chance rate.","monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.964},"sources":[{"name":"gorilla-llm/gorilla","url":"https://github.com/gorilla-llm/gorilla.git","revision":"916260dfc116bf06793a1af79b4ec8195b0453b6"}],"licences":[{"licence":"Apache-2.0","source":"archived source file","file":"LICENSE"}],"source_revision":"916260dfc116bf06793a1af79b4ec8195b0453b6"},{"id":2,"dataset":"ToolRet","short":"ToolRet","kind":"static","area":"tools","area_label":"Tools & Automation","area_color":"#b77a38","frozen_panel_area":"tools","in_panel":true,"in_frozen_panel":true,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"nDCG@10","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":"Rates how useful each of 32 search-hit tools is for a request; ranked by nDCG@10, on requests where at least one of the 32 is relevant.","comparability_note":null,"references":[{"kind":"paper","url":"https://arxiv.org/abs/2503.01763"},{"kind":"dataset","url":"https://huggingface.co/datasets/mangopy/ToolRet-Queries"}],"cases":685,"requests":1013,"fields":254650,"source_cases":7961,"unit":"retrieval query","scoring_metric":"nDCG@10; Recall@10; MRR; candidate recall","subset_rule":"1,000 of 7,704 scoreable v1 queries kept (1,328 requests): stratified by source domain, proportional allocation with at least one query per domain (seed 20260922); kept queries keep their complete candidate lists; no outcome-based selection.","source_filtering":"Candidate policy: query-only lexical overlap top32; no gold insertion; unjudged candidates are not treated as verified negatives.","note":"Query-only FTS5 BM25 candidate retrieval plus Jev relevance ranking. Candidate recall and nDCG/MRR required; unjudged candidates are not verified negatives. See retrieval audit for readiness. Release v1: All prepared cases retained.","suite_category":"Tool use, routing and automation","subcategory":"Tool retrieval","origin":"Jev-adapted benchmarks","adaptation":{"class":"B_mechanical","class_description":"Extra deterministic processing changes the evaluated view: filtering, extraction, search, enumeration, pairing, simulation or numeric encoding. No per-item semantic option authoring.","reason":"FTS5 BM25 candidate search, bounded pool, context packing and relevance ranking.","candidate_origin":"Query-only search over published tool corpus"},"random_baseline":{"value":0.1341,"label":"Random ranking, answerable queries","kind":"random_ranking","method":"Exact expectation, answerable queries","explanation":"Mean over the scored queries of the expected nDCG@10 under a uniform random permutation of that query's 32 candidates, using the full relevance judgments for the denominator; only queries with at least one relevant candidate are scored. This is the value the index uses.","monte_carlo_standard_error":null,"secondary_reference":{"label":"BM25 ranking","value":0.3717}},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.6505},"sources":[{"name":"mangopy/tool-retrieval-benchmark","url":"https://github.com/mangopy/tool-retrieval-benchmark.git","revision":"c4181d914a227134705ecb6bab13fbd92ccd2938"},{"name":"mangopy/ToolRet-Queries","url":"https://huggingface.co/datasets/mangopy/ToolRet-Queries","revision":null},{"name":"mangopy/ToolRet-Tools","url":"https://huggingface.co/datasets/mangopy/ToolRet-Tools","revision":null}],"licences":[{"licence":"Apache-2.0","source":"archived source file","file":"LICENSE"}],"source_revision":"c4181d914a227134705ecb6bab13fbd92ccd2938"},{"id":3,"dataset":"API-Bank","short":null,"kind":"static","area":"tools","area_label":"Tools & Automation","area_color":"#b77a38","frozen_panel_area":null,"in_panel":true,"in_frozen_panel":false,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"accuracy","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"gold":1.2,"explainer":"Reads an assistant conversation and picks the next API to call from a catalog of 53.","comparability_note":null,"references":[{"kind":"paper","url":"https://arxiv.org/abs/2304.08244"},{"kind":"code","url":"https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/api-bank"}],"cases":508,"requests":508,"fields":508,"source_cases":508,"unit":"API decision point","scoring_metric":"tool-name selection accuracy","subset_rule":"All 508 prepared cases retained.","source_filtering":"508 API decision points from 264 source files over the full 53-tool catalog; 3 excluded; every gold answer verified to be a candidate.","note":"508 source API decision points, full53-tool published catalog; prior observed results allowed, current/future results hidden. Tool-name selection only, not argument/execution success. Release v1: All prepared cases retained.","suite_category":"Tool use, routing and automation","subcategory":"Sequential tool-name selection","origin":"Jev-adapted benchmarks","adaptation":{"class":"B_mechanical","class_description":"Extra deterministic processing changes the evaluated view: filtering, extraction, search, enumeration, pairing, simulation or numeric encoding. No per-item semantic option authoring.","reason":"Extracts each source API decision and preceding dialogue; arguments/execution excluded.","candidate_origin":"Published API catalog"},"random_baseline":{"value":0.0189,"label":"Uniform random choice","kind":"random_accuracy","method":"Exact expectation","explanation":"Average accepted-answer count / option count across the actual sampled fields. Includes all tied correct moves/statements where the scorer accepts ties; variable option counts are averaged case by case.","monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.8661},"sources":[{"name":"AlibabaResearch/DAMO-ConvAI","url":"https://github.com/AlibabaResearch/DAMO-ConvAI.git","revision":"188835d4f9948563a6b9c8ac50cd0f3ae4021ed6"}],"licences":[{"licence":"MIT","source":"archived source file","file":"LICENSE"},{"licence":"Apache-2.0","source":"archived source file","file":"api-bank/LICENSE"}],"source_revision":"188835d4f9948563a6b9c8ac50cd0f3ae4021ed6"},{"id":4,"dataset":"BANKING77","short":null,"kind":"static","area":"retrieval","area_label":"Retrieval & Classification","area_color":"#3e8e5e","frozen_panel_area":null,"in_panel":true,"in_frozen_panel":false,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"macro-F1","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"gold":1.2,"explainer":"Picks the intent of a bank customer's message, e.g. \"How do I locate my card?\", from 77.","comparability_note":null,"references":[{"kind":"paper","url":"https://arxiv.org/abs/2003.04807"},{"kind":"dataset","url":"https://github.com/PolyAI-LDN/task-specific-datasets"}],"cases":3080,"requests":3080,"fields":3080,"source_cases":3080,"unit":"case","scoring_metric":"intent accuracy; macro-F1","subset_rule":"All 3,080 prepared cases retained.","source_filtering":null,"note":"Release v1: All prepared cases retained.","suite_category":"Language understanding and classification","subcategory":"Banking intent","origin":"Original-task benchmarks","adaptation":{"class":"A_direct","class_description":"Original finite choices, class labels, or probabilities map directly; only parsing, serialization and label mapping are needed.","reason":"Native intent classification with the released label vocabulary.","candidate_origin":"Original77 intent labels"},"random_baseline":{"value":0.0127,"label":"Random choices · F1","kind":"random_f1","method":"Monte Carlo","explanation":"Uniform independent choice among each field's actual options, scored with the same F1 definition and observed/predicted label union as the model. F1 is nonlinear and class-frequency dependent; 1/K is not its expected value. This simulation error is not uncertainty in model quality.","monte_carlo_standard_error":6.55e-05,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.7994},"sources":[{"name":"PolyAI/banking77","url":"https://huggingface.co/datasets/PolyAI/banking77","revision":null}],"licences":[{"licence":"CC-BY-4.0","source":"archived dataset card","file":"README.md"}],"source_revision":null},{"id":5,"dataset":"CLINC150+OOS","short":"CLINC150","kind":"static","area":"retrieval","area_label":"Retrieval & Classification","area_color":"#3e8e5e","frozen_panel_area":null,"in_panel":true,"in_frozen_panel":false,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"macro-F1","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"gold":1.2,"explainer":"Reads a short assistant request and picks its intent from 150, or \"out of scope\" when none fits.","comparability_note":null,"references":[{"kind":"dataset","url":"https://github.com/clinc/oos-eval"}],"cases":5500,"requests":5500,"fields":5500,"source_cases":5500,"unit":"case","scoring_metric":"intent accuracy; OOS F1","subset_rule":"All 5,500 prepared cases retained.","source_filtering":null,"note":"Release v1: All prepared cases retained.","suite_category":"Language understanding and classification","subcategory":"Intent and out-of-scope detection","origin":"Original-task benchmarks","adaptation":{"class":"A_direct","class_description":"Original finite choices, class labels, or probabilities map directly; only parsing, serialization and label mapping are needed.","reason":"Native intent/OOS classification; same fixed label set for every example.","candidate_origin":"Original150 intents plus OOS"},"random_baseline":{"value":0.006,"label":"Random choices · F1","kind":"random_f1","method":"Monte Carlo","explanation":"Uniform independent choice among each field's actual options, scored with the same F1 definition and observed/predicted label union as the model. F1 is nonlinear and class-frequency dependent; 1/K is not its expected value. This simulation error is not uncertainty in model quality.","monte_carlo_standard_error":3.34e-05,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.9028},"sources":[{"name":"clinc/oos-eval","url":"https://github.com/clinc/oos-eval","revision":null}],"licences":[],"source_revision":null},{"id":6,"dataset":"RouterBench","short":"RouterBench","kind":"static","area":"tools","area_label":"Tools & Automation","area_color":"#b77a38","frozen_panel_area":"tools","in_panel":false,"in_frozen_panel":true,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"selected quality (quality objective)","lower_is_better":false,"index_rule":null,"left_out_of_index":true,"index_note":"Not in the index: the prompt reveals that GPT-4 is the best route, so the top models all pick GPT-4 and score within a fraction of a point of each other. Scores stay visible.","explainer":"Picks which of 11 AI models should answer a prompt; scored by its real answer.","comparability_note":null,"references":[{"kind":"paper","url":"https://arxiv.org/abs/2403.12031"},{"kind":"dataset","url":"https://huggingface.co/datasets/withmartian/routerbench"}],"cases":5001,"requests":10000,"fields":20000,"source_cases":29163,"unit":"underlying routed prompt","scoring_metric":"realized quality; cost; utility regret","subset_rule":"10,000 requests within a 10,000-request cap: 5,001 complete linked case groups of 29,163 prepared; groups kept intact.","source_filtering":"Offline released outcomes; calibration-derived profiles only. No live candidate model calls. Both shot regimes retained; report separately and cluster by source group. 28 exclusions recorded; cost penalty 10.0 per USD.","note":"Both0-shot/5-shot historical outcome tables; shared hash-grouped20% calibration split, remaining cases evaluated. Profile quality/cost derived only from calibration. Custom quality/cost utility scorer accepts ties; no live candidate calls. Release v1: 10000 requests within 10000 cap; complete linked cases retained.","suite_category":"Tool use, routing and automation","subcategory":"Model routing","origin":"Jev-adapted benchmarks","adaptation":{"class":"B_mechanical","class_description":"Extra deterministic processing changes the evaluated view: filtering, extraction, search, enumeration, pairing, simulation or numeric encoding. No per-item semantic option authoring.","reason":"Calibration-only profiles, explicit quality/cost objectives, held-out outcome scoring.","candidate_origin":"Original11 candidate models"},"random_baseline":{"value":0.5735,"label":"Random model selection","kind":"random_routing","method":"Exact expectation","explanation":"Expected realized quality from selecting uniformly among the available models on each request. This is outcome quality, not the probability of matching one correct answer.","monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[{"track":"RouterBench-0shot","label":"Zero-shot outcome table","metric":"Realized selected-model quality","description":"Choose a model using profiles built from a separate calibration split; evaluate against recorded zero-shot outcomes. Each request tests quality-only and cost-aware objectives.","random":0.5245,"headline":false,"in_index":true},{"track":"RouterBench-5shot","label":"Five-shot outcome table","metric":"Realized selected-model quality","description":"The same routing decision, evaluated against recorded outcomes from models given five examples. Jev is not making live calls to the candidate models.","random":0.6225,"headline":false,"in_index":true}],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.7988},"sources":[{"name":"withmartian/routerbench","url":"https://github.com/withmartian/routerbench.git","revision":"cc67d1008bd8f3cf1e8040cc3ba4034d31b93c0c"},{"name":"withmartian/routerbench","url":"https://huggingface.co/datasets/withmartian/routerbench","revision":null}],"licences":[{"licence":"MIT","source":"archived source file","file":"LICENSE"}],"source_revision":"cc67d1008bd8f3cf1e8040cc3ba4034d31b93c0c"},{"id":7,"dataset":"MiniWoB++","short":"MiniWoB++","kind":"interactive","area":"tools","area_label":"Tools & Automation","area_color":"#b77a38","frozen_panel_area":"tools","in_panel":false,"in_frozen_panel":true,"dropped_interactive":true,"in_reproduction_suite":false,"metric":"episode success","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":null,"comparability_note":null,"references":[],"cases":9,"requests":null,"fields":null,"source_cases":null,"unit":"episode","scoring_metric":"native episode success","subset_rule":"9 configured episodes; fixed seeds; no performance-based replacement.","source_filtering":null,"note":"Dynamic request count depends on trajectory length. Configured count remains user-selectable. implemented; native headless browser smoke passed","suite_category":"Computer use and embodied decisions","subcategory":"Browser interaction","origin":"Jev-adapted benchmarks","adaptation":{"class":"B_mechanical","class_description":"Extra deterministic processing changes the evaluated view: filtering, extraction, search, enumeration, pairing, simulation or numeric encoding. No per-item semantic option authoring.","reason":"Click-only task filtering and native browser action enumeration.","candidate_origin":"Visible DOM actions"},"random_baseline":{"value":null,"label":null,"kind":null,"method":null,"explanation":null,"monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"metric":null,"episodes":null,"interval_95":null,"interval_method":null,"http_median_ms":null,"http_p95_ms":null,"score":null},"sources":[{"name":"Farama-Foundation/miniwob-plusplus","url":"https://github.com/Farama-Foundation/miniwob-plusplus.git","revision":"33c3b4ddef8c6eb67c57a29663d844b1eda7e614"}],"licences":[{"licence":"MIT","source":"archived source file","file":"LICENSE"}],"source_revision":"33c3b4ddef8c6eb67c57a29663d844b1eda7e614","protocol":null},{"id":9,"dataset":"Home appliance simulator","short":"Home appliances","kind":"static","area":"tools","area_label":"Tools & Automation","area_color":"#b77a38","frozen_panel_area":null,"in_panel":true,"in_frozen_panel":false,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"case exact accuracy","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":"Given a simulated home and a command, answers ~18 questions on handling it; all must be right. Duplicate and dev-set rows removed.","comparability_note":null,"references":[{"kind":"code","url":"https://github.com/apolinario/decision-index/blob/main/decision_index/suite/build/home_appliance.py"}],"cases":88,"requests":88,"fields":2895,"source_cases":160,"unit":"household request","scoring_metric":"whole-request exact match; executable state correctness; per-field accuracy","subset_rule":"All 160 prepared cases retained.","source_filtering":null,"note":"160 test households/requests, 2895 fields; separate 80-case dev file. Executable intent oracle and semantic validation; structural novelty does not establish pretraining or semantic independence. Release v1: All prepared cases retained.","suite_category":"Computer use and embodied decisions","subcategory":"Home appliance control","origin":"Purpose-built benchmarks","adaptation":{"class":"C_semantic_options","class_description":"The adaptation relies on authored semantic option spaces/scenarios or model-generated candidate answers absent from the original native answer format.","reason":"Custom household scenarios and decision fields authored for this suite, with executable oracle. Not a native public benchmark.","candidate_origin":"Locally authored device/command scenarios"},"random_baseline":{"value":0,"label":"Random whole-case guesses","kind":"random_exact_case","method":"Exact expectation, kept rows","explanation":"Mean probability that every field in a case is correct under independent uniform guesses, over the kept rows.","monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.4545},"sources":[{"name":"purpose-built locally (evaluation/home_benchmark)","url":null,"revision":null}],"licences":[],"source_revision":null},{"id":10,"dataset":"SGD/SGD-X","short":"SGD","kind":"static","area":"retrieval","area_label":"Retrieval & Classification","area_color":"#3e8e5e","frozen_panel_area":null,"in_panel":false,"in_frozen_panel":false,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"macro-F1","lower_is_better":false,"index_rule":null,"left_out_of_index":true,"index_note":"Not in the index: build bug, pending re-run. The builder gave every SGD row the whole dialogue, including the turns after the one being classified. Scores stay visible; SGD returns to the index once re-run with the fixed builder.","explainer":"Reads a service dialogue so far and picks the intent the user is pursuing now, or NONE.","comparability_note":null,"references":[{"kind":"paper","url":"https://arxiv.org/abs/1909.05855"},{"kind":"dataset","url":"https://github.com/google-research-datasets/dstc8-schema-guided-dialogue"}],"cases":2500,"requests":2500,"fields":2500,"source_cases":46116,"unit":"user-turn/service intent","scoring_metric":"service-given intent accuracy","subset_rule":"2,500 of 46,116 prepared cases, deterministic hash sample without replacement (seed 20260919); no outcome-based selection.","source_filtering":null,"note":"Service-given current-user-turn intent classification on official SGD test dialogues; SGD-X and freeform slot extraction are not yet evaluated. Release v1: 2500 seeded cases.","suite_category":"Language understanding and classification","subcategory":"Dialogue intent","origin":"Jev-adapted benchmarks","adaptation":{"class":"B_mechanical","class_description":"Extra deterministic processing changes the evaluated view: filtering, extraction, search, enumeration, pairing, simulation or numeric encoding. No per-item semantic option authoring.","reason":"Selects current user turns, preserves preceding history, and supplies the service; not full dialogue-state generation.","candidate_origin":"Original service schemas and intent labels"},"random_baseline":{"value":0.399,"label":"Random choices · F1","kind":"random_f1","method":"Monte Carlo","explanation":"Uniform independent choice among each field's actual options, scored with the same F1 definition and observed/predicted label union as the model. F1 is nonlinear and class-frequency dependent; 1/K is not its expected value. This simulation error is not uncertainty in model quality.","monte_carlo_standard_error":0.00036,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.0903},"sources":[{"name":"google-research-datasets/dstc8-schema-guided-dialogue","url":"https://github.com/google-research-datasets/dstc8-schema-guided-dialogue.git","revision":"e852981ae34990f4358979625854259302feaa78"}],"licences":[{"licence":"CC-BY-SA-4.0","source":"archived source file","file":"LICENSE.txt"}],"source_revision":"e852981ae34990f4358979625854259302feaa78"},{"id":11,"dataset":"ContractNLI","short":null,"kind":"static","area":"language","area_label":"Language Understanding","area_color":"#407e9e","frozen_panel_area":"language","in_panel":true,"in_frozen_panel":true,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"macro-F1","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":"Says whether a full NDA supports, contradicts or omits each of 17 fixed statements.","comparability_note":null,"references":[{"kind":"paper","url":"https://aclanthology.org/2021.findings-emnlp.164"},{"kind":"dataset","url":"https://stanfordnlp.github.io/contract-nli/"}],"cases":123,"requests":123,"fields":2091,"source_cases":123,"unit":"case","scoring_metric":"macro-F1; accuracy","subset_rule":"All 123 prepared cases retained.","source_filtering":"123 source cases, 2,091 decisions.","note":"123 test contracts, 17 original hypotheses each. Native classification track; evidence extraction not yet adapted. Release v1: All prepared cases retained.","suite_category":"Document reasoning and retrieval","subcategory":"Contract entailment","origin":"Original-task benchmarks","adaptation":{"class":"A_direct","class_description":"Original finite choices, class labels, or probabilities map directly; only parsing, serialization and label mapping are needed.","reason":"Released document/hypothesis classification projected directly; evidence extraction not claimed.","candidate_origin":"Original hypotheses and NLI labels"},"random_baseline":{"value":0.3085,"label":"Random choices · F1","kind":"random_f1","method":"Monte Carlo","explanation":"Uniform independent choice among each field's actual options, scored with the same F1 definition and observed/predicted label union as the model. F1 is nonlinear and class-frequency dependent; 1/K is not its expected value. This simulation error is not uncertainty in model quality.","monte_carlo_standard_error":0.000301,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.7076},"sources":[{"name":"stanfordnlp/contract-nli","url":"https://github.com/stanfordnlp/contract-nli.git","revision":"eced6528dd3c1d14d73f9a87df8f7bdbc03126f9"}],"licences":[{"licence":"CC-BY-4.0","source":"archived source file","file":"LICENSE"}],"source_revision":"eced6528dd3c1d14d73f9a87df8f7bdbc03126f9"},{"id":12,"dataset":"ANLI","short":null,"kind":"static","area":"language","area_label":"Language Understanding","area_color":"#407e9e","frozen_panel_area":null,"in_panel":true,"in_frozen_panel":false,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"macro-F1","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"gold":1.2,"explainer":"Decides whether a claim follows from, contradicts, or is neutral to a premise written to fool AI models.","comparability_note":null,"references":[{"kind":"paper","url":"https://arxiv.org/abs/1910.14599"},{"kind":"dataset","url":"https://huggingface.co/datasets/facebook/anli"}],"cases":3200,"requests":3200,"fields":3200,"source_cases":3200,"unit":"case","scoring_metric":"accuracy by ANLI round","subset_rule":"All 3,200 prepared cases retained.","source_filtering":null,"note":"Release v1: All prepared cases retained.","suite_category":"Language understanding and classification","subcategory":"Adversarial entailment","origin":"Original-task benchmarks","adaptation":{"class":"A_direct","class_description":"Original finite choices, class labels, or probabilities map directly; only parsing, serialization and label mapping are needed.","reason":"Native three-way entailment classification.","candidate_origin":"Original3 NLI labels"},"random_baseline":{"value":0.3324,"label":"Random choices · F1","kind":"random_f1","method":"Monte Carlo","explanation":"Uniform independent choice among each field's actual options, scored with the same F1 definition and observed/predicted label union as the model. F1 is nonlinear and class-frequency dependent; 1/K is not its expected value. This simulation error is not uncertainty in model quality.","monte_carlo_standard_error":0.000254,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.7427},"sources":[{"name":"facebookresearch/anli","url":"https://github.com/facebookresearch/anli","revision":null}],"licences":[{"licence":"CC-BY-NC-4.0","source":"archived dataset card","file":"README.md"}],"source_revision":null},{"id":13,"dataset":"Boxoban","short":"Boxoban","kind":"interactive","area":"games","area_label":"Games & Strategy","area_color":"#ad6f48","frozen_panel_area":"games","in_panel":false,"in_frozen_panel":true,"dropped_interactive":true,"in_reproduction_suite":false,"metric":"levels solved within 255 moves","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":null,"comparability_note":null,"references":[],"cases":100,"requests":null,"fields":null,"source_cases":null,"unit":"level","scoring_metric":"fraction of levels solved; moves and pushes","subset_rule":"100 hard Boxoban levels reused unchanged from the existing pilot; each scored within a 255-move budget.","source_filtering":null,"note":"Dynamic request count depends on trajectory length. Configured count remains user-selectable. implemented","suite_category":"Games and strategic decisions","subcategory":"Sokoban planning","origin":"Jev-adapted benchmarks","adaptation":{"class":"B_mechanical","class_description":"Extra deterministic processing changes the evaluated view: filtering, extraction, search, enumeration, pairing, simulation or numeric encoding. No per-item semantic option authoring.","reason":"Native levels and deterministic transitions; score level completion, not a fabricated next-move label.","candidate_origin":"Simulator movement actions"},"random_baseline":{"value":null,"label":null,"kind":null,"method":null,"explanation":null,"monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"metric":null,"episodes":null,"mean_best_goal_fraction":null,"score":null},"sources":[{"name":"google-deepmind/boxoban-levels","url":"https://github.com/google-deepmind/boxoban-levels.git","revision":"78011d8c86ad423bfb2e1005012393b4c3c24409"}],"licences":[{"licence":"Apache-2.0","source":"archived source file","file":"LICENSE"}],"source_revision":"78011d8c86ad423bfb2e1005012393b4c3c24409"},{"id":14,"dataset":"BabyAI/MiniGrid","short":null,"kind":"interactive","area":"tools","area_label":"Tools & Automation","area_color":"#b77a38","frozen_panel_area":null,"in_panel":false,"in_frozen_panel":false,"dropped_interactive":false,"in_reproduction_suite":false,"metric":"episode success","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":null,"comparability_note":null,"references":[],"cases":100,"requests":null,"fields":null,"source_cases":null,"unit":"episode","scoring_metric":"native success rate; reward","subset_rule":"12 configured episodes; fixed seeds; no performance-based replacement.","source_filtering":null,"note":"Dynamic request count depends on trajectory length. Configured count remains user-selectable. implemented; requires benchmark venv","suite_category":"Computer use and embodied decisions","subcategory":"Instruction-following navigation","origin":"Jev-adapted benchmarks","adaptation":{"class":"B_mechanical","class_description":"Extra deterministic processing changes the evaluated view: filtering, extraction, search, enumeration, pairing, simulation or numeric encoding. No per-item semantic option authoring.","reason":"Simulator wrapper with partial observation and native success.","candidate_origin":"Native MiniGrid actions"},"random_baseline":{"value":null,"label":null,"kind":null,"method":null,"explanation":null,"monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"metric":null,"episodes":null,"interval_95":null,"interval_method":null,"http_median_ms":null,"http_p95_ms":null,"score":null},"sources":[{"name":"Farama-Foundation/Minigrid","url":"https://github.com/Farama-Foundation/Minigrid.git","revision":"8ea099e114b7d6465afabc76957e3d689663534c"}],"licences":[{"licence":"Apache-2.0","source":"archived source file","file":"LICENSE"}],"source_revision":"8ea099e114b7d6465afabc76957e3d689663534c","protocol":null},{"id":15,"dataset":"RTFM","short":"RTFM","kind":"interactive","area":"games","area_label":"Games & Strategy","area_color":"#ad6f48","frozen_panel_area":"games","in_panel":false,"in_frozen_panel":true,"dropped_interactive":true,"in_reproduction_suite":false,"metric":"Native win rate (100-turn pilot)","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":null,"comparability_note":null,"references":[],"cases":100,"requests":null,"fields":null,"source_cases":null,"unit":"episode","scoring_metric":"episode win rate","subset_rule":"100 configured episodes; Fixed seeds 0..N-1; no performance-based replacement. Codenames uses each of66 eligible boards exactly once.","source_filtering":null,"note":"Dynamic request count depends on trajectory length. Configured count remains user-selectable. implemented; native source dynamics smoke passed","suite_category":"Computer use and embodied decisions","subcategory":"Manual-guided interaction","origin":"Jev-adapted benchmarks","adaptation":{"class":"B_mechanical","class_description":"Extra deterministic processing changes the evaluated view: filtering, extraction, search, enumeration, pairing, simulation or numeric encoding. No per-item semantic option authoring.","reason":"Source simulator, held-out generation configuration and player-visible observation.","candidate_origin":"Native RTFM actions"},"random_baseline":{"value":null,"label":null,"kind":null,"method":null,"explanation":null,"monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"metric":null,"episodes":null,"interval_95":null,"interval_method":null,"http_median_ms":null,"http_p95_ms":null,"score":null},"sources":[{"name":"facebookresearch/RTFM","url":"https://github.com/facebookresearch/RTFM.git","revision":"58f17955595b5a127c96d045d896fcbcc7d4b570"}],"licences":[{"licence":"CC-BY-NC-4.0","source":"archived source file","file":"LICENSE"}],"source_revision":"58f17955595b5a127c96d045d896fcbcc7d4b570","protocol":{"observation":"Native public observations plus explicit game rules and last12 public observations; hidden cards/roles omitted.","actions":"Full native option strings; one Jev decision followed by environment transition. One available action is automatic and logged.","sample":"Fixed seeds 0..N-1; no performance-based replacement. Codenames uses each of66 eligible boards exactly once.","partner":"Same seeded random teammate/opponent as pilot for Hanabi/Leduc; fixed lexical partner for Codenames.","scoring":"Native terminal team points for Hanabi; player0 net chips per hand for Leduc; native wins for RTFM; adapter wins for Codenames.","budget":{"room_shape":[10,10],"shuffle_wiki":true,"max_actions":100,"source_class":"GroupsNLDev","max_turns":100},"limitations":"Fixed-partner performance, not optimal partnership or exploitability. RTFM is a100-turn partial-observation track; Codenames is a lexical-partner adaptation. Intervals describe this protocol and sample, not all possible partners/boards."}},{"id":16,"dataset":"ScienceWorld","short":"ScienceWorld","kind":"interactive","area":"tools","area_label":"Tools & Automation","area_color":"#b77a38","frozen_panel_area":"tools","in_panel":false,"in_frozen_panel":true,"dropped_interactive":true,"in_reproduction_suite":false,"metric":"episode success","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":null,"comparability_note":null,"references":[],"cases":10,"requests":null,"fields":null,"source_cases":null,"unit":"episode","scoring_metric":"native task completion reward","subset_rule":"10 configured episodes; fixed seeds; no performance-based replacement.","source_filtering":null,"note":"Dynamic request count depends on trajectory length. Configured count remains user-selectable. implemented; native JVM semantic probe passed","suite_category":"Computer use and embodied decisions","subcategory":"Scientific experimentation","origin":"Jev-adapted benchmarks","adaptation":{"class":"B_mechanical","class_description":"Extra deterministic processing changes the evaluated view: filtering, extraction, search, enumeration, pairing, simulation or numeric encoding. No per-item semantic option authoring.","reason":"Simulator action enumeration and lossless hierarchy for more than255 commands.","candidate_origin":"Native legal commands"},"random_baseline":{"value":null,"label":null,"kind":null,"method":null,"explanation":null,"monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"metric":null,"episodes":null,"interval_95":null,"interval_method":null,"http_median_ms":null,"http_p95_ms":null,"score":null},"sources":[{"name":"allenai/ScienceWorld","url":"https://github.com/allenai/ScienceWorld.git","revision":"e8216d6044e8e39be9fcb185e3b2dfb602584b52"}],"licences":[{"licence":"Apache-2.0","source":"archived source file","file":"LICENSE"}],"source_revision":"e8216d6044e8e39be9fcb185e3b2dfb602584b52","protocol":null},{"id":17,"dataset":"Hanabi","short":"Hanabi","kind":"interactive","area":"games","area_label":"Games & Strategy","area_color":"#ad6f48","frozen_panel_area":"games","in_panel":false,"in_frozen_panel":true,"dropped_interactive":true,"in_reproduction_suite":false,"metric":"Mean team score / 25 with random teammate","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":null,"comparability_note":null,"references":[],"cases":100,"requests":null,"fields":null,"source_cases":null,"unit":"episode","scoring_metric":"team score","subset_rule":"100 configured episodes; Fixed seeds 0..N-1; no performance-based replacement. Codenames uses each of66 eligible boards exactly once.","source_filtering":null,"note":"Dynamic request count depends on trajectory length. Configured count remains user-selectable. implemented; requires benchmark venv","suite_category":"Games and strategic decisions","subcategory":"Cooperative hidden-information cards","origin":"Jev-adapted benchmarks","adaptation":{"class":"B_mechanical","class_description":"Extra deterministic processing changes the evaluated view: filtering, extraction, search, enumeration, pairing, simulation or numeric encoding. No per-item semantic option authoring.","reason":"Player-visible observation and fixed partner; native team return.","candidate_origin":"Native OpenSpiel legal actions"},"random_baseline":{"value":null,"label":null,"kind":null,"method":null,"explanation":null,"monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"metric":null,"episodes":null,"interval_95":null,"interval_method":null,"http_median_ms":null,"http_p95_ms":null,"score":null},"sources":[{"name":"google-deepmind/open_spiel","url":"https://github.com/google-deepmind/open_spiel.git","revision":"48401890ee9857e611678302371378175a8e4c6b"}],"licences":[{"licence":"Apache-2.0","source":"archived source file","file":"LICENSE"}],"source_revision":"48401890ee9857e611678302371378175a8e4c6b","protocol":{"observation":"Native public observations plus explicit game rules and last12 public observations; hidden cards/roles omitted.","actions":"Full native option strings; one Jev decision followed by environment transition. One available action is automatic and logged.","sample":"Fixed seeds 0..N-1; no performance-based replacement. Codenames uses each of66 eligible boards exactly once.","partner":"Same seeded random teammate/opponent as pilot for Hanabi/Leduc; fixed lexical partner for Codenames.","scoring":"Native terminal team points for Hanabi; player0 net chips per hand for Leduc; native wins for RTFM; adapter wins for Codenames.","budget":{"game":"hanabi","players":2,"opponent":"seeded random","rules":"standard OpenSpiel Hanabi; player 0 information state only","max_actions":255,"max_turns":100},"limitations":"Fixed-partner performance, not optimal partnership or exploitability. RTFM is a100-turn partial-observation track; Codenames is a lexical-partner adaptation. Intervals describe this protocol and sample, not all possible partners/boards."}},{"id":18,"dataset":"Leduc poker","short":null,"kind":"interactive","area":"games","area_label":"Games & Strategy","area_color":"#ad6f48","frozen_panel_area":null,"in_panel":false,"in_frozen_panel":false,"dropped_interactive":false,"in_reproduction_suite":false,"metric":"Mean net chips per hand versus random opponent","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":null,"comparability_note":null,"references":[],"cases":300,"requests":null,"fields":null,"source_cases":null,"unit":"episode","scoring_metric":"player-0 mean return","subset_rule":"300 configured episodes; Fixed seeds 0..N-1; no performance-based replacement. Codenames uses each of66 eligible boards exactly once.","source_filtering":null,"note":"Dynamic request count depends on trajectory length. Configured count remains user-selectable. implemented; requires benchmark venv","suite_category":"Games and strategic decisions","subcategory":"Imperfect-information poker","origin":"Jev-adapted benchmarks","adaptation":{"class":"B_mechanical","class_description":"Extra deterministic processing changes the evaluated view: filtering, extraction, search, enumeration, pairing, simulation or numeric encoding. No per-item semantic option authoring.","reason":"Player information state and fixed opponent; native game return.","candidate_origin":"Native OpenSpiel legal actions"},"random_baseline":{"value":null,"label":null,"kind":null,"method":null,"explanation":null,"monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"metric":null,"episodes":null,"interval_95":null,"interval_method":null,"http_median_ms":null,"http_p95_ms":null,"score":null},"sources":[{"name":"google-deepmind/open_spiel","url":"https://github.com/google-deepmind/open_spiel.git","revision":"48401890ee9857e611678302371378175a8e4c6b"}],"licences":[{"licence":"Apache-2.0","source":"archived source file","file":"LICENSE"}],"source_revision":"48401890ee9857e611678302371378175a8e4c6b","protocol":{"observation":"Native public observations plus explicit game rules and last12 public observations; hidden cards/roles omitted.","actions":"Full native option strings; one Jev decision followed by environment transition. One available action is automatic and logged.","sample":"Fixed seeds 0..N-1; no performance-based replacement. Codenames uses each of66 eligible boards exactly once.","partner":"Same seeded random teammate/opponent as pilot for Hanabi/Leduc; fixed lexical partner for Codenames.","scoring":"Native terminal team points for Hanabi; player0 net chips per hand for Leduc; native wins for RTFM; adapter wins for Codenames.","budget":{"game":"leduc_poker","opponent":"seeded random","rules":"standard OpenSpiel Leduc; chance outcomes seeded independently in metadata","max_actions":255,"max_turns":100},"limitations":"Fixed-partner performance, not optimal partnership or exploitability. RTFM is a100-turn partial-observation track; Codenames is a lexical-partner adaptation. Intervals describe this protocol and sample, not all possible partners/boards."}},{"id":19,"dataset":"Codenames","short":"Codenames","kind":"interactive","area":"games","area_label":"Games & Strategy","area_color":"#ad6f48","frozen_panel_area":"games","in_panel":false,"in_frozen_panel":true,"dropped_interactive":true,"in_reproduction_suite":false,"metric":"Win rate with fixed lexical clue partner","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":null,"comparability_note":null,"references":[],"cases":66,"requests":null,"fields":null,"source_cases":130,"unit":"episode","scoring_metric":"lexical-partner win rate; turns; safe guesses","subset_rule":"All 66 eligible source boards, one episode each, with a fixed WordNet lexical clue partner.","source_filtering":"130 source boards → 66 eligible with full own-word safe-clue coverage under WordNet 3.0; 64 excluded.","note":"66 unique eligible source boards; one episode per board in the lexical-partner benchmark.","suite_category":"Games and strategic decisions","subcategory":"Word association","origin":"Jev-adapted benchmarks","adaptation":{"class":"B_mechanical","class_description":"Extra deterministic processing changes the evaluated view: filtering, extraction, search, enumeration, pairing, simulation or numeric encoding. No per-item semantic option authoring.","reason":"WordNet rule-based clue partner and eligibility filtering; no model-authored clues or invented answer words. Named adaptation.","candidate_origin":"Original board words plus end-turn"},"random_baseline":{"value":null,"label":null,"kind":null,"method":null,"explanation":null,"monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"metric":null,"episodes":null,"interval_95":null,"interval_method":null,"http_median_ms":null,"http_p95_ms":null,"score":null},"sources":[{"name":"clembench/clembench","url":"https://github.com/clembench/clembench.git","revision":"adff278de1cf1cbc30a93f3000347b102cdfc0c5"},{"name":"aclanthology.org/2025.gem-1.63.pdf","url":"https://aclanthology.org/2025.gem-1.63.pdf","revision":null}],"licences":[{"licence":"MIT","source":"archived source file","file":"LICENSE"}],"source_revision":"adff278de1cf1cbc30a93f3000347b102cdfc0c5","protocol":{"observation":"Native public observations plus explicit game rules and last12 public observations; hidden cards/roles omitted.","actions":"Full native option strings; one Jev decision followed by environment transition. One available action is automatic and logged.","sample":"Fixed seeds 0..N-1; no performance-based replacement. Codenames uses each of66 eligible boards exactly once.","partner":"Same seeded random teammate/opponent as pilot for Hanabi/Leduc; fixed lexical partner for Codenames.","scoring":"Native terminal team points for Hanabi; player0 net chips per hand for Leduc; native wins for RTFM; adapter wins for Codenames.","budget":{"max_actions":100,"max_turns":100},"limitations":"Fixed-partner performance, not optimal partnership or exploitability. RTFM is a100-turn partial-observation track; Codenames is a lexical-partner adaptation. Intervals describe this protocol and sample, not all possible partners/boards."}},{"id":20,"dataset":"BPoMP","short":null,"kind":"static","area":"arts","area_label":"Arts & Human Taste","area_color":"#b6698a","frozen_panel_area":"arts","in_panel":true,"in_frozen_panel":true,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"accuracy","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":"Given two versions of a limerick, one with its rhyme, meter or sense broken, picks the original.","comparability_note":null,"references":[{"kind":"dataset","url":"https://zenodo.org/records/7299879"}],"cases":811,"requests":5000,"fields":5000,"source_cases":22419,"unit":"underlying poem (paired requests)","scoring_metric":"original-limerick discrimination; macro by perturbation","subset_rule":"5,000 requests within a 5,000-request cap: 811 complete linked case groups of 22,419 prepared; groups kept intact.","source_filtering":"134,005 original/perturbed limerick pairs across 14 perturbation variants; 9 indistinguishable pairs excluded; 5,000 requests sampled as complete poem groups.","note":"Original-versus-perturbed limerick discrimination across14 source variants. Exact ties excluded, labels shuffled. Macro by perturbation and poem-clustered uncertainty; not human aesthetic preference. Release v1: 5000 requests within 5000 cap; complete linked cases retained.","suite_category":"Arts and human preferences","subcategory":"Poetry perturbations","origin":"Jev-adapted benchmarks","adaptation":{"class":"B_mechanical","class_description":"Extra deterministic processing changes the evaluated view: filtering, extraction, search, enumeration, pairing, simulation or numeric encoding. No per-item semantic option authoring.","reason":"Filters indistinguishable pairs and randomizes order; no new poem options or aesthetic gold.","candidate_origin":"Released original/perturbed poem pairs"},"random_baseline":{"value":0.5,"label":"Uniform random choice","kind":"random_accuracy","method":"Exact expectation","explanation":"Average accepted-answer count / option count across the actual sampled fields. Includes all tied correct moves/statements where the scorer accepts ties; variable option counts are averaged case by case.","monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.9006},"sources":[{"name":"zenodo.org/records/7299879","url":"https://zenodo.org/records/7299879","revision":null}],"licences":[],"source_revision":null},{"id":21,"dataset":"Humicroedit","short":null,"kind":"static","area":"arts","area_label":"Arts & Human Taste","area_color":"#b6698a","frozen_panel_area":"arts","in_panel":true,"in_frozen_panel":true,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"accuracy","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":"Given two one-word edits of a news headline, picks the one people rated funnier.","comparability_note":null,"references":[{"kind":"paper","url":"https://arxiv.org/abs/1906.00274"},{"kind":"dataset","url":"https://cs.rochester.edu/u/nhossain/humicroedit.html"}],"cases":2628,"requests":2628,"fields":2628,"source_cases":2628,"unit":"headline-edit pair","scoring_metric":"human-preferred edit accuracy excluding source ties","subset_rule":"All 2,628 prepared cases retained.","source_filtering":"2,960 official subtask-2 test pairs → 2,628; 332 equal-rating ties excluded (IDs recorded).","note":"Official full subtask2 test source has2960 pairs; scorer excludes332 equal-rating ties, leaving2628 scored requests. Tie IDs recorded. Release v1: All prepared cases retained.","suite_category":"Arts and human preferences","subcategory":"Humor preference","origin":"Jev-adapted benchmarks","adaptation":{"class":"B_mechanical","class_description":"Extra deterministic processing changes the evaluated view: filtering, extraction, search, enumeration, pairing, simulation or numeric encoding. No per-item semantic option authoring.","reason":"Uses human-rated source pairs; equal-rating ties excluded from forced-choice score.","candidate_origin":"Released paired headline edits"},"random_baseline":{"value":0.5,"label":"Uniform random choice","kind":"random_accuracy","method":"Exact expectation","explanation":"Average accepted-answer count / option count across the actual sampled fields. Includes all tied correct moves/statements where the scorer accepts ties; variable option counts are averaged case by case.","monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.6176},"sources":[{"name":"n-hossain/semeval-2020-task-7-humicroedit","url":"https://github.com/n-hossain/semeval-2020-task-7-humicroedit.git","revision":"7c0921e0aaf9fc671b17b68013a9ceb9edd42944"}],"licences":[],"source_revision":"7c0921e0aaf9fc671b17b68013a9ceb9edd42944"},{"id":22,"dataset":"POP909-CL","short":"POP909","kind":"static","area":"arts","area_label":"Arts & Human Taste","area_color":"#b6698a","frozen_panel_area":"arts","in_panel":true,"in_frozen_panel":true,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"accuracy","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":"Given the notes around a beat in a pop song (not audio), picks the chord playing from 129.","comparability_note":null,"references":[{"kind":"paper","url":"https://arxiv.org/abs/2510.06528"},{"kind":"dataset","url":"https://github.com/AndyWeasley2004/POP909-CL-Dataset"}],"cases":2000,"requests":2000,"fields":2000,"source_cases":75582,"unit":"target chord window","scoring_metric":"chord pitch-class recognition; song macro","subset_rule":"2,000 of 75,582 prepared cases, deterministic hash sample without replacement (seed 20260919); no outcome-based selection.","source_filtering":"907 of 909 songs eligible (excluded: 518: algorithmic labels per source README; 620: known score/annotation misalignment); 75,582 chord windows, 129 fixed pitch-class labels; 2,000 sampled.","note":"907 source songs, symbolic chord recognition at fixed beat positions. Score channel0 visible; expert chord channel1 hidden.129 fixed pitch-class labels; song-clustered metrics, NoChord/Other reported separately. Release v1: 2000 seeded cases.","suite_category":"Arts and human preferences","subcategory":"Symbolic music harmony","origin":"Jev-adapted benchmarks","adaptation":{"class":"C_semantic_options","class_description":"The adaptation relies on authored semantic option spaces/scenarios or model-generated candidate answers absent from the original native answer format.","reason":"Musical judgment defines129 chord pitch-class categories and windows; gold comes from released expert annotation, not new model judgments.","candidate_origin":"Root-designed fixed chord vocabulary"},"random_baseline":{"value":0.0078,"label":"Uniform random choice","kind":"random_accuracy","method":"Exact expectation","explanation":"Average accepted-answer count / option count across the actual sampled fields. Includes all tied correct moves/statements where the scorer accepts ties; variable option counts are averaged case by case.","monte_carlo_standard_error":null,"secondary_reference":{"label":"Always predict GM (sample majority)","value":0.051,"uses_evaluation_labels":true,"explanation":"Descriptive class-imbalance reference selected using this evaluation sample; not a independently chosen predictive baseline."}},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.1645},"sources":[{"name":"AndyWeasley2004/POP909-CL-Dataset","url":"https://github.com/AndyWeasley2004/POP909-CL-Dataset.git","revision":"be9094392903c471a930519e1c0bacf8b6be5d62"}],"licences":[{"licence":"MIT","source":"archived source file","file":"LICENSE"}],"source_revision":"be9094392903c471a930519e1c0bacf8b6be5d62"},{"id":23,"dataset":"cfcolor","short":null,"kind":"static","area":"arts","area_label":"Arts & Human Taste","area_color":"#b6698a","frozen_panel_area":"arts","in_panel":true,"in_frozen_panel":true,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"accuracy","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":"Given a person's earlier ratings of color palettes, predicts which of two new palettes they rated higher.","comparability_note":null,"references":[{"kind":"dataset","url":"https://www.dgp.toronto.edu/~donovan/cfcolor/"}],"cases":5000,"requests":5000,"fields":5000,"source_cases":11821,"unit":"held-out palette pair","scoring_metric":"human preference agreement; user macro","subset_rule":"5,000 of 11,821 prepared cases, deterministic hash sample without replacement (seed 20260919); no outcome-based selection.","source_filtering":"11,821 held-out pairs from 869 users with rating gap ≥ 2 (gap 2: 8,733, 3: 2,546, 4: 542); 5,000 sampled.","note":"Held-out human palette-preference pairs with original train-rating history. Gap>=2, nonoverlapping pairs, visible RGB duplicates excluded; user-macro accuracy. Human preference, not objective beauty. Release v1: 5000 seeded cases.","suite_category":"Arts and human preferences","subcategory":"Personalized color preference","origin":"Jev-adapted benchmarks","adaptation":{"class":"B_mechanical","class_description":"Extra deterministic processing changes the evaluated view: filtering, extraction, search, enumeration, pairing, simulation or numeric encoding. No per-item semantic option authoring.","reason":"Deterministic held-out pairing and train-history selection; real human rating gap defines gold.","candidate_origin":"Released rated palettes"},"random_baseline":{"value":0.5,"label":"Uniform random choice","kind":"random_accuracy","method":"Exact expectation","explanation":"Average accepted-answer count / option count across the actual sampled fields. Includes all tied correct moves/statements where the scorer accepts ties; variable option counts are averaged case by case.","monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.6634},"sources":[{"name":"www.dgp.toronto.edu/~donovan/cfcolor/","url":"https://www.dgp.toronto.edu/~donovan/cfcolor/","revision":null}],"licences":[],"source_revision":null},{"id":24,"dataset":"MMLU","short":null,"kind":"static","area":"knowledge","area_label":"Knowledge & Reasoning","area_color":"#5976bd","frozen_panel_area":"knowledge","in_panel":false,"in_frozen_panel":true,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"accuracy","lower_is_better":false,"index_rule":null,"left_out_of_index":true,"explainer":"Four-option questions across 57 school and professional subjects, from law to algebra.","comparability_note":null,"references":[{"kind":"dataset","url":"https://huggingface.co/datasets/cais/mmlu"}],"cases":14042,"requests":14042,"fields":14042,"source_cases":14042,"unit":"case","scoring_metric":"accuracy by subject; micro/macro","subset_rule":"All 14,042 prepared cases retained.","source_filtering":null,"note":"Release v1: All prepared cases retained.","suite_category":"Knowledge and science","subcategory":"Multidomain knowledge","origin":"Original-task benchmarks","adaptation":{"class":"A_direct","class_description":"Original finite choices, class labels, or probabilities map directly; only parsing, serialization and label mapping are needed.","reason":"Native MMLU test questions.","candidate_origin":"Original multiple-choice options"},"random_baseline":{"value":0.25,"label":"Uniform random choice","kind":"random_accuracy","method":"Exact expectation","explanation":"Average accepted-answer count / option count across the actual sampled fields. Includes all tied correct moves/statements where the scorer accepts ties; variable option counts are averaged case by case.","monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.9167},"sources":[{"name":"cais/mmlu","url":"https://huggingface.co/datasets/cais/mmlu","revision":null}],"licences":[],"source_revision":null},{"id":25,"dataset":"GPQA Diamond","short":"GPQA Diamond","kind":"static","area":"knowledge","area_label":"Knowledge & Reasoning","area_color":"#5976bd","frozen_panel_area":"knowledge","in_panel":true,"in_frozen_panel":true,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"accuracy","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"gold":1.2,"explainer":"198 hard four-option science questions written by PhD experts in physics, chemistry and biology.","comparability_note":null,"references":[{"kind":"dataset","url":"https://github.com/idavidrein/gpqa"}],"cases":198,"requests":198,"fields":198,"source_cases":198,"unit":"case","scoring_metric":"accuracy","subset_rule":"All 198 prepared cases retained.","source_filtering":null,"note":"All198 Diamond questions converted locally. Encrypted original retained; do not upload benchmark content publicly. Release v1: All prepared cases retained.","suite_category":"Knowledge and science","subcategory":"Graduate science","origin":"Original-task benchmarks","adaptation":{"class":"A_direct","class_description":"Original finite choices, class labels, or probabilities map directly; only parsing, serialization and label mapping are needed.","reason":"Official Diamond subset with native options.","candidate_origin":"Original multiple-choice options"},"random_baseline":{"value":0.25,"label":"Uniform random choice","kind":"random_accuracy","method":"Exact expectation","explanation":"Average accepted-answer count / option count across the actual sampled fields. Includes all tied correct moves/statements where the scorer accepts ties; variable option counts are averaged case by case.","monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.7551},"sources":[{"name":"Idavidrein/gpqa","url":"https://huggingface.co/datasets/Idavidrein/gpqa","revision":null}],"licences":[],"source_revision":null},{"id":26,"dataset":"ARC-Easy","short":null,"kind":"static","area":"knowledge","area_label":"Knowledge & Reasoning","area_color":"#5976bd","frozen_panel_area":null,"in_panel":false,"in_frozen_panel":false,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"accuracy","lower_is_better":false,"index_rule":null,"left_out_of_index":true,"explainer":"Grade-school science multiple choice, the easier set.","comparability_note":null,"references":[{"kind":"dataset","url":"https://huggingface.co/datasets/allenai/ai2_arc"}],"cases":2376,"requests":2376,"fields":2376,"source_cases":2376,"unit":"case","scoring_metric":"accuracy","subset_rule":"All 2,376 prepared cases retained.","source_filtering":null,"note":"Release v1: All prepared cases retained.","suite_category":"Knowledge and science","subcategory":"Elementary science","origin":"Original-task benchmarks","adaptation":{"class":"A_direct","class_description":"Original finite choices, class labels, or probabilities map directly; only parsing, serialization and label mapping are needed.","reason":"Native ARC-Easy choices.","candidate_origin":"Original multiple-choice options"},"random_baseline":{"value":0.2502,"label":"Uniform random choice","kind":"random_accuracy","method":"Exact expectation","explanation":"Average accepted-answer count / option count across the actual sampled fields. Includes all tied correct moves/statements where the scorer accepts ties; variable option counts are averaged case by case.","monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.9933},"sources":[{"name":"allenai/ai2_arc","url":"https://huggingface.co/datasets/allenai/ai2_arc","revision":null}],"licences":[],"source_revision":null},{"id":27,"dataset":"ARC-Challenge","short":null,"kind":"static","area":"knowledge","area_label":"Knowledge & Reasoning","area_color":"#5976bd","frozen_panel_area":null,"in_panel":false,"in_frozen_panel":false,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"accuracy","lower_is_better":false,"index_rule":null,"left_out_of_index":true,"explainer":"Grade-school science multiple choice, the harder set that simple retrieval gets wrong.","comparability_note":null,"references":[{"kind":"dataset","url":"https://huggingface.co/datasets/allenai/ai2_arc"}],"cases":1172,"requests":1172,"fields":1172,"source_cases":1172,"unit":"case","scoring_metric":"accuracy","subset_rule":"All 1,172 prepared cases retained.","source_filtering":null,"note":"Release v1: All prepared cases retained.","suite_category":"Knowledge and science","subcategory":"Challenging science","origin":"Original-task benchmarks","adaptation":{"class":"A_direct","class_description":"Original finite choices, class labels, or probabilities map directly; only parsing, serialization and label mapping are needed.","reason":"Native ARC-Challenge choices.","candidate_origin":"Original multiple-choice options"},"random_baseline":{"value":0.2502,"label":"Uniform random choice","kind":"random_accuracy","method":"Exact expectation","explanation":"Average accepted-answer count / option count across the actual sampled fields. Includes all tied correct moves/statements where the scorer accepts ties; variable option counts are averaged case by case.","monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.9761},"sources":[{"name":"allenai/ai2_arc","url":"https://huggingface.co/datasets/allenai/ai2_arc","revision":null}],"licences":[],"source_revision":null},{"id":28,"dataset":"WinoGrande","short":null,"kind":"static","area":"knowledge","area_label":"Knowledge & Reasoning","area_color":"#407e9e","frozen_panel_area":null,"in_panel":true,"in_frozen_panel":false,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"accuracy","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"gold":1.2,"explainer":"Fills a blank with one of two options using common sense, e.g. who \"got the easier cases\".","comparability_note":null,"references":[{"kind":"dataset","url":"https://huggingface.co/datasets/allenai/winogrande"}],"cases":1267,"requests":1267,"fields":1267,"source_cases":1267,"unit":"case","scoring_metric":"accuracy","subset_rule":"All 1,267 prepared cases retained.","source_filtering":null,"note":"Release v1: All prepared cases retained.","suite_category":"Reasoning, mathematics and code","subcategory":"Commonsense coreference","origin":"Original-task benchmarks","adaptation":{"class":"A_direct","class_description":"Original finite choices, class labels, or probabilities map directly; only parsing, serialization and label mapping are needed.","reason":"Native WinoGrande alternatives.","candidate_origin":"Original2 completion options"},"random_baseline":{"value":0.5,"label":"Uniform random choice","kind":"random_accuracy","method":"Exact expectation","explanation":"Average accepted-answer count / option count across the actual sampled fields. Includes all tied correct moves/statements where the scorer accepts ties; variable option counts are averaged case by case.","monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.9108},"sources":[{"name":"allenai/winogrande","url":"https://huggingface.co/datasets/allenai/winogrande","revision":null}],"licences":[],"source_revision":null},{"id":29,"dataset":"HellaSwag","short":null,"kind":"static","area":"language","area_label":"Language Understanding","area_color":"#407e9e","frozen_panel_area":null,"in_panel":true,"in_frozen_panel":false,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"accuracy","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"gold":1.2,"explainer":"Picks the most sensible of four continuations of an everyday scene.","comparability_note":null,"references":[{"kind":"dataset","url":"https://huggingface.co/datasets/Rowan/hellaswag"}],"cases":10042,"requests":10042,"fields":10042,"source_cases":10042,"unit":"case","scoring_metric":"accuracy","subset_rule":"All 10,042 prepared cases retained.","source_filtering":null,"note":"Release v1: All prepared cases retained.","suite_category":"Reasoning, mathematics and code","subcategory":"Commonsense continuation","origin":"Original-task benchmarks","adaptation":{"class":"A_direct","class_description":"Original finite choices, class labels, or probabilities map directly; only parsing, serialization and label mapping are needed.","reason":"Native HellaSwag alternatives.","candidate_origin":"Original continuation options"},"random_baseline":{"value":0.25,"label":"Uniform random choice","kind":"random_accuracy","method":"Exact expectation","explanation":"Average accepted-answer count / option count across the actual sampled fields. Includes all tied correct moves/statements where the scorer accepts ties; variable option counts are averaged case by case.","monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.9434},"sources":[{"name":"Rowan/hellaswag","url":"https://huggingface.co/datasets/Rowan/hellaswag","revision":null}],"licences":[],"source_revision":null},{"id":30,"dataset":"GSM8K","short":null,"kind":"static","area":"knowledge","area_label":"Knowledge & Reasoning","area_color":"#5976bd","frozen_panel_area":"knowledge","in_panel":true,"in_frozen_panel":true,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"accuracy","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":"Picks a math word problem's answer from 4, and separately 10, numbers, with no working.","comparability_note":null,"references":[{"kind":"dataset","url":"https://huggingface.co/datasets/openai/gsm8k"}],"cases":1319,"requests":2638,"fields":2638,"source_cases":1319,"unit":"math problem (two MC variants)","scoring_metric":"MC accuracy separately for 4-choice and 10-choice variants","subset_rule":"All 1,319 prepared cases retained (2,638 requests).","source_filtering":"1,319 test problems, each rendered as a 4-choice and a 10-choice request (2,638 requests); synthetic numeric distractors.","note":"1319 source test problems in both 4- and 10-choice variants (2638 requests). Decimal-valued synthetic distractors, shared source grouping. Adapted MC accuracy only, not native free-response accuracy. Release v1: All prepared cases retained.","suite_category":"Reasoning, mathematics and code","subcategory":"Arithmetic word problems","origin":"Jev-adapted benchmarks","adaptation":{"class":"B_mechanical","class_description":"Extra deterministic processing changes the evaluated view: filtering, extraction, search, enumeration, pairing, simulation or numeric encoding. No per-item semantic option authoring.","reason":"Fixed numeric recipe creates4/10 choices from free-response answers. Requires no per-item reasoning, but is adapted MC accuracy, not native GSM8K.","candidate_origin":"Algorithmic numeric distractors"},"random_baseline":{"value":0.175,"label":"Uniform random choice","kind":"random_accuracy","method":"Exact expectation","explanation":"Average accepted-answer count / option count across the actual sampled fields. Includes all tied correct moves/statements where the scorer accepts ties; variable option counts are averaged case by case.","monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[{"track":"GSM8K-4choice","label":"Four answer choices","metric":"Multiple-choice accuracy","description":"Each original word problem is paired with its correct numeric answer and three synthetic distractors. Random guessing scores 25%.","random":0.25,"headline":false,"in_index":true},{"track":"GSM8K-10choice","label":"Ten answer choices","metric":"Multiple-choice accuracy","description":"The same source problems with nine synthetic distractors. Random guessing scores 10%. Neither track is native free-response GSM8K.","random":0.1,"headline":false,"in_index":true}],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.7301},"sources":[{"name":"openai/gsm8k","url":"https://huggingface.co/datasets/openai/gsm8k","revision":null}],"licences":[],"source_revision":null},{"id":31,"dataset":"ChessBench","short":null,"kind":"static","area":"knowledge","area_label":"Knowledge & Reasoning","area_color":"#5976bd","frozen_panel_area":"games","in_panel":true,"in_frozen_panel":true,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"accuracy","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":"Given a chess position and every legal move, picks the strongest one; tied best moves count.","comparability_note":null,"references":[{"kind":"paper","url":"https://arxiv.org/abs/2402.04494"},{"kind":"code","url":"https://github.com/google-deepmind/searchless_chess"}],"cases":5000,"requests":5000,"fields":5000,"source_cases":62561,"unit":"chess position","scoring_metric":"best-move accuracy accepting reference ties; mean value regret","subset_rule":"5,000 of 61,833 prepared cases, deterministic hash sample without replacement (seed 20260919); no outcome-based selection.","source_filtering":"62,561 test positions → 61,833 with complete legal-move coverage (728 excluded); 3,654 positions accept tied best moves; 5,000 sampled.","note":"Official DeepMind ChessBench action-value test data. All legal moves, exact ties accepted by custom scorer, reference value regret. Incomplete move coverage and non-decision positions explicitly excluded; no source sampling. Release v1: 5000 seeded cases.","suite_category":"Games and strategic decisions","subcategory":"Chess move quality","origin":"Jev-adapted benchmarks","adaptation":{"class":"B_mechanical","class_description":"Extra deterministic processing changes the evaluated view: filtering, extraction, search, enumeration, pairing, simulation or numeric encoding. No per-item semantic option authoring.","reason":"Group published move values by position, verify complete legal coverage, accept reference ties.","candidate_origin":"All legal moves from source positions"},"random_baseline":{"value":0.0819,"label":"Uniform random choice","kind":"random_accuracy","method":"Exact expectation","explanation":"Average accepted-answer count / option count across the actual sampled fields. Includes all tied correct moves/statements where the scorer accepts ties; variable option counts are averaged case by case.","monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.1688},"sources":[{"name":"google-deepmind/searchless_chess","url":"https://github.com/google-deepmind/searchless_chess.git","revision":"90ae0e6b121673fc3079aaeffa047580bb600c0a"}],"licences":[{"licence":"Apache-2.0","source":"archived source file","file":"LICENSE"}],"source_revision":"90ae0e6b121673fc3079aaeffa047580bb600c0a"},{"id":32,"dataset":"MuSR","short":null,"kind":"static","area":"knowledge","area_label":"Knowledge & Reasoning","area_color":"#5976bd","frozen_panel_area":null,"in_panel":true,"in_frozen_panel":false,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"accuracy","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":"Reads a long story and answers one question needing clues from across it, e.g. who the murderer is.","comparability_note":null,"references":[{"kind":"paper","url":"https://arxiv.org/abs/2310.16049"},{"kind":"dataset","url":"https://github.com/Zayne-sprague/MuSR"}],"cases":752,"requests":756,"fields":756,"source_cases":756,"unit":"case","scoring_metric":"accuracy by task","subset_rule":"All 756 prepared cases retained.","source_filtering":null,"note":"Release v1: All prepared cases retained.","suite_category":"Reasoning, mathematics and code","subcategory":"Multistep reasoning","origin":"Original-task benchmarks","adaptation":{"class":"A_direct","class_description":"Original finite choices, class labels, or probabilities map directly; only parsing, serialization and label mapping are needed.","reason":"Native MuSR alternatives.","candidate_origin":"Original multiple-choice options"},"random_baseline":{"value":0.371,"label":"Uniform random choice","kind":"random_accuracy","method":"Exact expectation","explanation":"Average accepted-answer count / option count across the actual sampled fields. Includes all tied correct moves/statements where the scorer accepts ties; variable option counts are averaged case by case.","monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.7261},"sources":[{"name":"Zayne-sprague/MuSR","url":"https://github.com/Zayne-sprague/MuSR.git","revision":"b1f4d4168a9cfc6760e8b74d728e4516023dfaa5"},{"name":"TAUR-Lab/MuSR","url":"https://huggingface.co/datasets/TAUR-Lab/MuSR","revision":null}],"licences":[{"licence":"MIT","source":"archived source file","file":"LICENSE"}],"source_revision":"b1f4d4168a9cfc6760e8b74d728e4516023dfaa5"},{"id":33,"dataset":"SATA-Bench","short":null,"kind":"static","area":"knowledge","area_label":"Knowledge & Reasoning","area_color":"#5976bd","frozen_panel_area":null,"in_panel":true,"in_frozen_panel":false,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"case exact accuracy","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":"Given a passage, a question and candidate answers, says yes or no to each; all must be right.","comparability_note":null,"references":[{"kind":"dataset","url":"https://github.com/sata-bench/sata-bench"}],"cases":1650,"requests":1650,"fields":15517,"source_cases":1650,"unit":"select-all source question","scoring_metric":"exact answer-set accuracy; option F1","subset_rule":"All 1,650 prepared cases retained.","source_filtering":"All 1,650 source questions projected to 15,517 binary option decisions; scored as exact answer sets.","note":"All1650 source questions projected to15517 binary fields. Primary scoring unit remains exact answer set per source question. Release v1: All prepared cases retained.","suite_category":"Reasoning, mathematics and code","subcategory":"Select-all knowledge/reasoning","origin":"Original-task benchmarks","adaptation":{"class":"A_direct","class_description":"Original finite choices, class labels, or probabilities map directly; only parsing, serialization and label mapping are needed.","reason":"One binary decision per supplied option preserves the original answer-set task; exact-set score.","candidate_origin":"Original select-all alternatives"},"random_baseline":{"value":0.0131,"label":"Random whole-case guesses","kind":"random_exact_case","method":"Exact expectation","explanation":"Mean probability that every field in a case is correct under independent uniform guesses; multiply field probabilities within each case, then average cases. This is not the per-field chance rate.","monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.2788},"sources":[{"name":"sata-bench/sata-bench","url":"https://github.com/sata-bench/sata-bench.git","revision":"371dd0c18fe75a96fbbcf2d1507ceeaf0d5263c5"}],"licences":[{"licence":"MIT","source":"archived source file","file":"LICENSE"}],"source_revision":"371dd0c18fe75a96fbbcf2d1507ceeaf0d5263c5"},{"id":36,"dataset":"BRIGHT","short":null,"kind":"static","area":"retrieval","area_label":"Retrieval & Classification","area_color":"#3e8e5e","frozen_panel_area":"language","in_panel":true,"in_frozen_panel":true,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"nDCG@10","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"gold":1.2,"explainer":"Rates how helpful each of 32 search-hit documents is for a hard question; ranked by nDCG@10, on questions where at least one of the 32 is relevant.","comparability_note":null,"references":[{"kind":"paper","url":"https://arxiv.org/abs/2407.12883"},{"kind":"dataset","url":"https://huggingface.co/datasets/xlangai/BRIGHT"}],"cases":220,"requests":307,"fields":44286,"source_cases":1384,"unit":"retrieval query","scoring_metric":"nDCG@k; candidate recall","subset_rule":"550 of 1,297 scoreable v1 queries kept (637 requests): stratified by source domain, proportional allocation with at least one query per domain (seed 20260922); kept queries keep their complete candidate lists; no outcome-based selection.","source_filtering":"Candidate policy: query-only lexical overlap top32; no gold insertion; unjudged candidates are not treated as verified negatives.","note":"Official corpus, queries, judgments and ancillary release files downloaded. Unjudged documents must not be treated as verified negatives. Release v1: All prepared cases retained.","suite_category":"Document reasoning and retrieval","subcategory":"Reasoning-intensive retrieval","origin":"Jev-adapted benchmarks","adaptation":{"class":"B_mechanical","class_description":"Extra deterministic processing changes the evaluated view: filtering, extraction, search, enumeration, pairing, simulation or numeric encoding. No per-item semantic option authoring.","reason":"BM25 retrieval, source exclusion lists, full-document context checks and probability ranking.","candidate_origin":"Query-only search over published document corpus"},"random_baseline":{"value":0.116,"label":"Random ranking, answerable queries","kind":"random_ranking","method":"Exact expectation, answerable queries","explanation":"Mean over the scored queries of the expected nDCG@10 under a uniform random permutation of that query's 32 candidates, using the full relevance judgments for the denominator; only queries with at least one relevant candidate are scored. This is the value the index uses.","monte_carlo_standard_error":null,"secondary_reference":{"label":"BM25 ranking","value":0.1113}},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.4705},"sources":[{"name":"brightbenchmark.github.io","url":"https://brightbenchmark.github.io","revision":null}],"licences":[{"licence":"CC-BY-4.0","source":"archived dataset card","file":"README.md"}],"source_revision":null},{"id":37,"dataset":"Amazon ESCI","short":"Amazon ESCI","kind":"static","area":"retrieval","area_label":"Retrieval & Classification","area_color":"#3e8e5e","frozen_panel_area":"language","in_panel":true,"in_frozen_panel":true,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"macro-F1","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":"Labels a product as an exact match, substitute, complement or irrelevant to a shopping query.","comparability_note":null,"references":[{"kind":"dataset","url":"https://github.com/amazon-science/esci-data"}],"cases":5000,"requests":5000,"fields":5000,"source_cases":638016,"unit":"query-product pair","scoring_metric":"4-class accuracy; macro-F1","subset_rule":"5,000 of 638,016 source test pairs, proportional locale × relevance subset (seed 20260919).","source_filtering":"638,016 source test pairs (us 425,762, es 93,347, jp 118,907; labels I 70,440, E 393,363, S 153,357, C 20,856); 5,000 sampled proportionally by locale × relevance.","note":"Release v1: proportional locale × relevance subset of 5,000 source test pairs; replaces the former 2,000-pair active limit.","suite_category":"Document reasoning and retrieval","subcategory":"Product relevance","origin":"Original-task benchmarks","adaptation":{"class":"A_direct","class_description":"Original finite choices, class labels, or probabilities map directly; only parsing, serialization and label mapping are needed.","reason":"Native query-product classification. The separately recorded2,000-case sampling policy does not change this decision format.","candidate_origin":"Original4 relevance labels"},"random_baseline":{"value":0.2027,"label":"Random choices · F1","kind":"random_f1","method":"Monte Carlo","explanation":"Uniform independent choice among each field's actual options, scored with the same F1 definition and observed/predicted label union as the model. F1 is nonlinear and class-frequency dependent; 1/K is not its expected value. This simulation error is not uncertainty in model quality.","monte_carlo_standard_error":0.00017,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.5718},"sources":[{"name":"amazon-science/esci-data","url":"https://github.com/amazon-science/esci-data.git","revision":"7916cdf6ab75a462e77f20ab40428a10923998d5"}],"licences":[{"licence":"Apache-2.0","source":"archived source file","file":"LICENSE"}],"source_revision":"7916cdf6ab75a462e77f20ab40428a10923998d5"},{"id":38,"dataset":"ACOS","short":null,"kind":"static","area":"language","area_label":"Language Understanding","area_color":"#407e9e","frozen_panel_area":null,"in_panel":true,"in_frozen_panel":false,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"per-review F1","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":"Reads a restaurant or laptop review and says yes/no for each aspect-and-sentiment pair; scored by F1 per review, averaged over reviews.","comparability_note":null,"references":[{"kind":"paper","url":"https://aclanthology.org/2021.acl-long.29"},{"kind":"dataset","url":"https://github.com/NUSTM/ACOS"}],"cases":400,"requests":1565,"fields":318945,"source_cases":1399,"unit":"source review","scoring_metric":"category/sentiment exact-set accuracy; pair micro-F1","subset_rule":"All 1,399 prepared cases retained (5,479 requests).","source_filtering":null,"note":"Official test reviews projected to fixed category-by-sentiment binary questions; chunks group back to source review. This is not full tuple extraction. Release v1: All prepared cases retained.","suite_category":"Language understanding and classification","subcategory":"Aspect/category sentiment","origin":"Jev-adapted benchmarks","adaptation":{"class":"B_mechanical","class_description":"Extra deterministic processing changes the evaluated view: filtering, extraction, search, enumeration, pairing, simulation or numeric encoding. No per-item semantic option authoring.","reason":"Projects extraction tuples to fixed category-sentiment presence decisions; not full span/tuple extraction.","candidate_origin":"Source categories crossed with3 source sentiment labels"},"random_baseline":{"value":0.031,"label":"Always yes · per-review F1","kind":"random_exact_case","method":"Exact, always yes","explanation":"Per-review F1 of answering yes to every aspect-and-sentiment pair, averaged over reviews. A fair coin on every pair scores slightly lower, so always-yes is the chance level.","monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.3314},"sources":[{"name":"NUSTM/ACOS","url":"https://github.com/NUSTM/ACOS.git","revision":"45d179a3dcc6a3dedd848d81b16f2552454805fe"}],"licences":[],"source_revision":"45d179a3dcc6a3dedd848d81b16f2552454805fe"},{"id":39,"dataset":"FinEntity","short":"FinEntity","kind":"static","area":"language","area_label":"Language Understanding","area_color":"#407e9e","frozen_panel_area":null,"in_panel":true,"in_frozen_panel":false,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"macro-F1","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":"Says whether a financial news snippet is positive, negative or neutral about a named firm.","comparability_note":null,"references":[{"kind":"paper","url":"https://aclanthology.org/2023.emnlp-main.956"},{"kind":"dataset","url":"https://github.com/yixuantt/FinEntity"}],"cases":979,"requests":979,"fields":2129,"source_cases":979,"unit":"source document","scoring_metric":"entity-given sentiment macro-F1; accuracy","subset_rule":"All 979 prepared cases retained.","source_filtering":null,"note":"979 original documents projected to entity-given sentiment; one conflicting exact span quarantined. Evaluation-only unsplit source, not an invented official test split. Release v1: All prepared cases retained.","suite_category":"Language understanding and classification","subcategory":"Financial entity sentiment","origin":"Jev-adapted benchmarks","adaptation":{"class":"B_mechanical","class_description":"Extra deterministic processing changes the evaluated view: filtering, extraction, search, enumeration, pairing, simulation or numeric encoding. No per-item semantic option authoring.","reason":"Entity-given projection of extraction; conflicting source annotation quarantined.","candidate_origin":"Original sentiment labels, provided source entity spans"},"random_baseline":{"value":0.3201,"label":"Random choices · F1","kind":"random_f1","method":"Monte Carlo","explanation":"Uniform independent choice among each field's actual options, scored with the same F1 definition and observed/predicted label union as the model. F1 is nonlinear and class-frequency dependent; 1/K is not its expected value. This simulation error is not uncertainty in model quality.","monte_carlo_standard_error":0.000314,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.8768},"sources":[{"name":"yixuantt/FinEntity","url":"https://github.com/yixuantt/FinEntity.git","revision":"3b6cedc5485b669c2ed168f1d949f517636eb7b8"}],"licences":[],"source_revision":"3b6cedc5485b669c2ed168f1d949f517636eb7b8"},{"id":40,"dataset":"iSarcasmEval","short":"iSarcasmEval","kind":"static","area":"language","area_label":"Language Understanding","area_color":"#407e9e","frozen_panel_area":"language","in_panel":true,"in_frozen_panel":true,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"Sarcasm F1 · track A, English","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":"Says yes or no: is this tweet sarcastic? Scored with F1 on sarcastic English tweets (track A); other tracks listed under each result.","comparability_note":null,"references":[{"kind":"paper","url":"https://aclanthology.org/2022.semeval-1.111"},{"kind":"dataset","url":"https://github.com/iabufarha/iSarcasmEval"}],"cases":4600,"requests":4600,"fields":11600,"source_cases":4600,"unit":"case","scoring_metric":"A sarcasm F1; B category macro-F1; C pairwise accuracy (report separately)","subset_rule":"All 4,600 prepared cases retained.","source_filtering":"All five official test tasks: iSarcasmEval-A-Ar 1,400, iSarcasmEval-A-En 1,400, iSarcasmEval-B-En 1,400, iSarcasmEval-C-Ar 200, iSarcasmEval-C-En 200. Track B is excluded from the index; track A English is the headline.","note":"All five official test tasks converted; primary metrics differ: sarcasm F1, label macro-F1, pairwise accuracy. Release v1: All prepared cases retained.","suite_category":"Language understanding and classification","subcategory":"Sarcasm and irony","origin":"Original-task benchmarks","adaptation":{"class":"A_direct","class_description":"Original finite choices, class labels, or probabilities map directly; only parsing, serialization and label mapping are needed.","reason":"Native SemEval task labels and supplied paired utterances, scored separately by subtask.","candidate_origin":"Original binary/multilabel/pair alternatives"},"random_baseline":{"value":0.2227,"label":"Random choices · F1","kind":"random_f1","method":"Monte Carlo","explanation":"Uniform independent choice among each field's actual options, scored with the same F1 definition and observed/predicted label union as the model. F1 is nonlinear and class-frequency dependent; 1/K is not its expected value. This simulation error is not uncertainty in model quality.","monte_carlo_standard_error":0.00047,"secondary_reference":null},"tracks":[{"track":"iSarcasmEval-A-En","label":"A · English","metric":"Sarcasm F1","description":"Decide whether a single English text is sarcastic. The headline uses F1 for the sarcastic class, balancing missed sarcasm against false alarms; it is not overall accuracy.","random":0.2227,"headline":true,"in_index":true},{"track":"iSarcasmEval-A-Ar","label":"A · Arabic","metric":"Sarcasm F1","description":"The same single-text sarcasm judgment in Arabic, scored with F1 for the sarcastic class.","random":0.2225,"headline":false,"in_index":true},{"track":"iSarcasmEval-B-En","label":"B · English figurative","metric":"Six-label macro-F1","description":"Identify sarcasm, irony, satire, understatement, overstatement, and rhetorical questions. Multiple labels may apply; the metric averages positive-class F1 across the six labels.","random":0.0547,"headline":false,"in_index":false},{"track":"iSarcasmEval-C-En","label":"C · English pairs","metric":"Pairwise accuracy","description":"Given a sarcastic text and its non-sarcastic rephrasing, choose the sarcastic one. This provides a contrast that task A does not; random guessing scores 50%.","random":0.5,"headline":false,"in_index":true},{"track":"iSarcasmEval-C-Ar","label":"C · Arabic pairs","metric":"Pairwise accuracy","description":"Choose the sarcastic text from an Arabic sarcastic/literal pair. Random guessing scores 50%.","random":0.5,"headline":false,"in_index":true}],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.5327},"sources":[{"name":"iabufarha/iSarcasmEval","url":"https://github.com/iabufarha/iSarcasmEval.git","revision":"dfc708b53bde1bb571abfb5692f63231c2232195"}],"licences":[{"licence":"MIT","source":"archived source file","file":"LICENSE"}],"source_revision":"dfc708b53bde1bb571abfb5692f63231c2232195"},{"id":41,"dataset":"VAST","short":null,"kind":"static","area":"language","area_label":"Language Understanding","area_color":"#407e9e","frozen_panel_area":"language","in_panel":true,"in_frozen_panel":true,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"macro-F1","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":"Says whether a comment is for, against, or neutral on a given topic.","comparability_note":null,"references":[{"kind":"dataset","url":"https://github.com/emilyallaway/zero-shot-stance"}],"cases":3006,"requests":3006,"fields":3006,"source_cases":3006,"unit":"case","scoring_metric":"stance macro-F1","subset_rule":"All 3,006 prepared cases retained.","source_filtering":null,"note":"Official VAST test split, all three labels, topic included. Release v1: All prepared cases retained.","suite_category":"Language understanding and classification","subcategory":"Topic-conditioned stance","origin":"Original-task benchmarks","adaptation":{"class":"A_direct","class_description":"Original finite choices, class labels, or probabilities map directly; only parsing, serialization and label mapping are needed.","reason":"Native stance classification with the source topic visible.","candidate_origin":"Original3 stance labels"},"random_baseline":{"value":0.3333,"label":"Random choices · F1","kind":"random_f1","method":"Monte Carlo","explanation":"Uniform independent choice among each field's actual options, scored with the same F1 definition and observed/predicted label union as the model. F1 is nonlinear and class-frequency dependent; 1/K is not its expected value. This simulation error is not uncertainty in model quality.","monte_carlo_standard_error":0.000273,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.7365},"sources":[{"name":"emilyallaway/zero-shot-stance","url":"https://github.com/emilyallaway/zero-shot-stance.git","revision":"e7c4775182b184730f350995f8260579c9e066fe"}],"licences":[],"source_revision":"e7c4775182b184730f350995f8260579c9e066fe"},{"id":42,"dataset":"NLI4CT","short":"NLI4CT","kind":"static","area":"language","area_label":"Language Understanding","area_color":"#407e9e","frozen_panel_area":null,"in_panel":true,"in_frozen_panel":false,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"macro-F1","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":"Says whether clinical-trial report sections support or contradict a statement, some deliberately altered.","comparability_note":null,"references":[{"kind":"dataset","url":"https://github.com/ai-systems/Task-2-SemEval-2024"}],"cases":5500,"requests":5500,"fields":5500,"source_cases":5500,"unit":"case","scoring_metric":"NLI macro-F1; accuracy","subset_rule":"All 5,500 prepared cases retained.","source_filtering":"5,500 source cases, 5,500 decisions.","note":"All5500 official2024 test statements joined by ID to gold and complete specified trial sections; no gold evidence-span selection. Release v1: All prepared cases retained.","suite_category":"Document reasoning and retrieval","subcategory":"Clinical-trial entailment","origin":"Original-task benchmarks","adaptation":{"class":"A_direct","class_description":"Original finite choices, class labels, or probabilities map directly; only parsing, serialization and label mapping are needed.","reason":"Source-designated trial sections and native binary judgment; gold evidence spans hidden.","candidate_origin":"Original entailment labels"},"random_baseline":{"value":0.486,"label":"Random choices · F1","kind":"random_f1","method":"Monte Carlo","explanation":"Uniform independent choice among each field's actual options, scored with the same F1 definition and observed/predicted label union as the model. F1 is nonlinear and class-frequency dependent; 1/K is not its expected value. This simulation error is not uncertainty in model quality.","monte_carlo_standard_error":0.000209,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.8512},"sources":[{"name":"ai-systems/Task-2-SemEval-2024","url":"https://github.com/ai-systems/Task-2-SemEval-2024.git","revision":"7f32fa6c7db43e577f22e9fc6c28eef1cc6d223e"}],"licences":[],"source_revision":"7f32fa6c7db43e577f22e9fc6c28eef1cc6d223e"},{"id":43,"dataset":"CRUXEval","short":null,"kind":"static","area":"knowledge","area_label":"Knowledge & Reasoning","area_color":"#5976bd","frozen_panel_area":"knowledge","in_panel":true,"in_frozen_panel":true,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"accuracy","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":"Reads a short Python function and an input and picks its output from 2 to 10 candidates, without running it.","comparability_note":null,"references":[{"kind":"paper","url":"https://arxiv.org/abs/2401.03065"},{"kind":"dataset","url":"https://github.com/facebookresearch/cruxeval"}],"cases":570,"requests":570,"fields":570,"source_cases":800,"unit":"code execution case","scoring_metric":"frozen output-choice accuracy","subset_rule":"All 570 prepared cases retained.","source_filtering":"800 source cases → 570 output-choice cases; 230 excluded for too few distinct archived candidates.","note":"Output-selection projection from all800 source cases, using archived incorrect model generations as distractors. Insufficient distinct candidates explicitly excluded. Parsed literal gold and Python-equivalence checks; not native pass@1. Release v1: All prepared cases retained.","suite_category":"Reasoning, mathematics and code","subcategory":"Code execution prediction","origin":"Jev-adapted benchmarks","adaptation":{"class":"C_semantic_options","class_description":"The adaptation relies on authored semantic option spaces/scenarios or model-generated candidate answers absent from the original native answer format.","reason":"Options rely on prior model generations of candidate answers, then mechanical equivalence checks. No new generation in this adaptation run.","candidate_origin":"Archived model-generated wrong outputs plus source gold"},"random_baseline":{"value":0.3697,"label":"Uniform random choice","kind":"random_accuracy","method":"Exact expectation","explanation":"Average accepted-answer count / option count across the actual sampled fields. Includes all tied correct moves/statements where the scorer accepts ties; variable option counts are averaged case by case.","monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.7474},"sources":[{"name":"facebookresearch/cruxeval","url":"https://github.com/facebookresearch/cruxeval.git","revision":"190faf16d175b5847b0af05d937872b1fb395942"}],"licences":[{"licence":"MIT","source":"archived source file","file":"LICENSE"}],"source_revision":"190faf16d175b5847b0af05d937872b1fb395942"},{"id":44,"dataset":"CLadder","short":null,"kind":"static","area":"knowledge","area_label":"Knowledge & Reasoning","area_color":"#5976bd","frozen_panel_area":"knowledge","in_panel":true,"in_frozen_panel":true,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"accuracy","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":"Answers a yes/no question about cause and effect in a small made-up world with stated probabilities.","comparability_note":null,"references":[{"kind":"paper","url":"https://arxiv.org/abs/2312.04350"},{"kind":"dataset","url":"https://github.com/causalNLP/cladder"}],"cases":5000,"requests":5000,"fields":5000,"source_cases":10112,"unit":"case","scoring_metric":"causal yes/no accuracy","subset_rule":"5,000 of 10,112 prepared cases, deterministic hash sample without replacement (seed 20260919); no outcome-based selection.","source_filtering":null,"note":"Full10112-question balanced CLadder release; background, given information and question only. Not a newly held-out sample. Release v1: 5000 seeded cases.","suite_category":"Reasoning, mathematics and code","subcategory":"Causal inference","origin":"Original-task benchmarks","adaptation":{"class":"A_direct","class_description":"Original finite choices, class labels, or probabilities map directly; only parsing, serialization and label mapping are needed.","reason":"Native causal binary judgment from supplied background and question.","candidate_origin":"Original yes/no outcomes"},"random_baseline":{"value":0.5,"label":"Uniform random choice","kind":"random_accuracy","method":"Exact expectation","explanation":"Average accepted-answer count / option count across the actual sampled fields. Includes all tied correct moves/statements where the scorer accepts ties; variable option counts are averaged case by case.","monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.8858},"sources":[{"name":"causalNLP/cladder","url":"https://github.com/causalNLP/cladder.git","revision":"3d2d1169b4b939a09048a6a75956c8972a93cc38"}],"licences":[{"licence":"MIT","source":"archived source file","file":"LICENSE"}],"source_revision":"3d2d1169b4b939a09048a6a75956c8972a93cc38"},{"id":45,"dataset":"HLE","short":null,"kind":"static","area":"knowledge","area_label":"Knowledge & Reasoning","area_color":"#5976bd","frozen_panel_area":null,"in_panel":true,"in_frozen_panel":false,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"accuracy","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"gold":1.2,"explainer":"Expert-level multiple choice from Humanity's Last Exam, text-only questions, 5 to 26 options.","comparability_note":"Not used in the frontier comparison on Jev's page: published HLE numbers cover the full benchmark and are not comparable to this suite's text-only multiple-choice subset.","references":[{"kind":"dataset","url":"https://huggingface.co/datasets/cais/hle"}],"cases":501,"requests":513,"fields":513,"source_cases":513,"unit":"case","scoring_metric":"text-only original-MC accuracy","subset_rule":"All 513 prepared cases retained.","source_filtering":"2,500 HLE rows → 513 text-only original multiple-choice items; excluded 342 image-dependent and 1645 open-answer items (IDs recorded).","note":"Only original text-only MC items; image-dependent/open-answer items excluded with recorded IDs. Not full HLE accuracy. Release v1: All prepared cases retained.","suite_category":"Knowledge and science","subcategory":"Frontier expert knowledge","origin":"Original-task benchmarks","adaptation":{"class":"B_mechanical","class_description":"Extra deterministic processing changes the evaluated view: filtering, extraction, search, enumeration, pairing, simulation or numeric encoding. No per-item semantic option authoring.","reason":"Filters to text-only original-MC HLE items; does not invent choices for free-response items.","candidate_origin":"Original multiple-choice options only"},"random_baseline":{"value":0.1641,"label":"Uniform random choice","kind":"random_accuracy","method":"Exact expectation","explanation":"Average accepted-answer count / option count across the actual sampled fields. Includes all tied correct moves/statements where the scorer accepts ties; variable option counts are averaged case by case.","monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.1936},"sources":[{"name":"cais/hle","url":"https://huggingface.co/datasets/cais/hle","revision":null}],"licences":[{"licence":"MIT","source":"archived dataset card","file":"README.md"}],"source_revision":null},{"id":48,"dataset":"ForecastBench","short":null,"kind":"static","area":"arts","area_label":"Arts & Human Taste","area_color":"#b6698a","frozen_panel_area":null,"in_panel":false,"in_frozen_panel":false,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"Brier (lower is better)","lower_is_better":false,"index_rule":"clip((0.25 - Brier) / (0.25 - 0)) x coverage","left_out_of_index":false,"gold":null,"explainer":"Gives the probability that an event resolving July–September 2026 happens, from earlier information only.","comparability_note":null,"references":[{"kind":"paper","url":"https://arxiv.org/abs/2409.19839"},{"kind":"dataset","url":"https://github.com/forecastingresearch/forecastbench-datasets"}],"cases":10139,"requests":10139,"fields":10139,"source_cases":10139,"unit":"dated event forecast","scoring_metric":"Brier score; calibration; log loss","subset_rule":"All 10,139 prepared cases retained.","source_filtering":"123,475 resolution records → 10,139 eligible forecasts resolving 2026-07-01 to 2026-09-18; excluded 104,563 outside the date policy and 8,773 unresolved or non-binary.","note":"10139 resolved binary/joint-event forecasts, resolution dates July1–September18 2026. Frozen pre-forecast context; market prior probabilities omitted, historical series baselines retained. Brier/log loss; unpublished Jev cutoff means contamination not certified. Release v1: All prepared cases retained.","suite_category":"Forecasting and collective decisions","subcategory":"Resolved-event forecasting","origin":"Jev-adapted benchmarks","adaptation":{"class":"B_mechanical","class_description":"Extra deterministic processing changes the evaluated view: filtering, extraction, search, enumeration, pairing, simulation or numeric encoding. No per-item semantic option authoring.","reason":"Date/resolution filtering, frozen evidence extraction and joint-event rendering; source probabilities define the task, not invented outcomes.","candidate_origin":"Binary events and source joint-event directions"},"random_baseline":{"value":0.25,"label":"Always predict p = 0.5","kind":"uninformative_forecast","method":"Exact","explanation":"Binary Brier loss: 0 is perfect and 1 is maximally wrong. Reporting probability0.5 for every event always scores0.25. This is an uninformative probability forecast, not a random hard yes/no guess (whose expected Brier loss is0.5).","monte_carlo_standard_error":null,"secondary_reference":{"label":"Always p=0.5 log loss","value":0.6931}},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":null},"sources":[{"name":"forecastingresearch/forecastbench-datasets","url":"https://github.com/forecastingresearch/forecastbench-datasets.git","revision":"da48cfb37db3e674fd57030266c92009873be4b6"}],"licences":[{"licence":"CC-BY-SA-4.0","source":"archived source file","file":"LICENSE"}],"source_revision":"da48cfb37db3e674fd57030266c92009873be4b6","retired":true},{"id":50,"dataset":"Habermas Machine","short":"Habermas","kind":"static","area":"arts","area_label":"Arts & Human Taste","area_color":"#b6698a","frozen_panel_area":"arts","in_panel":true,"in_frozen_panel":true,"dropped_interactive":false,"in_reproduction_suite":true,"metric":"accuracy","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":"Given a group's opinions on a policy question, predicts which candidate statement the group ranked highest.","comparability_note":null,"references":[{"kind":"dataset","url":"https://github.com/google-deepmind/habermas_machine"}],"cases":1676,"requests":1676,"fields":1676,"source_cases":2779,"unit":"initial discussion panel","scoring_metric":"human preference agreement; preference regret","subset_rule":"All 1,676 prepared cases retained.","source_filtering":null,"note":"Complete initial human panels only; official evaluation cohorts. All tied minimum-summed-rank candidates accepted by the custom scorer. No count cap. Release v1: All prepared cases retained.","suite_category":"Forecasting and collective decisions","subcategory":"Democratic deliberation","origin":"Jev-adapted benchmarks","adaptation":{"class":"B_mechanical","class_description":"Extra deterministic processing changes the evaluated view: filtering, extraction, search, enumeration, pairing, simulation or numeric encoding. No per-item semantic option authoring.","reason":"Complete-panel filtering and predeclared rank aggregation; candidate prose/human preferences are source-provided.","candidate_origin":"Released consensus statements"},"random_baseline":{"value":0.311,"label":"Uniform random choice","kind":"random_accuracy","method":"Exact expectation","explanation":"Average accepted-answer count / option count across the actual sampled fields. Includes all tied correct moves/statements where the scorer accepts ties; variable option counts are averaged case by case.","monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.4457},"sources":[{"name":"google-deepmind/habermas_machine","url":"https://github.com/google-deepmind/habermas_machine.git","revision":"7923b71966c14136077c78d7841bc9e1a182dfe0"}],"licences":[{"licence":"Apache-2.0","source":"archived source file","file":"LICENSE"}],"source_revision":"7923b71966c14136077c78d7841bc9e1a182dfe0"},{"id":51,"dataset":"OpenBookQA","short":null,"kind":"calibration extension","area":"knowledge","area_label":"Knowledge & Reasoning","area_color":"#5976bd","frozen_panel_area":null,"in_panel":false,"in_frozen_panel":false,"dropped_interactive":false,"in_reproduction_suite":false,"metric":"accuracy","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":"Answer elementary-science questions from four original choices, without the supporting fact.","comparability_note":null,"references":[],"cases":500,"requests":500,"fields":500,"source_cases":500,"unit":"case","scoring_metric":"accuracy; Brier; NLL; ECE; per-type/tier diagnostics","subset_rule":"OpenBookQA validation split, all 500 questions; main split without auxiliary gold facts.","source_filtering":null,"note":"See calibration-extension-v1/README.md; source protocol preserved.","suite_category":"Knowledge & Reasoning","subcategory":"Calibration and decision accuracy","origin":"Original-task benchmarks","adaptation":{"class":"A_direct","class_description":"Original finite choices, class labels, or probabilities map directly; only parsing, serialization and label mapping are needed.","reason":"Wire-format conversion; original task retained.","candidate_origin":"Published options and labels"},"random_baseline":{"value":0.25,"label":"Uniform random choice","kind":"random_accuracy","method":null,"explanation":"Uniform guesses among the supplied answer classes; this is not a majority-class baseline.","monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":null},"sources":[{"name":"allenai/openbookqa","url":"https://huggingface.co/datasets/allenai/openbookqa","revision":null},{"name":"scienthoon/jev-ood-calibration","url":"https://github.com/scienthoon/jev-ood-calibration","revision":null}],"licences":[{"licence":"MIT","source":"archived source file","file":"LICENSE"}],"source_revision":null},{"id":52,"dataset":"CommonsenseQA","short":null,"kind":"calibration extension","area":"knowledge","area_label":"Knowledge & Reasoning","area_color":"#5976bd","frozen_panel_area":null,"in_panel":false,"in_frozen_panel":false,"dropped_interactive":false,"in_reproduction_suite":false,"metric":"accuracy","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":"Answer everyday commonsense questions from five original choices.","comparability_note":null,"references":[],"cases":1221,"requests":1221,"fields":1221,"source_cases":1221,"unit":"case","scoring_metric":"accuracy; Brier; NLL; ECE; per-type/tier diagnostics","subset_rule":"CommonsenseQA validation split, all 1,221 questions.","source_filtering":null,"note":"See calibration-extension-v1/README.md; source protocol preserved.","suite_category":"Knowledge & Reasoning","subcategory":"Calibration and decision accuracy","origin":"Original-task benchmarks","adaptation":{"class":"A_direct","class_description":"Original finite choices, class labels, or probabilities map directly; only parsing, serialization and label mapping are needed.","reason":"Wire-format conversion; original task retained.","candidate_origin":"Published options and labels"},"random_baseline":{"value":0.2,"label":"Uniform random choice","kind":"random_accuracy","method":null,"explanation":"Uniform guesses among the supplied answer classes; this is not a majority-class baseline.","monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":null},"sources":[{"name":"tau/commonsense_qa","url":"https://huggingface.co/datasets/tau/commonsense_qa","revision":null},{"name":"scienthoon/jev-ood-calibration","url":"https://github.com/scienthoon/jev-ood-calibration","revision":null}],"licences":[{"licence":"MIT","source":"archived source file","file":"LICENSE"}],"source_revision":null},{"id":53,"dataset":"Support-ticket calibration","short":"Support tickets","kind":"calibration extension","area":"tools","area_label":"Tools & Automation","area_color":"#b77a38","frozen_panel_area":null,"in_panel":false,"in_frozen_panel":false,"dropped_interactive":false,"in_reproduction_suite":false,"metric":"accuracy","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":"Judge a support ticket for queue, anger and priority, against deliberately noisy labels.","comparability_note":null,"references":[],"cases":300,"requests":900,"fields":900,"source_cases":300,"unit":"case","scoring_metric":"accuracy; Brier; NLL; ECE; per-type/tier diagnostics","subset_rule":"Exact published 900 records grouped as 300 generated tickets; three tracks (queue, anger, priority); noisy labels preserved.","source_filtering":null,"note":"See calibration-extension-v1/README.md; source protocol preserved.","suite_category":"Tools & Automation","subcategory":"Calibration and decision accuracy","origin":"Original-task benchmarks","adaptation":{"class":"A_direct","class_description":"Original finite choices, class labels, or probabilities map directly; only parsing, serialization and label mapping are needed.","reason":"Wire-format conversion; original task retained.","candidate_origin":"Published options and labels"},"random_baseline":{"value":null,"label":null,"kind":null,"method":null,"explanation":null,"monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":null},"sources":[{"name":"scienthoon/jev-ood-calibration","url":"https://github.com/scienthoon/jev-ood-calibration","revision":null}],"licences":[{"licence":"MIT","source":"archived source file","file":"LICENSE"}],"source_revision":null},{"id":54,"dataset":"Phishing difficulty gradient","short":"Phishing gradient","kind":"calibration extension","area":"retrieval","area_label":"Retrieval & Classification","area_color":"#3e8e5e","frozen_panel_area":null,"in_panel":false,"in_frozen_panel":false,"dropped_interactive":false,"in_reproduction_suite":false,"metric":"accuracy","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":"Judge generated emails as phishing across four difficulty tiers, from obvious to misleading.","comparability_note":null,"references":[],"cases":800,"requests":20,"fields":800,"source_cases":800,"unit":"message","scoring_metric":"accuracy; Brier; NLL; ECE; per-type/tier diagnostics","subset_rule":"Original four tiers, 200 messages each, batched in 40s with the published prompt (20 requests, 800 decisions).","source_filtering":null,"note":"See calibration-extension-v1/README.md; source protocol preserved.","suite_category":"Language & Retrieval","subcategory":"Calibration and decision accuracy","origin":"Original-task benchmarks","adaptation":{"class":"A_direct","class_description":"Original finite choices, class labels, or probabilities map directly; only parsing, serialization and label mapping are needed.","reason":"Wire-format conversion; original task retained.","candidate_origin":"Published options and labels"},"random_baseline":{"value":0.5,"label":"Uniform random choice","kind":"random_accuracy","method":null,"explanation":"Uniform guesses among the supplied answer classes; this is not a majority-class baseline.","monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":null},"sources":[{"name":"SamuelSacco/jev-exploration","url":"https://github.com/SamuelSacco/jev-exploration","revision":null}],"licences":[],"source_revision":null},{"id":55,"dataset":"Email spam classification","short":"Email spam","kind":"calibration extension","area":"retrieval","area_label":"Retrieval & Classification","area_color":"#3e8e5e","frozen_panel_area":null,"in_panel":false,"in_frozen_panel":false,"dropped_interactive":false,"in_reproduction_suite":false,"metric":"accuracy","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":"Detect spam in real emails, scored on the structured-focus question wording.","comparability_note":null,"references":[],"cases":3000,"requests":3000,"fields":12000,"source_cases":19528,"unit":"case","scoring_metric":"accuracy; Brier; NLL; ECE; per-type/tier diagnostics","subset_rule":"3,000 uniformly sampled emails without replacement from 19,528; Python Random seed 20260920. See calibration-extension-v1/README.md; source protocol preserved.","source_filtering":"3,000 emails sampled uniformly without replacement from 19,528 (Python random.Random(20260920)); selected IDs pinned in the extension manifest.","note":"3,000 uniformly sampled emails without replacement from 19,528; Python Random seed 20260920. See calibration-extension-v1/README.md; source protocol preserved.","suite_category":"Language & Retrieval","subcategory":"Calibration and decision accuracy","origin":"Original-task benchmarks","adaptation":{"class":"A_direct","class_description":"Original finite choices, class labels, or probabilities map directly; only parsing, serialization and label mapping are needed.","reason":"Wire-format conversion; original task retained.","candidate_origin":"Published options and labels"},"random_baseline":{"value":0.5,"label":"Uniform random choice","kind":"random_accuracy","method":null,"explanation":"Uniform guesses among the supplied answer classes; this is not a majority-class baseline.","monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":null},"sources":[{"name":"bitnovus/jev-spam-eval","url":"https://github.com/bitnovus/jev-spam-eval","revision":null}],"licences":[{"licence":"MIT","source":"archived source file","file":"LICENSE"}],"source_revision":null},{"id":56,"dataset":"PhishNChips phishing decisions","short":"PhishNChips","kind":"calibration extension","area":"retrieval","area_label":"Retrieval & Classification","area_color":"#3e8e5e","frozen_panel_area":null,"in_panel":true,"in_frozen_panel":false,"dropped_interactive":false,"in_reproduction_suite":false,"metric":"accuracy","lower_is_better":false,"index_rule":null,"left_out_of_index":false,"explainer":"Says whether an email and its link are phishing or legitimate.","comparability_note":null,"references":[{"kind":"dataset","url":"https://huggingface.co/datasets/AreLit/PhishNChips"},{"kind":"code","url":"https://github.com/anisselbd/jev-phishing-bench"}],"cases":2000,"requests":2000,"fields":18000,"source_cases":2000,"unit":"case","scoring_metric":"accuracy; Brier; NLL; ECE; per-type/tier diagnostics","subset_rule":"Exact PhishNChips v5.2 source checksum, all 2,000 emails with the original nine-question payload.","source_filtering":null,"note":"See calibration-extension-v1/README.md; source protocol preserved.","suite_category":"Language & Retrieval","subcategory":"Calibration and decision accuracy","origin":"Original-task benchmarks","adaptation":{"class":"A_direct","class_description":"Original finite choices, class labels, or probabilities map directly; only parsing, serialization and label mapping are needed.","reason":"Wire-format conversion; original task retained.","candidate_origin":"Published options and labels"},"random_baseline":{"value":0.5,"label":"Uniform random choice","kind":"random_accuracy","method":null,"explanation":"Uniform guesses among the supplied answer classes; this is not a majority-class baseline.","monte_carlo_standard_error":null,"secondary_reference":null},"tracks":[],"jev":{"http_median_ms":null,"http_p95_ms":null,"score":0.8375},"sources":[{"name":"anisselbd/jev-phishing-bench","url":"https://github.com/anisselbd/jev-phishing-bench","revision":null},{"name":"AreLit/PhishNChips","url":"https://github.com/AreLit/PhishNChips","revision":null}],"licences":[{"licence":"per-source (MIT project code; CC-BY-4.0 Nazario corpus; OpenPhish/PhishTank/Tranco terms)","source":"archived SOURCE_LICENSES.md","file":"SOURCE_LICENSES.md"}],"source_revision":null}],"index":{"panel_id":"decision-index-0.3","proposal_panel_id":"proposed-core25-v1","proposal_status":"proposal_not_implemented","proposal_date":"2026-09-19","panel_full":37,"panel":37,"areas_count":5,"dropped":[{"id":7,"name":"MiniWoB++","frozen_area":"tools"},{"id":13,"name":"Boxoban","frozen_area":"games"},{"id":15,"name":"RTFM","frozen_area":"games"},{"id":16,"name":"ScienceWorld","frozen_area":"tools"},{"id":17,"name":"Hanabi","frozen_area":"games"},{"id":19,"name":"Codenames","frozen_area":"games"}],"folded":[{"from":"games","into":"knowledge","benchmarks":[{"id":31,"name":"ChessBench"}]}],"split":[{"from":"language","into":[{"id":"language","label":"Language Understanding","panel":[11,40,41]},{"id":"retrieval","label":"Retrieval & Classification","panel":[36,37]}]}],"headline_tracks":[{"id":40,"track":"iSarcasmEval-A-En","label":"A · English"}],"areas":[{"id":"knowledge","label":"Knowledge & Reasoning","color":"#5976bd","description":"Academic knowledge, commonsense, mathematics, and code.","jev_only":false,"folded_into":null,"weight":0.2585,"panel":[{"id":25,"name":"GPQA Diamond","short":"GPQA Diamond","gold":1.2},{"id":30,"name":"GSM8K","short":"GSM8K"},{"id":31,"name":"ChessBench","short":"ChessBench"},{"id":32,"name":"MuSR","short":"MuSR"},{"id":33,"name":"SATA-Bench","short":"SATA-Bench"},{"id":43,"name":"CRUXEval","short":"CRUXEval"},{"id":44,"name":"CLadder","short":"CLadder"},{"id":45,"name":"HLE","short":"HLE","gold":1.2},{"id":57,"name":"MMLU-Pro","short":"MMLU-Pro","gold":1.2},{"id":58,"name":"BBH","short":"BBH","gold":1.2},{"id":28,"name":"WinoGrande","short":"WinoGrande","gold":1.2}],"frozen_panel":[{"id":25,"name":"GPQA Diamond","dropped":false},{"id":30,"name":"GSM8K","dropped":false},{"id":31,"name":"ChessBench","dropped":false},{"id":32,"name":"MuSR","dropped":false},{"id":33,"name":"SATA-Bench","dropped":false},{"id":43,"name":"CRUXEval","dropped":false},{"id":44,"name":"CLadder","dropped":false},{"id":45,"name":"HLE","dropped":false},{"id":57,"name":"57","dropped":false},{"id":58,"name":"58","dropped":false}],"shown_only":[{"id":26,"name":"ARC-Easy"},{"id":27,"name":"ARC-Challenge"}]},{"id":"language","label":"Language Understanding","color":"#407e9e","description":"Read a text and judge what it means: entailment, sentiment, sarcasm and stance.","jev_only":false,"folded_into":null,"weight":0.2585,"panel":[{"id":11,"name":"ContractNLI","short":"ContractNLI"},{"id":12,"name":"ANLI","short":"ANLI","gold":1.2},{"id":29,"name":"HellaSwag","short":"HellaSwag","gold":1.2},{"id":38,"name":"ACOS","short":"ACOS"},{"id":39,"name":"FinEntity","short":"FinEntity"},{"id":40,"name":"iSarcasmEval","short":"iSarcasmEval"},{"id":41,"name":"VAST","short":"VAST"},{"id":42,"name":"NLI4CT","short":"NLI4CT"},{"id":59,"name":"RAGTruth","short":"RAGTruth"}],"frozen_panel":[{"id":11,"name":"ContractNLI","dropped":false},{"id":12,"name":"ANLI","dropped":false},{"id":28,"name":"WinoGrande","dropped":false},{"id":29,"name":"HellaSwag","dropped":false},{"id":38,"name":"ACOS","dropped":false},{"id":39,"name":"FinEntity","dropped":false},{"id":40,"name":"iSarcasmEval","dropped":false},{"id":41,"name":"VAST","dropped":false},{"id":42,"name":"NLI4CT","dropped":false},{"id":59,"name":"59","dropped":false}],"shown_only":[]},{"id":"retrieval","label":"Retrieval & Classification","color":"#3e8e5e","description":"Pick the right item from a large set: rank documents and products by relevance, route a message to one of many intents, sort out spam and phishing.","jev_only":false,"folded_into":null,"weight":0.2002,"panel":[{"id":4,"name":"BANKING77","short":"BANKING77","gold":1.2},{"id":5,"name":"CLINC150+OOS","short":"CLINC150","gold":1.2},{"id":36,"name":"BRIGHT","short":"BRIGHT","gold":1.2},{"id":37,"name":"Amazon ESCI","short":"Amazon ESCI"},{"id":56,"name":"PhishNChips phishing decisions","short":"PhishNChips"},{"id":61,"name":"HoVer","short":"HoVer"}],"frozen_panel":[{"id":4,"name":"BANKING77","dropped":false},{"id":5,"name":"CLINC150+OOS","dropped":false},{"id":36,"name":"BRIGHT","dropped":false},{"id":37,"name":"Amazon ESCI","dropped":false},{"id":56,"name":"PhishNChips phishing decisions","dropped":false},{"id":61,"name":"61","dropped":false}],"shown_only":[{"id":10,"name":"SGD/SGD-X"}]},{"id":"tools","label":"Tools & Automation","color":"#b77a38","description":"Tool choice, model routing, household control, and browser or simulator actions.","jev_only":false,"folded_into":null,"weight":0.1828,"panel":[{"id":1,"name":"BFCL","short":"BFCL","gold":1.2},{"id":2,"name":"ToolRet","short":"ToolRet"},{"id":3,"name":"API-Bank","short":"API-Bank","gold":1.2},{"id":9,"name":"Home appliance simulator","short":"Home appliances"},{"id":62,"name":"When2Call","short":"When2Call"}],"frozen_panel":[{"id":1,"name":"BFCL","dropped":false},{"id":2,"name":"ToolRet","dropped":false},{"id":3,"name":"API-Bank","dropped":false},{"id":9,"name":"Home appliance simulator","dropped":false},{"id":62,"name":"62","dropped":false}],"shown_only":[{"id":6,"name":"RouterBench"}]},{"id":"arts","label":"Arts & Human Taste","color":"#b6698a","description":"Poetry, music, humor, human preferences, and forecasting.","jev_only":false,"folded_into":null,"weight":0.1,"panel":[{"id":20,"name":"BPoMP","short":"BPoMP"},{"id":21,"name":"Humicroedit","short":"Humicroedit"},{"id":22,"name":"POP909-CL","short":"POP909"},{"id":23,"name":"cfcolor","short":"cfcolor"},{"id":50,"name":"Habermas Machine","short":"Habermas"},{"id":64,"name":"New Yorker","short":"New Yorker"}],"frozen_panel":[{"id":20,"name":"BPoMP","dropped":false},{"id":21,"name":"Humicroedit","dropped":false},{"id":22,"name":"POP909-CL","dropped":false},{"id":23,"name":"cfcolor","dropped":false},{"id":48,"name":"ForecastBench","dropped":false},{"id":50,"name":"Habermas Machine","dropped":false},{"id":64,"name":"64","dropped":false}],"shown_only":[]},{"id":"games","label":"Games & Strategy","color":"#ad6f48","description":"Move selection, spatial planning, rules, and decisions against other players.","jev_only":true,"folded_into":"knowledge","weight":0,"panel":[],"frozen_panel":[{"id":31,"name":"ChessBench","dropped":false}],"shown_only":[]}],"frozen_panel":[{"id":"knowledge","label":"Knowledge & Reasoning","benchmarks":[{"id":24,"name":null,"dropped":false},{"id":25,"name":null,"dropped":false},{"id":30,"name":null,"dropped":false},{"id":43,"name":null,"dropped":false},{"id":44,"name":null,"dropped":false}]},{"id":"language","label":"Language & Retrieval","benchmarks":[{"id":11,"name":null,"dropped":false},{"id":36,"name":null,"dropped":false},{"id":37,"name":null,"dropped":false},{"id":40,"name":null,"dropped":false},{"id":41,"name":null,"dropped":false}]},{"id":"tools","label":"Tools & Automation","benchmarks":[{"id":1,"name":null,"dropped":false},{"id":2,"name":null,"dropped":false},{"id":6,"name":null,"dropped":false},{"id":7,"name":null,"dropped":true},{"id":16,"name":null,"dropped":true}]},{"id":"games","label":"Games & strategy","benchmarks":[{"id":31,"name":null,"dropped":false},{"id":13,"name":null,"dropped":true},{"id":15,"name":null,"dropped":true},{"id":17,"name":null,"dropped":true},{"id":19,"name":null,"dropped":true}]},{"id":"arts","label":"Arts & Human Taste","benchmarks":[{"id":20,"name":null,"dropped":false},{"id":21,"name":null,"dropped":false},{"id":22,"name":null,"dropped":false},{"id":23,"name":null,"dropped":false},{"id":50,"name":null,"dropped":false}]}],"formulas":[{"id":"balanced_raw","normalization":"raw","formula":"100*(0.2585*category[knowledge] + 0.2585*category[language] + 0.2002*category[retrieval] + 0.1828*category[tools] + 0.1000*category[arts])"},{"id":"balanced_skill","normalization":"skill","formula":"Public index: 100*(0.2585*category[knowledge] + 0.2585*category[language] + 0.2002*category[retrieval] + 0.1828*category[tools] + 0.1000*category[arts]). Full score: 0.60*private_same_skills + 0.40*private_new_domains, both mapped onto the public scale with untuned stock models; the public index is shown but not counted"},{"id":"breadth_skill","normalization":"skill","shift":0.1,"formula":"100*(product_c((0.1+0.9*category[c])**w[c])-0.1)/0.9"}],"shown":"balanced_skill","hidden":["breadth_skill"],"functions":{"clip":"min(1,max(0,z))","raw_fraction":"clip(x)","raw_brier":"clip(1-x)","raw_bounded_score":"clip((x-L)/(U-L))","skill":"clip((raw-reference)/(1-reference))","benchmark":"sum_t(track_weight[t]*normalized_track[t])","category":"weighted mean of the area's benchmark scores (gold ★ benchmarks weigh 1.2, the others 1.0)"},"baseline_policy":{"versioned":true,"empirical_baselines_status":"must_freeze_before_ranking","uniform_f1_monte_carlo_se_max":0.001,"uniform_f1_fixed_label_universe":true,"raw_maximum":1,"selected_interactive_reference":0,"selected_interactive_reference_label":"task floor, not measured random policy"},"coverage":{"denominator":"all frozen source cases or episodes, preserving fixed subtrack and category weights","formula":"sum_c(0.2*mean_b(sum_t(w_bt*valid_bt/planned_bt)))","multiply_score_by_coverage":false,"official_requires":"all scheduled cases terminal valid or certified model failure; no pending evaluator defects or unrun results"},"failures":{"accuracy":0,"ndcg_incomplete_query":0,"quality":0,"game":"minimum terminal utility; zero for selected games","brier":1,"f1_per_class":"2*TP/(2*TP+FP+FN+M); denominator zero maps to zero","transient_retry_count":2,"valid_wrong_answer_retries":0,"unsupported":"full-denominator failure; never omit","infrastructure_bug":"repair; pending until fixed","unrun":"zero only in labeled provisional lower bound; final rank withheld"},"uncertainty":{"method":"paired hierarchical cluster bootstrap through complete index formula","replicates":2000,"confidence_level":0.95,"weights":"fixed","difference_intervals":true,"boundary_binary_diagnostics":"exact binomial or Wilson interval"},"supplementary":{"forecastbench":{"catalog_id":48,"reason":"retrospective resolved events and uncertified model cutoffs","raw":"clip(1-Brier)","skill":"clip(1-Brier/0.25)","index_weight":0},"leduc":{"catalog_id":18,"reason":"freeze terminal utility anchors/opponent panel before inclusion","index_weight":0}},"efficiency_in_score":false,"clarifications":["CRUXEval uses the sole frozen output-choice track; the proposal incorrectly mentioned an input track that does not exist in this suite.","Interactive scores use existing frozen pilots and first complete benchmark runs, not the proposed expanded three-repeat protocol.","Reproductions have no interactive runs. Those six benchmarks receive zero only as labeled provisional lower bounds, never an official final ranking.","Unsupported source cases receive zero before chance normalization. F1 uses fixed label universes and conservative missing-label penalties.","All linked requests must succeed for source-case support; no successful-intersection score shortcut.","Point estimates only; paired cluster-bootstrap uncertainty is not yet calculated.","The index panel is the Decision Index 0.3 panel (decision-index-0.3), not the frozen core25 proposal; core25 scores are still computed and kept as frozen_scores.","Benchmarks outside the frozen core25 panel enter with their board score times the answered share (unanswered or unsupported counts as wrong); the 19 core25 static benchmarks keep their case-group coverage adjustment."],"steps":["Score every benchmark with its own primary metric and adjust it for coverage: an unanswered, refused, errored or unsupported request counts as wrong (for the 19 core25 static benchmarks a case whose linked requests did not all succeed scores zero; for the others the score on answered requests is multiplied by the answered share).","Benchmarks with several tracks (RouterBench 0-shot/5-shot, GSM8K 4-/10-choice) average their tracks equally; iSarcasmEval uses track A English as its headline and track B never enters the index.","Lower-is-better metrics are converted against their baseline: ForecastBench value = clip((0.25 - Brier) / (0.25 - 0)) x coverage, so always predict p = 0.5 (Brier 0.25) or worse scores 0 and a perfect forecast scores 1.","Each of the five areas is the gold-weighted mean of its benchmarks (★ = gold, weight 1.2): Knowledge & Reasoning (10: GPQA Diamond ★, GSM8K, ChessBench, MuSR, SATA-Bench, CRUXEval, CLadder, HLE ★, MMLU-Pro ★, BBH ★), Language Understanding (10: ContractNLI, ANLI ★, WinoGrande ★, HellaSwag ★, ACOS, FinEntity, iSarcasmEval, VAST, NLI4CT, RAGTruth), Retrieval & Classification (6: BANKING77 ★, CLINC150 ★, BRIGHT ★, Amazon ESCI, PhishNChips, HoVer), Tools & Automation (5: BFCL ★, ToolRet, API-Bank ★, Home appliances, When2Call), Arts & Human Taste (7: BPoMP, Humicroedit, POP909, cfcolor, ForecastBench ★, Habermas, New Yorker).","Chance-correct every benchmark: k = clip((s − chance) / (1 − chance)) with the chance level listed below, so 0 is random guessing and 1 is perfect. Multi-track and retrieval benchmarks correct each track or query against its own chance level; ForecastBench is already 0 at its baseline.","Decision Index = 100 × weighted mean of the five area scores, each the gold-weighted mean of its chance-corrected benchmarks. Arts & Human Taste is fixed at 10%; the other four areas keep the 0.2.1 weights (Knowledge & Reasoning 25.8%, Language Understanding 25.8%, Retrieval & Classification 20.0%, Tools & Automation 18.3%). Gold ★ benchmarks weigh 1.2 inside their area and every other benchmark 1.0, normalized within the area. GSM8K is left out of the gold tier because its distractor options leak the answer. 37 benchmarks in all.","Not in the index, still shown on the board: MMLU, ARC-Easy, ARC-Challenge, RouterBench, SGD. RouterBench and SGD left the index in Decision Index 0.2.1; the reasons are listed below.","Raw accuracy (not chance-corrected) repeats the same averaging on the uncorrected scores and is shown on model pages; the breadth variant takes a shifted geometric mean of the five chance-corrected area scores and is computed but hidden.","The six interactive environments of the frozen core25 panel stay out for every model, Jev included, because no reproduction has run them."],"worked_example":{"model":"Jev","areas":[{"id":"knowledge","raw":0.6391,"skill":0.5387,"n":11},{"id":"language","raw":0.7097,"skill":0.5923,"n":9},{"id":"retrieval","raw":0.6825,"skill":0.5542,"n":6},{"id":"tools","raw":0.7764,"skill":0.7509,"n":5},{"id":"arts","raw":0.583,"skill":0.3908,"n":6}],"scores":{"balanced_raw":68.55,"balanced_skill":57.96,"breadth_skill":57.27},"frozen_scores":{"balanced_raw":54.47,"balanced_skill":43.61,"breadth_skill":36.98}},"frontier_comparison":{"note":"Jev's own page compares its suite scores with published frontier numbers on a few benchmarks; a benchmark enters that comparison only when the published protocol matches the suite's subset.","excluded":[{"id":45,"name":"HLE","why":"Not used in the frontier comparison on Jev's page: published HLE numbers cover the full benchmark and are not comparable to this suite's text-only multiple-choice subset."}]},"hashes":{"panel_proposal_sha256":"a9823d5d8ce1dcf81da91f285d9de5b77b8bb1c9a5ad4fa2d51df6c3d0850a21","static_corpus_sha256":"b2b56d6fb636837ca469e689087bdbf373dda8de7638aa2da6793e6eda0792d5","random_baselines_sha256":"b68bb94e1ce0d2a40b4f69d0b7a9acf796c4bc64c973d5c291625aec34629bfa"},"frozen_panel_id":"core25-observed-protocol-v1","also_shown":[{"id":"balanced_raw","label":"raw accuracy (not chance-corrected)"}],"chance_levels":[{"id":25,"name":"GPQA Diamond","area":"knowledge","chance":0.25,"label":"Uniform random choice","method":"expected score of uniform random choice over each question's options"},{"id":30,"name":"GSM8K","area":"knowledge","chance":0.175,"label":"Uniform random choice","method":"per track: 4-choice and 10-choice versions corrected separately, then averaged"},{"id":31,"name":"ChessBench","area":"knowledge","chance":0.0819,"label":"Uniform random choice","method":"expected score of uniform random choice over each question's options"},{"id":32,"name":"MuSR","area":"knowledge","chance":0.371,"label":"Uniform random choice","method":"expected score of uniform random choice over each question's options"},{"id":33,"name":"SATA-Bench","area":"knowledge","chance":0.0131,"label":"Random whole-case guesses","method":"expected score of uniform random choice over each question's options"},{"id":43,"name":"CRUXEval","area":"knowledge","chance":0.3697,"label":"Uniform random choice","method":"expected score of uniform random choice over each question's options"},{"id":44,"name":"CLadder","area":"knowledge","chance":0.5,"label":"Uniform random choice","method":"expected score of uniform random choice over each question's options"},{"id":45,"name":"HLE","area":"knowledge","chance":0.1641,"label":"Uniform random choice","method":"expected score of uniform random choice over each question's options"},{"id":57,"name":"MMLU-Pro","area":"knowledge","chance":0.1109,"label":"uniform random choice, mean of 1/options","method":"expected score of uniform random choice over each question's options"},{"id":58,"name":"BBH","area":"knowledge","chance":0.3101,"label":"uniform random choice, mean of 1/options","method":"expected score of uniform random choice over each question's options"},{"id":11,"name":"ContractNLI","area":"language","chance":0.3085,"label":"Random choices · F1","method":"F1 of random guessing over the fixed label set"},{"id":12,"name":"ANLI","area":"language","chance":0.3324,"label":"Random choices · F1","method":"expected score of uniform random choice over each question's options"},{"id":28,"name":"WinoGrande","area":"knowledge","chance":0.5,"label":"Uniform random choice","method":"expected score of uniform random choice over each question's options"},{"id":29,"name":"HellaSwag","area":"language","chance":0.25,"label":"Uniform random choice","method":"expected score of uniform random choice over each question's options"},{"id":38,"name":"ACOS","area":"language","chance":0.031,"label":"Always yes · per-review F1","method":"per-review F1 of answering yes to every aspect-and-sentiment pair (a fair coin scores lower)"},{"id":39,"name":"FinEntity","area":"language","chance":0.3201,"label":"Random choices · F1","method":"expected score of uniform random choice over each question's options"},{"id":40,"name":"iSarcasmEval","area":"language","chance":0.2227,"label":"Random choices · F1","method":"track A English: F1 of random guessing on the sarcastic class"},{"id":41,"name":"VAST","area":"language","chance":0.3333,"label":"Random choices · F1","method":"F1 of random guessing over the fixed label set"},{"id":42,"name":"NLI4CT","area":"language","chance":0.486,"label":"Random choices · F1","method":"expected score of uniform random choice over each question's options"},{"id":59,"name":"RAGTruth","area":"language","chance":0.5177,"label":"Always \"hallucinated\" · F1","method":"F1 of always answering \"hallucinated\" (a fair coin scores lower)"},{"id":4,"name":"BANKING77","area":"retrieval","chance":0.0127,"label":"Random choices · F1","method":"expected score of uniform random choice over each question's options"},{"id":5,"name":"CLINC150","area":"retrieval","chance":0.006,"label":"Random choices · F1","method":"expected score of uniform random choice over each question's options"},{"id":36,"name":"BRIGHT","area":"retrieval","chance":0.116,"label":"Random ranking, answerable queries","method":"per query: expected nDCG@10 of a random ranking of that query's candidates, answerable queries only"},{"id":37,"name":"Amazon ESCI","area":"retrieval","chance":0.2027,"label":"Random choices · F1","method":"F1 of random guessing over the fixed label set"},{"id":56,"name":"PhishNChips","area":"retrieval","chance":0.5,"label":"uniform random choice, mean of 1/options","method":"expected score of uniform random choice over each question's options"},{"id":61,"name":"HoVer","area":"retrieval","chance":0.5,"label":"uniform random choice, mean of 1/options","method":"expected score of uniform random choice over each question's options"},{"id":1,"name":"BFCL","area":"tools","chance":0.2592,"label":"Random whole-case guesses","method":"all fields of a case right by chance"},{"id":2,"name":"ToolRet","area":"tools","chance":0.1341,"label":"Random ranking, answerable queries","method":"per query: expected nDCG@10 of a random ranking of that query's candidates, answerable queries only"},{"id":3,"name":"API-Bank","area":"tools","chance":0.0189,"label":"Uniform random choice","method":"expected score of uniform random choice over each question's options"},{"id":9,"name":"Home appliances","area":"tools","chance":0,"label":"Random whole-case guesses","method":"all fields of a case right by chance, on the kept rows"},{"id":62,"name":"When2Call","area":"tools","chance":0.25,"label":"uniform random choice, mean of 1/options","method":"expected score of uniform random choice over each question's options"},{"id":20,"name":"BPoMP","area":"arts","chance":0.5,"label":"Uniform random choice","method":"expected score of uniform random choice over each question's options"},{"id":21,"name":"Humicroedit","area":"arts","chance":0.5,"label":"Uniform random choice","method":"expected score of uniform random choice over each question's options"},{"id":22,"name":"POP909","area":"arts","chance":0.0078,"label":"Uniform random choice","method":"expected score of uniform random choice over each question's options"},{"id":23,"name":"cfcolor","area":"arts","chance":0.5,"label":"Uniform random choice","method":"expected score of uniform random choice over each question's options"},{"id":50,"name":"Habermas","area":"arts","chance":0.311,"label":"Uniform random choice","method":"expected score of uniform random choice over each question's options"},{"id":64,"name":"New Yorker","area":"arts","chance":0.2,"label":"uniform random choice, mean of 1/options","method":"expected score of uniform random choice over each question's options"}],"chance_paragraph":"The Decision Index is chance-corrected. Every benchmark score is first mapped to (score − chance) / (1 − chance), clipped to 0–1, where chance is what random guessing scores on that benchmark: 0.5 on a yes/no question, 0.25 with four options, the expected nDCG of a random ranking for retrieval, the F1 of random labels for F1-scored tasks. 0 means no better than random guessing and 100 means perfect, so a benchmark made of yes/no questions no longer hands out 50 free points and a benchmark with hundreds of options no longer looks harder than it is. Area scores and the index average these corrected values exactly as before. The raw, uncorrected accuracy is still computed and shown on each model page as raw accuracy (not chance-corrected).","lower_rules":[],"not_in_index":[{"id":6,"name":"RouterBench","note":"Not in the index: the prompt reveals that GPT-4 is the best route, so the top models all pick GPT-4 and score within a fraction of a point of each other. Scores stay visible."},{"id":10,"name":"SGD","note":"Not in the index: build bug, pending re-run. The builder gave every SGD row the whole dialogue, including the turns after the one being classified. Scores stay visible; SGD returns to the index once re-run with the fixed builder."},{"id":24,"name":"MMLU","note":null},{"id":26,"name":"ARC-Easy","note":null},{"id":27,"name":"ARC-Challenge","note":null}],"panel_change":"Decision Index 0.3 keeps the 0.2.1 public panel and rules with these changes: GSM8K is rebuilt, ForecastBench is retired and WinoGrande counts under Knowledge & Reasoning, so the public index averages 37 benchmarks. The headline is the Full score: 60% private tests of the same skills and 40% private tasks from new domains, both mapped onto the public scale with untuned stock models. Since Decision Index 0.4 the public index is shown alongside but does not count toward the Full score. ACOS, ToolRet, BRIGHT, RAGTruth and Home appliances stay rescored as described under Rescored benchmarks.","weights":{"knowledge":0.2585,"language":0.2585,"retrieval":0.2002,"tools":0.1828,"arts":0.1},"weights_pct":{"knowledge":"25.8%","language":"25.8%","retrieval":"20.0%","tools":"18.3%","arts":"10.0%"},"sizing":{"rule":"sqrt","fixed":{"arts":0.1},"text":"Arts & Human Taste is fixed at 10%; the other four areas keep the 0.2.1 weights, set in proportion to √n of their benchmark count at the time (Tools & Automation 18.3%, Retrieval & Classification 20.0%, Knowledge & Reasoning 25.8%, Language Understanding 25.8%). In 0.3 Knowledge & Reasoning holds 11 benchmarks and Language Understanding 9."},"gold":{"weight":1.2,"text":"Gold ★ benchmarks weigh 1.2 inside their area and every other benchmark 1.0, normalized within the area. GSM8K is left out of the gold tier because its distractor options leak the answer.","benchmarks":[{"id":57,"name":"MMLU-Pro","area":"knowledge"},{"id":58,"name":"BBH","area":"knowledge"},{"id":25,"name":"GPQA Diamond","area":"knowledge"},{"id":45,"name":"HLE","area":"knowledge"},{"id":12,"name":"ANLI","area":"language"},{"id":28,"name":"WinoGrande","area":"knowledge"},{"id":29,"name":"HellaSwag","area":"language"},{"id":4,"name":"BANKING77","area":"retrieval"},{"id":5,"name":"CLINC150","area":"retrieval"},{"id":36,"name":"BRIGHT","area":"retrieval"},{"id":1,"name":"BFCL","area":"tools"},{"id":3,"name":"API-Bank","area":"tools"}]},"rescored":[{"id":38,"name":"ACOS","what":"metric","line":"Scored with F1 per review, averaged over reviews: the aspect-and-sentiment pairs a model says yes to are compared with the gold pairs of that review. Whole-review exact match left almost everyone near zero (the best entrant got 9.2% of reviews fully right). Chance is answering yes to every pair (0.031), which beats a fair coin."},{"id":2,"name":"ToolRet","what":"subset","line":"Only queries with at least one relevant tool among their 32 candidates are scored (685 of 1,000): on the others every ranking scores 0. Chance is recomputed per query on the same subset (0.1341); 0.2 displayed 0.0937 while the index used the per-query value."},{"id":36,"name":"BRIGHT","what":"subset","line":"Same rule as ToolRet: 220 of 550 queries have a relevant document among their 32 candidates. Chance 0.1160 on those queries; 0.2 displayed 0.0448."},{"id":59,"name":"RAGTruth","what":"chance","line":"Chance is the F1 of always answering \"hallucinated\" (0.518), a stronger trivial baseline than a fair coin (0.411)."},{"id":9,"name":"Home appliances","what":"subset","line":"24 duplicate test rows (one copy of each pair kept) and 48 test rows identical to rows of the published dev split are dropped; 88 of 160 rows are scored."}]},"rules":{"evaluation_rules":["Native inference helpers, prompts and readout rules are preserved. Mechanical schema translations and any probability conversion are recorded with the adapter.","No prompt tuning, truncation, option pruning, retraining, or correctness-based retry.","Two requests per benchmark/subtrack form the 86-request compatibility pass. It is not a representative accuracy estimate. A model advances only after that pass completes without runtime errors; supported capacity exclusions stay visible.","Entire linked source groups must be complete and supported before they enter the preliminary successful-case scorer. Missing coverage is reported separately. These successful-case metrics must not be directly averaged into an overall index.","Full records retain original state/questions, native outputs, mapped answers, probabilities, errors and CUDA-synchronized request time. Model loading and toy warmup are separate. Compute timing excludes HTTP/network and is not directly equivalent to Jev HTTPS latency.","Decider's eager and compilation-heavy runs are separate. The initial compiled smoke measured compilation at new shapes, not warmed serving latency.","Native abstentions are retained as terminal abstentions; the wrapper never forces them into a listed answer.","Games require fresh episodes using each model's own decisions under the same seeds, observations, options, partners and budgets. **Interactive reproduction runs have not started; do not reuse Jev trajectories as their scores.**"],"hard_rules":[{"rule":"No truncation","detail":"Every request is sent whole. A model whose context window cannot hold a prompt records that request as unsupported; it is never shortened to fit."},{"rule":"No label filtering","detail":"The full option set of every question is offered. No option pruning, no candidate reduction, no label-set narrowing per model."},{"rule":"No prompt tuning","detail":"Each entrant runs with its own native inference helpers, prompts and readout rules as published. Mechanical schema translations and any probability conversion are recorded with the adapter."},{"rule":"No labels in the adapter","detail":"Gold answers never enter model state; requests.jsonl carries the exact payloads with labels omitted."},{"rule":"No correctness-based retry","detail":"Transient transport errors retry up to twice; a valid wrong answer is never retried."},{"rule":"Native abstentions stand","detail":"A model that abstains keeps its abstention as a terminal answer; the wrapper never forces it into a listed option."},{"rule":"Unanswered = wrong","detail":"Unsupported, refused and errored requests score zero against the full denominator. Nobody climbs by skipping the hard cases."}],"complete_run_gate":{"total_requests":120340,"excluded_questions":442,"scoreable_rows":119898,"rule":"A run enters the index only when its status event is \"complete\" and it has processed at least 119,898 rows (the 120,340-request corpus minus the 442 questions excluded for every engine)."},"superseded_policy":"A model appears once, at the configuration that stands. Lifting a cap the author set outside the model, or replacing a checkpoint, retires the earlier run from the ranking; the results file stays on disk, it just no longer competes.","superseded":[{"retired":"nanojev","replaced_by":"nanojev-8k","why":"moved to the 8k-context checkpoint"},{"retired":"jeff","replaced_by":"jeff-uncapped","why":"author-set output cap lifted"},{"retired":"jevfire","replaced_by":"jevfire-uncapped","why":"author-set output cap lifted"},{"retired":"kev-4b","replaced_by":"kev-4b-raised","why":"label window raised"},{"retired":"kev-9b","replaced_by":"kev-9b-raised","why":"label window raised"},{"retired":"kev-0.8b","replaced_by":"kev-0.8b-raised","why":"label window raised"},{"retired":"pngwn-space-uncapped","replaced_by":"pngwn-space-uncapped-full","why":"full run replaced the earlier partial run"},{"retired":"joshua-diffusion","replaced_by":"joshua-diffusion-full","why":"full run replaced the earlier partial run"},{"retired":"decision-nox","replaced_by":"decision-nox-full","why":"full run replaced the earlier partial run"},{"retired":"solomon-v11-bf16","replaced_by":"solomon-v11-bf16-encoding","why":"encoding path replaced the earlier serving path"},{"retired":"razorback","replaced_by":"razorback-one-read","why":"same build serving the same weights as the one-read path; one row per project"},{"retired":"laptop-pcd","replaced_by":"harsha","why":"same decoding harness over the same stock Qwen2.5-1.5B-Instruct as Qwen-2.5-1B-RLCD, identical scores; one row"},{"retired":"decision-kai","replaced_by":"decision-kai-7185f514","why":"newer Decision 1.0 Kai revision 7185f514"},{"retired":"decision-sol","replaced_by":"decision-sol-0665a411","why":"newer Decision 1.0 Sol revision 0665a411"},{"retired":"decision-nox-full","replaced_by":"decision-nox-0bb83350","why":"newer Decision 1.0 Nox revision 0bb83350"},{"retired":"this-that-1.0","replaced_by":"this-that-1.2","why":"newer this-that release"},{"retired":"nimble","replaced_by":"nimble-v2","why":"newer Nimble release (v2)"},{"retired":"razorback-one-read","replaced_by":"razorback-one-read-3e296f08","why":"newer openjev build 0.4.0 @3e296f08"},{"retired":"laya","replaced_by":"laya-1e28ac20","why":"newer laya code revision 1e28ac20 (same weights)"},{"retired":"verdict","replaced_by":"verdict-8af2496e","why":"newer Verdict weights 8af2496e and code 30f15564"},{"retired":"rune-26b-a4b","replaced_by":"rune-26b-a4b-v3","why":"author-submitted v3 checkpoint (kit PR #9), adopted after a 3,000-row reproduction matched it bit for bit; v1 stays on the 0.2 board"},{"retired":"reflex-27b","replaced_by":"reflex-27b-v2","why":"author-submitted full-coverage rerun with wide-choice one-token labels (kit PR #13), adopted after our latency-v1 run agreed with it within reflex's own run-to-run noise; v1 stays on the 0.2 board"},{"retired":"hopper","replaced_by":"hopper-g-1.2","why":"Hopper (G) 1.2, the author-submitted run of the next Hopper adapter (same Qwen3.5-4B base, continued from Hopper 1.0's adapter, same 1.1.1 serving code and map; kit PR #17), adopted after our latency-v1 run agreed with it within the A10G-vs-RTX noise; 1.1.1 stays on the 0.2 board"}],"withdrawn":[{"engine":"kev-0.5b-raised","why":"Withdrawn at the author's request: an older Kev generation (Qwen2.5-0.5B base); the released family is Qwen3.5 0.8B / 4B / 9B.","source":"https://huggingface.co/spaces/multimodalart/jev-decision-index/discussions/10"},{"engine":"kev-0.6b-raised","why":"Withdrawn at the author's request: an older Kev generation (Qwen3-0.6B-Base); the released family is Qwen3.5 0.8B / 4B / 9B.","source":"https://huggingface.co/spaces/multimodalart/jev-decision-index/discussions/10"},{"engine":"kev-8b-raised","why":"Withdrawn at the author's request: an older Kev generation (Qwen3-8B-Base); the released family is Qwen3.5 0.8B / 4B / 9B.","source":"https://huggingface.co/spaces/multimodalart/jev-decision-index/discussions/10"}],"jev_reference_notes":["Metrics exclude failed/incomplete cases; coverage is explicit. No silent replacement of failures.","Different primary metrics are not averaged across benchmarks.","Latency includes HTTPS and concurrent service scheduling; not isolated model inference.","Full raw HTTP responses/payloads and all retry timings are in http-attempts.jsonl.","BPoMP/RouterBench/GSM8K and chunked tasks retain all linked requests for sampled cases."],"interactive":"Games require fresh episodes using each model's own decisions under the same seeds, observations, options, partners and budgets. Interactive reproduction runs have not started; Jev trajectories are never reused as their scores.","removed":[{"engine":"nanojev-8k","why":"NanoJev retired; removed from the board."},{"engine":"reflex-4b","why":"Removed: the author's recommended configuration is the frozen Qwen3.5-4B base without the published LoRA adapter, and that adapter was trained on 800 MMLU-Pro test items. reflex 27B stays."},{"engine":"jqv-targeted-v2","why":"Parked; not run for 0.2."},{"engine":"bev","why":"Cancelled; not run."},{"engine":"jqv","why":"Left out of 0.2: the new index benchmarks were not run for it; its results stay on disk."}]},"entrants":{"principle":"Nothing is inferred from a name. The kind comes from what the run recorded it loaded, the base comes from the Hub card's own base_model field or from the run's recorded runtime model, and anything that resolves to neither is left null rather than guessed.","kinds":[{"kind":"inference technique","rule":"The entrant published no weights of its own: it serves a stock checkpoint from a base-model vendor namespace (Qwen, Google, LiquidAI, Meta, Mistral, Microsoft, NVIDIA, DeepSeek, AI2, HuggingFaceTB, Answer.AI, OpenAI, IBM Granite, TII) through its own decoding or scoring code."},{"kind":"full fine-tune","rule":"The entrant's own repo ships whole model weights (.safetensors, .bin, .pt, .pth, .gguf or .npz) with no adapter files."},{"kind":"LoRA","rule":"The entrant's files include adapter_config.json or adapter_model.* and no separate head, scorer or pointer weights."},{"kind":"LoRA + head","rule":"Adapter files plus a head, scorer or pointer module."},{"kind":"head / adapter","rule":"A head, scorer or pointer module on a frozen backbone the entrant did not train; classed as full fine-tune when no base model is recorded."},{"kind":"fine-tune","rule":"Own repo with weights the file list could not classify further."}],"weights_in_tree":"A GitHub entrant that carries trained weights inside its own checkout (for example a scorer adapter) counts as trained, not as a technique over the stock base.","techniques":[{"technique":"autoregressive","rule":"The executing model class ends in ForCausalLM or ForConditionalGeneration, read from the config of the entrant's own checkpoint or, failing that, of the backbone it rides on."},{"technique":"diffusion","rule":"The model_type or architecture class contains \"diffusion\", however the model is served."},{"technique":"encoder","rule":"A masked-LM or BERT-family class."},{"technique":"GLi family","rule":"GLiClass, GLiNER2 (BoundaryExtractor / extractor) and GLiFormer classes, bucketed together for the badge; the exact published class is kept alongside."}],"parameter_count_precedence":["Backbone parameter count from the Hub safetensors metadata of the base model (a LoRA or a head reports the backbone it rides on, which is what it costs to run).","Otherwise the entrant's own repo's safetensors metadata total.","Otherwise an exact count from each safetensors header (shapes read over a range request, no tensor download).","Otherwise, for .bin-only repos, weight bytes divided by the declared torch_dtype width, flagged params_estimated.","Otherwise the size stated in the model name (9B, 350M, 35B-A3B gives the total), flagged params_from_name.","trained_bytes is the entrant's own published weight bytes, which is what separates a 9B full fine-tune from a 9B backbone carrying a few megabytes of adapter."],"base_model_rule":"base_model is followed from the Hub card until the owner changes (a card often points at a sibling: an NVFP4 quantization declares the unquantized copy; Decision Lex declares Kai), so the row says what the weights descend from. A base under a vendor namespace that is itself a quantization is resolved two more hops to the original model. The chain is kept for the record.","naming":"Each row uses what the project calls itself in its own README or model card, with the author in parentheses where the name alone is ambiguous (five separate entrants are some spelling of \"openjev\"). variant names a serving choice made here, never part of the published name.","counts":{"kind":{"LoRA + head":8,"full fine-tune":24,"inference technique":15,"head / adapter":9,"LoRA":14},"technique":{"autoregressive":55,"encoder":5,"diffusion":4,"GLi family":6},"entrants":70},"entrants":[{"engine":"akash-gemma","name":"system-one-gemma","variant":null,"kind":"LoRA + head","technique":"autoregressive","technique_name":"autoregressive","served_params":268098176,"params_from_name":false,"params_estimated":false,"trained_params":null,"trained_bytes":null,"base_model":"google/gemma-3-270m","base_chain":null,"weights_repo":null,"served_checkpoint":null,"model_url":null,"code_url":"https://github.com/akash-kamat/system-one-gemma","unanswered_share":0},{"engine":"autojev-27b","name":"pplx-decider-v1-27b","variant":null,"kind":"full fine-tune","technique":"autoregressive","technique_name":"autoregressive","served_params":27781427952,"params_from_name":false,"params_estimated":false,"trained_params":26085330160,"trained_bytes":52173413928,"base_model":"Qwen/Qwen3.8-27B","base_chain":["Qwen/Qwen3.8-27B"],"weights_repo":"perplexity-ai/pplx-decider-v1-27b","served_checkpoint":null,"model_url":"https://huggingface.co/perplexity-ai/pplx-decider-v1-27b","code_url":"https://huggingface.co/perplexity-ai/pplx-decider-v1-27b/tree/main/source","unanswered_share":0},{"engine":"bosun-v3.1-0.6b","name":"Bosun v3.1 0.6B","variant":null,"kind":"LoRA + head","technique":"autoregressive","technique_name":"autoregressive","served_params":596049920,"params_from_name":false,"params_estimated":false,"trained_params":10616832,"trained_bytes":41470928,"base_model":"Qwen/Qwen3-0.6B-Base","base_chain":["Qwen/Qwen3-0.6B","Qwen/Qwen3-0.6B-Base"],"weights_repo":"Hanno-Labs/bosun-v3.1-0.6b","served_checkpoint":null,"model_url":"https://huggingface.co/Hanno-Labs/bosun-v3.1-0.6b","code_url":"https://github.com/Hanno-Labs/jev-compatible-server","unanswered_share":0},{"engine":"bosun-v3.1-1.7b","name":"Bosun v3.1 1.7B","variant":null,"kind":"LoRA + head","technique":"autoregressive","technique_name":"autoregressive","served_params":1720574976,"params_from_name":false,"params_estimated":false,"trained_params":18481152,"trained_bytes":71879720,"base_model":"Qwen/Qwen3-1.7B-Base","base_chain":["Qwen/Qwen3-1.7B","Qwen/Qwen3-1.7B-Base"],"weights_repo":"Hanno-Labs/bosun-v3.1-1.7b","served_checkpoint":null,"model_url":"https://huggingface.co/Hanno-Labs/bosun-v3.1-1.7b","code_url":"https://github.com/Hanno-Labs/jev-compatible-server","unanswered_share":0},{"engine":"clm-v0.1-8b","name":"CLM-v0.1-8B","variant":null,"kind":"full fine-tune","technique":"autoregressive","technique_name":"autoregressive","served_params":8190735360,"params_from_name":false,"params_estimated":false,"trained_params":null,"trained_bytes":75557149,"base_model":"Qwen/Qwen3-8B-Base","base_chain":["Qwen/Qwen3-8B","Qwen/Qwen3-8B-Base"],"weights_repo":"Contrastive-LM/CLM-v0.1-8B","served_checkpoint":"Contrastive-LM/CLM-v0.1-8B","model_url":"https://huggingface.co/Contrastive-LM/CLM-v0.1-8B","code_url":"https://github.com/Contrastive-LM/CLM","unanswered_share":0},{"engine":"decider-2b-fp8-http","name":"Decider 2B","variant":"FP8","kind":"full fine-tune","technique":"autoregressive","technique_name":"autoregressive","served_params":2274069824,"params_from_name":false,"params_estimated":false,"trained_params":1881825088,"trained_bytes":3763692048,"base_model":"Qwen/Qwen3.5-2B-Base","base_chain":["Qwen/Qwen3.5-2B-Base"],"weights_repo":"Mapika/decider-2b","served_checkpoint":"Mapika/decider-2b","model_url":"https://huggingface.co/Mapika/decider-2b","code_url":null,"unanswered_share":3.32e-05},{"engine":"decider-35b-nvfp4","name":"Decider 35B-A3B","variant":"NVFP4","kind":"full fine-tune","technique":"autoregressive","technique_name":"autoregressive","served_params":35951822704,"params_from_name":false,"params_estimated":false,"trained_params":17851937408,"trained_bytes":21013206328,"base_model":"Qwen/Qwen3.5-35B-A3B-Base","base_chain":["Mapika/decider-35b-a3b","Qwen/Qwen3.5-35B-A3B-Base"],"weights_repo":"Mapika/decider-35b-a3b-nvfp4","served_checkpoint":"Mapika/decider-35b-a3b-nvfp4","model_url":"https://huggingface.co/Mapika/decider-35b-a3b-nvfp4","code_url":null,"unanswered_share":0},{"engine":"decider-4b","name":"Decider 4B","variant":null,"kind":"full fine-tune","technique":"autoregressive","technique_name":"autoregressive","served_params":4659865088,"params_from_name":false,"params_estimated":false,"trained_params":4205751296,"trained_bytes":8411558400,"base_model":"Qwen/Qwen3.5-4B-Base","base_chain":["Qwen/Qwen3.5-4B-Base"],"weights_repo":"Mapika/decider-4b","served_checkpoint":null,"model_url":"https://huggingface.co/Mapika/decider-4b","code_url":"https://github.com/Mapika/decider","unanswered_share":0},{"engine":"decider-chat-gemma4-31b","name":"Decider chat · Gemma-4-31B","variant":null,"kind":"inference technique","technique":"autoregressive","technique_name":"autoregressive","served_params":32682372656,"params_from_name":false,"params_estimated":false,"trained_params":null,"trained_bytes":null,"base_model":"google/gemma-4-31B","base_chain":["google/gemma-4-31B"],"weights_repo":null,"served_checkpoint":"google/gemma-4-31B-it","model_url":null,"code_url":"https://github.com/Mapika/decider","unanswered_share":0},{"engine":"decider-chat-qwen3.6-27b","name":"Decider chat · Qwen3.6-27B","variant":null,"kind":"inference technique","technique":"autoregressive","technique_name":"autoregressive","served_params":27781427952,"params_from_name":false,"params_estimated":false,"trained_params":null,"trained_bytes":null,"base_model":"Qwen/Qwen3.6-27B","base_chain":null,"weights_repo":null,"served_checkpoint":"Qwen/Qwen3.6-27B","model_url":null,"code_url":"https://github.com/Mapika/decider","unanswered_share":0},{"engine":"decision-eos-0.8b","name":"Decision 1.0 Eos","variant":null,"kind":"head / adapter","technique":"autoregressive","technique_name":"autoregressive","served_params":873438784,"params_from_name":false,"params_estimated":false,"trained_params":753446208,"trained_bytes":1509034472,"base_model":"Qwen/Qwen3.5-0.8B-Base","base_chain":["Qwen/Qwen3.5-0.8B","Qwen/Qwen3.5-0.8B-Base"],"weights_repo":"llm-semantic-router/Decision-1.0-Eos-0.8B","served_checkpoint":null,"model_url":"https://huggingface.co/llm-semantic-router/Decision-1.0-Eos-0.8B","code_url":null,"unanswered_share":1.67e-05},{"engine":"decision-kai-7185f514","name":"Decision 1.0 Kai","variant":null,"kind":"head / adapter","technique":"encoder","technique_name":"encoder","served_params":307797035,"params_from_name":false,"params_estimated":true,"trained_params":571909635,"trained_bytes":2287687932,"base_model":"jhu-clsp/mmBERT-base","base_chain":["llm-semantic-router/Vela-1.0-Encoder-307M","jhu-clsp/mmBERT-base"],"weights_repo":"llm-semantic-router/Decision-1.0-Kai","served_checkpoint":null,"model_url":"https://huggingface.co/llm-semantic-router/Decision-1.0-Kai","code_url":null,"unanswered_share":0.0027},{"engine":"decision-lex","name":"Decision 1.0 Lex","variant":null,"kind":"head / adapter","technique":"encoder","technique_name":"encoder","served_params":307797035,"params_from_name":false,"params_estimated":true,"trained_params":571909635,"trained_bytes":2287687932,"base_model":"jhu-clsp/mmBERT-base","base_chain":["llm-semantic-router/Decision-1.0-Kai","llm-semantic-router/Vela-1.0-Encoder-307M","jhu-clsp/mmBERT-base"],"weights_repo":"llm-semantic-router/decision-1.0-lex","served_checkpoint":null,"model_url":"https://huggingface.co/llm-semantic-router/decision-1.0-lex","code_url":null,"unanswered_share":0.0027},{"engine":"decision-lux-9b","name":"Decision 1.0 Lux","variant":null,"kind":"head / adapter","technique":"autoregressive","technique_name":"autoregressive","served_params":9653104368,"params_from_name":false,"params_estimated":false,"trained_params":7940895744,"trained_bytes":15890261552,"base_model":"Qwen/Qwen3.5-9B-Base","base_chain":["Qwen/Qwen3.5-9B","Qwen/Qwen3.5-9B-Base"],"weights_repo":"llm-semantic-router/Decision-1.0-Lux-9B","served_checkpoint":null,"model_url":"https://huggingface.co/llm-semantic-router/Decision-1.0-Lux-9B","code_url":null,"unanswered_share":0},{"engine":"decision-nox-0bb83350","name":"Decision 1.0 Nox","variant":null,"kind":"head / adapter","technique":"autoregressive","technique_name":"autoregressive","served_params":4659865088,"params_from_name":false,"params_estimated":false,"trained_params":4208383488,"trained_bytes":8422078976,"base_model":"Qwen/Qwen3.5-4B-Base","base_chain":["Qwen/Qwen3.5-4B","Qwen/Qwen3.5-4B-Base"],"weights_repo":"llm-semantic-router/Decision-1.0-Nox","served_checkpoint":null,"model_url":"https://huggingface.co/llm-semantic-router/Decision-1.0-Nox","code_url":null,"unanswered_share":0},{"engine":"decision-sol-0665a411","name":"Decision 1.0 Sol","variant":null,"kind":"head / adapter","technique":"autoregressive","technique_name":"autoregressive","served_params":2274069824,"params_from_name":false,"params_estimated":false,"trained_params":1883930944,"trained_bytes":3772109600,"base_model":"Qwen/Qwen3.5-2B-Base","base_chain":["Qwen/Qwen3.5-2B","Qwen/Qwen3.5-2B-Base"],"weights_repo":"llm-semantic-router/Decision-1.0-Sol","served_checkpoint":null,"model_url":"https://huggingface.co/llm-semantic-router/Decision-1.0-Sol","code_url":null,"unanswered_share":0},{"engine":"djev","name":"djev","variant":null,"kind":"inference technique","technique":"diffusion","technique_name":"diffusion","served_params":25823778864,"params_from_name":false,"params_estimated":false,"trained_params":null,"trained_bytes":null,"base_model":"google/diffusiongemma-26B-A4B-it","base_chain":null,"weights_repo":null,"served_checkpoint":"google/diffusiongemma-26B-A4B-it","model_url":null,"code_url":"https://github.com/Davipar/djev-dev","unanswered_share":0.0097},{"engine":"gliner-base","name":"GLiNER 2.5 base","variant":null,"kind":"full fine-tune","technique":"GLi family","technique_name":"GLiNER2","served_params":193581591,"params_from_name":false,"params_estimated":false,"trained_params":193581591,"trained_bytes":774366564,"base_model":null,"base_chain":null,"weights_repo":"fastino/gliner2.5-base-v1","served_checkpoint":null,"model_url":"https://huggingface.co/fastino/gliner2.5-base-v1","code_url":null,"unanswered_share":7.48e-05},{"engine":"gliner-decide","name":"GLiNER2.5-Decide","variant":null,"kind":"full fine-tune","technique":"GLi family","technique_name":"GLiNER2","served_params":486444053,"params_from_name":false,"params_estimated":false,"trained_params":486444053,"trained_bytes":1945828140,"base_model":"fastino/gliner2-large-v1","base_chain":["fastino/gliner2-large-v1"],"weights_repo":"fastino/GLiNER2.5-Decide","served_checkpoint":null,"model_url":"https://huggingface.co/fastino/GLiNER2.5-Decide","code_url":"https://github.com/fastino-ai/GLiNER2","unanswered_share":0},{"engine":"gliner-multi","name":"GLiNER 2.5 multilingual","variant":null,"kind":"full fine-tune","technique":"GLi family","technique_name":"GLiNER2","served_params":287355159,"params_from_name":false,"params_estimated":false,"trained_params":287355159,"trained_bytes":1149461028,"base_model":null,"base_chain":null,"weights_repo":"fastino/gliner2.5-multi-v1","served_checkpoint":null,"model_url":"https://huggingface.co/fastino/gliner2.5-multi-v1","code_url":null,"unanswered_share":7.48e-05},{"engine":"gliner-small","name":"GLiNER 2.5 small","variant":null,"kind":"full fine-tune","technique":"GLi family","technique_name":"GLiNER2","served_params":73881879,"params_from_name":false,"params_estimated":false,"trained_params":73881879,"trained_bytes":295567700,"base_model":null,"base_chain":null,"weights_repo":"fastino/gliner2.5-small-v1","served_checkpoint":null,"model_url":"https://huggingface.co/fastino/gliner2.5-small-v1","code_url":null,"unanswered_share":7.48e-05},{"engine":"harsha","name":"Qwen-2.5-1B-RLCD","variant":null,"kind":"inference technique","technique":"autoregressive","technique_name":"autoregressive","served_params":1543714304,"params_from_name":false,"params_estimated":false,"trained_params":null,"trained_bytes":null,"base_model":"Qwen/Qwen2.5-1.5B","base_chain":["Qwen/Qwen2.5-1.5B"],"weights_repo":null,"served_checkpoint":null,"model_url":null,"code_url":"https://huggingface.co/harshatheg/Qwen-2.5-1B-RLCD","unanswered_share":0.0032},{"engine":"hopper-g-1.2","name":"Hopper (G) 1.2","variant":null,"kind":"LoRA","technique":"autoregressive","technique_name":"autoregressive","served_params":4659865088,"params_from_name":false,"params_estimated":false,"trained_params":32464896,"trained_bytes":129927008,"base_model":"Qwen/Qwen3.5-4B-Base","base_chain":["Qwen/Qwen3.5-4B","Qwen/Qwen3.5-4B-Base"],"weights_repo":"HopitAI/hopper-g","served_checkpoint":null,"model_url":"https://huggingface.co/HopitAI/hopper-g","code_url":"https://github.com/hopit-ai/hopper","unanswered_share":0},{"engine":"jeff-uncapped","name":"jeff","variant":null,"kind":"inference technique","technique":"GLi family","technique_name":"GLiFormer","served_params":575683963,"params_from_name":false,"params_estimated":true,"trained_params":null,"trained_bytes":null,"base_model":"knowledgator/gliformer-large-v1","base_chain":null,"weights_repo":null,"served_checkpoint":"knowledgator/gliformer-large-v1","model_url":null,"code_url":"https://github.com/logan-markewich/jeff","unanswered_share":0},{"engine":"jev-omni","name":"Jev-Omni","variant":null,"kind":"LoRA + head","technique":"autoregressive","technique_name":"autoregressive","served_params":11959730224,"params_from_name":false,"params_estimated":false,"trained_params":11959730224,"trained_bytes":23923515487,"base_model":"google/gemma-4-12B","base_chain":["google/gemma-4-12B-it","google/gemma-4-12B"],"weights_repo":"akhilaaa3/Jev-Omni","served_checkpoint":null,"model_url":"https://huggingface.co/akhilaaa3/Jev-Omni","code_url":null,"unanswered_share":0},{"engine":"jevfire-uncapped","name":"Jevfire","variant":null,"kind":"inference technique","technique":"autoregressive","technique_name":"autoregressive","served_params":27781427952,"params_from_name":false,"params_estimated":false,"trained_params":null,"trained_bytes":null,"base_model":"Qwen/Qwen3.8-27B","base_chain":["Qwen/Qwen3.8-27B"],"weights_repo":null,"served_checkpoint":"Qwen/Qwen3.8-27B-FP8","model_url":null,"code_url":"https://github.com/kikoncuo/jevfire","unanswered_share":0},{"engine":"jevk5","name":"JevK5","variant":null,"kind":"LoRA","technique":"autoregressive","technique_name":"autoregressive","served_params":4659865088,"params_from_name":false,"params_estimated":false,"trained_params":4205751296,"trained_bytes":8411558400,"base_model":"Qwen/Qwen3.5-4B-Base","base_chain":["Qwen/Qwen3.5-4B-Base"],"weights_repo":"alibiserikbay/JevK5","served_checkpoint":null,"model_url":"https://huggingface.co/alibiserikbay/JevK5","code_url":"https://github.com/allebee/jevk5","unanswered_share":0},{"engine":"jobe","name":"Jobe","variant":null,"kind":"inference technique","technique":"autoregressive","technique_name":"autoregressive","served_params":4659865088,"params_from_name":false,"params_estimated":false,"trained_params":null,"trained_bytes":null,"base_model":"Qwen/Qwen3.5-4B-Base","base_chain":["Qwen/Qwen3.5-4B-Base"],"weights_repo":null,"served_checkpoint":"Qwen/Qwen3.5-4B","model_url":null,"code_url":"https://github.com/MantisShrimpdev/jobe","unanswered_share":0},{"engine":"joshua-diffusion-full","name":"open-jev (JoshuaSP)","variant":null,"kind":"inference technique","technique":"diffusion","technique_name":"diffusion","served_params":25823778864,"params_from_name":false,"params_estimated":false,"trained_params":null,"trained_bytes":null,"base_model":"google/diffusiongemma-26B-A4B-it","base_chain":null,"weights_repo":null,"served_checkpoint":"google/diffusiongemma-26B-A4B-it","model_url":null,"code_url":"https://github.com/JoshuaSP/open-jev","unanswered_share":0},{"engine":"kev-0.8b-raised","name":"Kev 0.8B","variant":null,"kind":"LoRA + head","technique":"autoregressive","technique_name":"autoregressive","served_params":873438784,"params_from_name":false,"params_estimated":false,"trained_params":10822656,"trained_bytes":45441727,"base_model":"Qwen/Qwen3.5-0.8B-Base","base_chain":null,"weights_repo":"jaredpalmer/kev-0.8b","served_checkpoint":"jaredpalmer/kev-0.8b","model_url":"https://huggingface.co/jaredpalmer/kev-0.8b","code_url":"https://github.com/jaredpalmer/kev","unanswered_share":0},{"engine":"kev-4b-raised","name":"Kev 4B","variant":null,"kind":"LoRA + head","technique":"autoregressive","technique_name":"autoregressive","served_params":4659865088,"params_from_name":false,"params_estimated":false,"trained_params":32464896,"trained_bytes":135172799,"base_model":"Qwen/Qwen3.5-4B-Base","base_chain":null,"weights_repo":"jaredpalmer/kev-4b","served_checkpoint":"jaredpalmer/kev-4b","model_url":"https://huggingface.co/jaredpalmer/kev-4b","code_url":"https://github.com/jaredpalmer/kev","unanswered_share":0},{"engine":"kev-9b-raised","name":"Kev 9B","variant":null,"kind":"LoRA + head","technique":"autoregressive","technique_name":"autoregressive","served_params":9653104368,"params_from_name":false,"params_estimated":false,"trained_params":43278336,"trained_bytes":181572591,"base_model":"Qwen/Qwen3.5-9B-Base","base_chain":null,"weights_repo":"jaredpalmer/kev-9b","served_checkpoint":"jaredpalmer/kev-9b","model_url":"https://huggingface.co/jaredpalmer/kev-9b","code_url":"https://github.com/jaredpalmer/kev","unanswered_share":0},{"engine":"laya-1e28ac20","name":"Laya","variant":null,"kind":"full fine-tune","technique":"encoder","technique_name":"encoder","served_params":421293830,"params_from_name":false,"params_estimated":false,"trained_params":421293830,"trained_bytes":2329053944,"base_model":null,"base_chain":null,"weights_repo":"convaiinnovations/laya","served_checkpoint":null,"model_url":"https://huggingface.co/convaiinnovations/laya","code_url":"https://github.com/NandhaKishorM/laya","unanswered_share":0.0027},{"engine":"lfm2600","name":"LFM2.5-2.6B-RLCD","variant":null,"kind":"full fine-tune","technique":"autoregressive","technique_name":"autoregressive","served_params":2697198592,"params_from_name":false,"params_estimated":false,"trained_params":2697198592,"trained_bytes":5394427456,"base_model":"LiquidAI/LFM2.5-2.6B-Base","base_chain":["LiquidAI/LFM2.5-2.6B","LiquidAI/LFM2.5-2.6B-Base"],"weights_repo":"monotykamary/LFM2.5-2.6B-RLCD","served_checkpoint":null,"model_url":"https://huggingface.co/monotykamary/LFM2.5-2.6B-RLCD","code_url":null,"unanswered_share":0.0044},{"engine":"lfm350-sdpa","name":"LFM2.5-350M-RLCD","variant":null,"kind":"full fine-tune","technique":"autoregressive","technique_name":"autoregressive","served_params":354483968,"params_from_name":false,"params_estimated":false,"trained_params":354483968,"trained_bytes":708984464,"base_model":null,"base_chain":null,"weights_repo":"notnotsamuel/LFM2.5-350M-RLCD","served_checkpoint":null,"model_url":"https://huggingface.co/notnotsamuel/LFM2.5-350M-RLCD","code_url":null,"unanswered_share":0},{"engine":"lumma-fev-0.1b","name":"Lumma-Fev-0.1B","variant":null,"kind":"full fine-tune","technique":"autoregressive","technique_name":"autoregressive","served_params":153412928,"params_from_name":false,"params_estimated":false,"trained_params":154102848,"trained_bytes":309928008,"base_model":"FrontiersMind/Nandi-Mini-150M","base_chain":["FrontiersMind/Nandi-Mini-150M"],"weights_repo":"FrontiersMind/Lumma-fev-0.1b","served_checkpoint":null,"model_url":"https://huggingface.co/FrontiersMind/Lumma-fev-0.1b","code_url":null,"unanswered_share":0.0175},{"engine":"lumma-fev-0.6b","name":"Lumma-Fev-0.6B","variant":null,"kind":"LoRA","technique":"autoregressive","technique_name":"autoregressive","served_params":649281964,"params_from_name":false,"params_estimated":false,"trained_params":649282476,"trained_bytes":1300071072,"base_model":"FrontiersMind/Lumma-0.6B-Base","base_chain":["FrontiersMind/Lumma-0.6B-Base"],"weights_repo":"FrontiersMind/Lumma-fev-0.6b","served_checkpoint":null,"model_url":"https://huggingface.co/FrontiersMind/Lumma-fev-0.6b","code_url":null,"unanswered_share":3.34e-05},{"engine":"metask-jev-4b","name":"Metask-Jev-4B","variant":null,"kind":"LoRA","technique":"autoregressive","technique_name":"autoregressive","served_params":4659865088,"params_from_name":false,"params_estimated":false,"trained_params":4539265536,"trained_bytes":9078620536,"base_model":"Qwen/Qwen3.5-4B-Base","base_chain":["Qwen/Qwen3.5-4B-Base"],"weights_repo":"wayfind/metask-jev-4b-policy-mix","served_checkpoint":null,"model_url":"https://huggingface.co/wayfind/metask-jev-4b-policy-mix","code_url":"https://github.com/metask-ai/metask-jev","unanswered_share":0.1205},{"engine":"mini-jev","name":"mini-jev","variant":null,"kind":"inference technique","technique":"autoregressive","technique_name":"autoregressive","served_params":4022468096,"params_from_name":false,"params_estimated":false,"trained_params":null,"trained_bytes":null,"base_model":"Qwen/Qwen3-4B-Instruct-2507","base_chain":null,"weights_repo":null,"served_checkpoint":"Qwen/Qwen3-4B-Instruct-2507","model_url":null,"code_url":"https://github.com/r-ms/mini-jev","unanswered_share":0.1205},{"engine":"mojev","name":"MoJev","variant":null,"kind":"head / adapter","technique":"autoregressive","technique_name":"autoregressive","served_params":873438784,"params_from_name":false,"params_estimated":false,"trained_params":854036544,"trained_bytes":1965507264,"base_model":"Qwen/Qwen3.5-0.8B-Base","base_chain":["Qwen/Qwen3.5-0.8B","Qwen/Qwen3.5-0.8B-Base"],"weights_repo":"MoLeMo-Lab/mojev","served_checkpoint":null,"model_url":"https://huggingface.co/MoLeMo-Lab/mojev","code_url":"https://github.com/MoLeMo-Lab/mojev","unanswered_share":0},{"engine":"nimble-v2","name":"Bespoke Nimble 9B v2","variant":null,"kind":"LoRA","technique":"autoregressive","technique_name":"autoregressive","served_params":9653104368,"params_from_name":false,"params_estimated":false,"trained_params":43278336,"trained_bytes":173188512,"base_model":"Qwen/Qwen3.5-9B-Base","base_chain":["Qwen/Qwen3.5-9B","Qwen/Qwen3.5-9B-Base"],"weights_repo":"bespokelabs/Bespoke-Nimble-9B-v2","served_checkpoint":null,"model_url":"https://huggingface.co/bespokelabs/Bespoke-Nimble-9B-v2","code_url":"https://github.com/bespokelabsai/nimble","unanswered_share":0},{"engine":"openvons","name":"openvons","variant":null,"kind":"inference technique","technique":"autoregressive","technique_name":"autoregressive","served_params":4022468096,"params_from_name":false,"params_estimated":false,"trained_params":null,"trained_bytes":null,"base_model":"Qwen/Qwen3-4B-Instruct-2507","base_chain":null,"weights_repo":null,"served_checkpoint":"Qwen/Qwen3-4B-Instruct-2507","model_url":null,"code_url":"https://github.com/genai-craft/openvons","unanswered_share":0},{"engine":"pngwn-space-uncapped-full","name":"open-jev (pngwn)","variant":null,"kind":"LoRA","technique":"autoregressive","technique_name":"autoregressive","served_params":4659865088,"params_from_name":false,"params_estimated":false,"trained_params":30476800,"trained_bytes":121956232,"base_model":"Qwen/Qwen3.5-4B-Base","base_chain":null,"weights_repo":"pngwn/system-one-qwen3.5-4b-scorer-v2b","served_checkpoint":null,"model_url":"https://huggingface.co/pngwn/system-one-qwen3.5-4b-scorer-v2b","code_url":null,"unanswered_share":0},{"engine":"razorback-one-read-3e296f08","name":"openjev (razorback16)","variant":"one read, NVFP4","kind":"inference technique","technique":"diffusion","technique_name":"diffusion","served_params":25823778864,"params_from_name":false,"params_estimated":false,"trained_params":null,"trained_bytes":null,"base_model":"google/diffusiongemma-26B-A4B-it","base_chain":["google/diffusiongemma-26B-A4B-it"],"weights_repo":null,"served_checkpoint":"nvidia/diffusiongemma-26B-A4B-it-NVFP4","model_url":null,"code_url":"https://github.com/vllm-project/vllm/pull/57250","unanswered_share":0},{"engine":"reflex-27b-v2","name":"reflex 27B","variant":"FP8 · wide choice","kind":"inference technique","technique":"autoregressive","technique_name":"autoregressive","served_params":27781427952,"params_from_name":false,"params_estimated":false,"trained_params":null,"trained_bytes":null,"base_model":"Qwen/Qwen3.8-27B","base_chain":["Qwen/Qwen3.8-27B"],"weights_repo":null,"served_checkpoint":"Qwen/Qwen3.8-27B-FP8","model_url":null,"code_url":"https://github.com/kshetrajna12/reflex","unanswered_share":0},{"engine":"rune-26b-a4b-v3","name":"Surogate Rune 26B-A4B v3","variant":"bf16","kind":"full fine-tune","technique":"autoregressive","technique_name":"autoregressive","served_params":25805936206,"params_from_name":false,"params_estimated":false,"trained_params":25805936206,"trained_bytes":51612008580,"base_model":"google/gemma-4-26B-A4B-it","base_chain":null,"weights_repo":"surogate/rune-26b-a4b-GGUF","served_checkpoint":"bd4a7cbbed66af1a95dfbac12cabf52e6e677411","model_url":"https://huggingface.co/surogate/rune-26b-a4b-GGUF","code_url":"https://github.com/invergent-ai/surogate","unanswered_share":8.34e-06},{"engine":"semif","name":"SemIf","variant":null,"kind":"inference technique","technique":"autoregressive","technique_name":"autoregressive","served_params":4659865088,"params_from_name":false,"params_estimated":false,"trained_params":null,"trained_bytes":null,"base_model":"Qwen/Qwen3.5-4B-Base","base_chain":["Qwen/Qwen3.5-4B-Base"],"weights_repo":null,"served_checkpoint":null,"model_url":null,"code_url":"https://github.com/TheoLeeCJ/openjev","unanswered_share":0.1273},{"engine":"solomon-v11-bf16-encoding","name":"Solomon v1.1","variant":null,"kind":"head / adapter","technique":"autoregressive","technique_name":"autoregressive","served_params":27781427952,"params_from_name":false,"params_estimated":false,"trained_params":225444865,"trained_bytes":903468342,"base_model":"Qwen/Qwen3.8-27B","base_chain":null,"weights_repo":"DoccyHealth/Solomon","served_checkpoint":null,"model_url":"https://huggingface.co/DoccyHealth/Solomon","code_url":null,"unanswered_share":0.2285},{"engine":"tev1-0.8b","name":"Tev1-0.8B-experimental","variant":null,"kind":"full fine-tune","technique":"autoregressive","technique_name":"autoregressive","served_params":873438784,"params_from_name":false,"params_estimated":false,"trained_params":873438784,"trained_bytes":1746942600,"base_model":"Qwen/Qwen3.5-0.8B-Base","base_chain":["Qwen/Qwen3.5-0.8B","Qwen/Qwen3.5-0.8B-Base"],"weights_repo":"togethercomputer/Tev1-0.8B-experimental","served_checkpoint":null,"model_url":"https://huggingface.co/togethercomputer/Tev1-0.8B-experimental","code_url":"https://github.com/togethercomputer/tev1","unanswered_share":0.1229},{"engine":"tev1-4b","name":"Tev1-4B-experimental","variant":null,"kind":"full fine-tune","technique":"autoregressive","technique_name":"autoregressive","served_params":4659865088,"params_from_name":false,"params_estimated":false,"trained_params":4659865088,"trained_bytes":9319828096,"base_model":"Qwen/Qwen3.5-4B-Base","base_chain":["Qwen/Qwen3.5-4B","Qwen/Qwen3.5-4B-Base"],"weights_repo":"togethercomputer/Tev1-4B-experimental","served_checkpoint":null,"model_url":"https://huggingface.co/togethercomputer/Tev1-4B-experimental","code_url":"https://github.com/togethercomputer/tev1","unanswered_share":0.1229},{"engine":"this-that-1.2","name":"this-that 1.2","variant":null,"kind":"full fine-tune","technique":"autoregressive","technique_name":"autoregressive","served_params":1881825088,"params_from_name":false,"params_estimated":false,"trained_params":1881825088,"trained_bytes":3763692048,"base_model":"Mapika/decider-2b","base_chain":null,"weights_repo":"flock-io/this-that-model-1.2","served_checkpoint":null,"model_url":"https://huggingface.co/flock-io/this-that-model-1.2","code_url":"https://github.com/FLock-io/this-that-model","unanswered_share":0},{"engine":"verdict-8af2496e","name":"Verdict","variant":null,"kind":"full fine-tune","technique":"GLi family","technique_name":"GLiClass","served_params":151378176,"params_from_name":false,"params_estimated":false,"trained_params":151378176,"trained_bytes":605529340,"base_model":"knowledgator/gliclass-modern-base-v2.0","base_chain":["knowledgator/gliclass-modern-base-v2.0"],"weights_repo":"heman10x/rlcd-modernbert-151m","served_checkpoint":null,"model_url":"https://huggingface.co/heman10x/rlcd-modernbert-151m","code_url":"https://github.com/Heman10x-NGU/Verdict-open-jev","unanswered_share":0.5908},{"engine":"vllm-pr57250","name":"mmastrac diffusiongemma (vLLM PR 57250)","variant":null,"kind":"inference technique","technique":"diffusion","technique_name":"diffusion","served_params":25823778864,"params_from_name":false,"params_estimated":false,"trained_params":null,"trained_bytes":null,"base_model":"google/diffusiongemma-26B-A4B-it","base_chain":null,"weights_repo":null,"served_checkpoint":"google/diffusiongemma-26B-A4B-it","model_url":null,"code_url":"https://github.com/vllm-project/vllm/pull/57250","unanswered_share":0.1207},{"engine":"winnow-12b-q8","name":"Winnow-12B","variant":"Q8_0","kind":"LoRA","technique":"autoregressive","technique_name":"autoregressive","served_params":11959730224,"params_from_name":false,"params_estimated":false,"trained_params":11907350576,"trained_bytes":12669646592,"base_model":"google/gemma-4-12B","base_chain":["google/gemma-4-12B-it","google/gemma-4-12B"],"weights_repo":"EldanRing/Winnow-12B","served_checkpoint":"EldanRing/Winnow-12B","model_url":"https://huggingface.co/EldanRing/Winnow-12B","code_url":"https://github.com/EldanRing/winnow-inference","unanswered_share":0},{"engine":"winnow-e4b-q8","name":"Winnow-E4B","variant":"Q8_0","kind":"LoRA","technique":"autoregressive","technique_name":"autoregressive","served_params":7996156490,"params_from_name":false,"params_estimated":false,"trained_params":7518069290,"trained_bytes":8005437472,"base_model":"google/gemma-4-E4B","base_chain":["google/gemma-4-E4B-it","google/gemma-4-E4B"],"weights_repo":"EldanRing/Winnow-E4B","served_checkpoint":"EldanRing/Winnow-E4B","model_url":"https://huggingface.co/EldanRing/Winnow-E4B","code_url":"https://github.com/EldanRing/winnow-inference","unanswered_share":0},{"engine":"xor","name":"Xor","variant":null,"kind":"full fine-tune","technique":"autoregressive","technique_name":"autoregressive","served_params":35951822704,"params_from_name":false,"params_estimated":false,"trained_params":35107181936,"trained_bytes":70214495944,"base_model":"Qwen/Qwen3.6-35B-A3B","base_chain":["Qwen/Qwen3.6-35B-A3B"],"weights_repo":"juspay/xor","served_checkpoint":null,"model_url":"https://huggingface.co/juspay/xor","code_url":null,"unanswered_share":0.1209},{"engine":"lev","name":"lev","variant":null,"kind":"LoRA + head","technique":"autoregressive","technique_name":"autoregressive","served_params":4659865088,"params_from_name":false,"params_estimated":false,"trained_params":42467328,"trained_bytes":184601885,"base_model":"Qwen/Qwen3.5-4B-Base","base_chain":["Qwen/Qwen3.5-4B-Base"],"weights_repo":"interfaze-ai/lev","served_checkpoint":null,"model_url":"https://huggingface.co/interfaze-ai/lev","code_url":"https://github.com/Abhinavexists/lev","unanswered_share":0},{"engine":"intern-decision-0.8b","name":"Intern-Decision-0.8B","variant":null,"kind":"full fine-tune","technique":"autoregressive","technique_name":"autoregressive","served_params":873438784,"params_from_name":false,"params_estimated":false,"trained_params":852985920,"trained_bytes":1706034736,"base_model":"Qwen/Qwen3.5-0.8B-Base","base_chain":["Qwen/Qwen3.5-0.8B","Qwen/Qwen3.5-0.8B-Base"],"weights_repo":"internlm/Intern-Decision-0.8B","served_checkpoint":null,"model_url":"https://huggingface.co/internlm/Intern-Decision-0.8B","code_url":"https://github.com/internlm/Intern-Decision","unanswered_share":0},{"engine":"intern-decision-2b","name":"Intern-Decision-2B","variant":null,"kind":"full fine-tune","technique":"autoregressive","technique_name":"autoregressive","served_params":2274069824,"params_from_name":false,"params_estimated":false,"trained_params":2213241664,"trained_bytes":4426562832,"base_model":"Qwen/Qwen3.5-2B-Base","base_chain":["Qwen/Qwen3.5-2B","Qwen/Qwen3.5-2B-Base"],"weights_repo":"internlm/Intern-Decision-2B","served_checkpoint":null,"model_url":"https://huggingface.co/internlm/Intern-Decision-2B","code_url":"https://github.com/internlm/Intern-Decision","unanswered_share":0},{"engine":"intern-decision-4b","name":"Intern-Decision-4B","variant":null,"kind":"full fine-tune","technique":"autoregressive","technique_name":"autoregressive","served_params":4659865088,"params_from_name":false,"params_estimated":false,"trained_params":4539265536,"trained_bytes":9078626736,"base_model":"Qwen/Qwen3.5-4B-Base","base_chain":["Qwen/Qwen3.5-4B","Qwen/Qwen3.5-4B-Base"],"weights_repo":"internlm/Intern-Decision-4B","served_checkpoint":null,"model_url":"https://huggingface.co/internlm/Intern-Decision-4B","code_url":"https://github.com/internlm/Intern-Decision","unanswered_share":0},{"engine":"simple-jev-qwen3.8-27b","name":"simple-jev · Qwen3.8-27B (featherless)","variant":null,"kind":"inference technique","technique":"autoregressive","technique_name":"autoregressive","served_params":27781427952,"params_from_name":false,"params_estimated":false,"trained_params":null,"trained_bytes":null,"base_model":"Qwen/Qwen3.8-27B","base_chain":null,"weights_repo":null,"served_checkpoint":"Qwen/Qwen3.8-27B","model_url":null,"code_url":"https://github.com/featherless-ai/simple-jev","unanswered_share":0},{"engine":"neohorse-jev-4b","name":"NeoHorse-Jev-4B","variant":null,"kind":"head / adapter","technique":"autoregressive","technique_name":"autoregressive","served_params":4659865088,"params_from_name":false,"params_estimated":false,"trained_params":4540576768,"trained_bytes":9083860816,"base_model":"Qwen/Qwen3.5-4B-Base","base_chain":["TokenRhythm/NeoHorse-1-4B","Qwen/Qwen3.5-4B","Qwen/Qwen3.5-4B-Base"],"weights_repo":"TokenRhythm/NeoHorse-Jev-4B","served_checkpoint":null,"model_url":"https://huggingface.co/TokenRhythm/NeoHorse-Jev-4B","code_url":"https://github.com/TokenRhythm/NeoHorse/tree/main/jev","unanswered_share":0},{"engine":"julia-1","name":"Julia 1","variant":null,"kind":"full fine-tune","technique":"encoder","technique_name":"encoder","served_params":140908331,"params_from_name":false,"params_estimated":true,"trained_params":144292870,"trained_bytes":577189056,"base_model":"jhu-clsp/mmBERT-small","base_chain":["jhu-clsp/mmBERT-small"],"weights_repo":"SupersonicLabs/Julia-1","served_checkpoint":null,"model_url":"https://huggingface.co/SupersonicLabs/Julia-1","code_url":null,"unanswered_share":8.34e-05},{"engine":"jebadiah-27b","name":"Jebadiah 27B","variant":null,"kind":"LoRA","technique":"autoregressive","technique_name":"autoregressive","served_params":27781427952,"params_from_name":false,"params_estimated":false,"trained_params":116727808,"trained_bytes":null,"base_model":"Qwen/Qwen3.8-27B","base_chain":null,"weights_repo":"frontier-infra/jebadiah-27b","served_checkpoint":"1c0d794fc24b24b15175c6a912654fa5cd76e1bb","model_url":"https://huggingface.co/frontier-infra/jebadiah-27b","code_url":"https://github.com/getainode/jebadiah","unanswered_share":0},{"engine":"lavoir","name":"Lavoir","variant":null,"kind":"full fine-tune","technique":"encoder","technique_name":"encoder","served_params":395881664,"params_from_name":false,"params_estimated":false,"trained_params":421562382,"trained_bytes":1686272004,"base_model":"answerdotai/ModernBERT-large","base_chain":null,"weights_repo":"moganai/lavoir","served_checkpoint":"4c5eaeb99b2368f1416b9893c7aacc58235aa09f","model_url":"https://huggingface.co/moganai/lavoir","code_url":"https://github.com/moganai/lavoir","unanswered_share":0.0027},{"engine":"jpt-0.8b","name":"JPT-0.8B","variant":null,"kind":"LoRA","technique":"autoregressive","technique_name":"autoregressive","served_params":873438784,"params_from_name":false,"params_estimated":false,"trained_params":10822656,"trained_bytes":1706035272,"base_model":"Qwen/Qwen3.5-0.8B-Base","base_chain":["Qwen/Qwen3.5-0.8B","Qwen/Qwen3.5-0.8B-Base"],"weights_repo":"kirp/jpt-0.8b","served_checkpoint":"5921ae03fbaff98fcbe781c1ff3f1200dc897fd1","model_url":"https://huggingface.co/kirp/jpt-0.8b","code_url":"https://github.com/tic-top/llm2jev","unanswered_share":0},{"engine":"jpt-4b","name":"JPT-4B","variant":null,"kind":"LoRA","technique":"autoregressive","technique_name":"autoregressive","served_params":4659865088,"params_from_name":false,"params_estimated":false,"trained_params":32464896,"trained_bytes":9078627496,"base_model":"Qwen/Qwen3.5-4B-Base","base_chain":["Qwen/Qwen3.5-4B","Qwen/Qwen3.5-4B-Base"],"weights_repo":"kirp/jpt-4b","served_checkpoint":"277ae37b7a6398a8ec3b86712ac16c1991ab6ef1","model_url":"https://huggingface.co/kirp/jpt-4b","code_url":"https://github.com/tic-top/llm2jev","unanswered_share":0},{"engine":"jpt-9b","name":"JPT-9B","variant":null,"kind":"LoRA","technique":"autoregressive","technique_name":"autoregressive","served_params":9653104368,"params_from_name":false,"params_estimated":false,"trained_params":43278336,"trained_bytes":18819728184,"base_model":"Qwen/Qwen3.5-9B-Base","base_chain":["Qwen/Qwen3.5-9B","Qwen/Qwen3.5-9B-Base"],"weights_repo":"kirp/jpt-9b","served_checkpoint":"7114b0c3d9bea6b82dfa2d0691e8d5562cd26d4e","model_url":"https://huggingface.co/kirp/jpt-9b","code_url":"https://github.com/tic-top/llm2jev","unanswered_share":0},{"engine":"eikos-27b-fp8","name":"Eikos-27B","variant":"FP8","kind":"LoRA","technique":"autoregressive","technique_name":"autoregressive","served_params":27781427952,"params_from_name":false,"params_estimated":false,"trained_params":27781427952,"trained_bytes":31220328864,"base_model":"Qwen/Qwen3.8-27B","base_chain":null,"weights_repo":"caiovicentino1/Eikos-27B-FP8","served_checkpoint":null,"model_url":"https://huggingface.co/caiovicentino1/Eikos-27B-FP8","code_url":"https://github.com/caiovicentino/eikos","unanswered_share":0},{"engine":"jet-v6.2","name":"Jet v6.2","variant":null,"kind":"LoRA","technique":"autoregressive","technique_name":"autoregressive","served_params":4659865088,"params_from_name":false,"params_estimated":false,"trained_params":4205751296,"trained_bytes":8411558448,"base_model":"Qwen/Qwen3.5-4B-Base","base_chain":["Qwen/Qwen3.5-4B","Qwen/Qwen3.5-4B-Base"],"weights_repo":"michaljach/jet","served_checkpoint":null,"model_url":"https://huggingface.co/michaljach/jet","code_url":"https://github.com/michaljach/jet","unanswered_share":0.002}]},"latency":{"reproductions":{"summary":"Latency is measured on its own, after scoring: every entrant runs the same 750 rows (stratified by benchmark and prompt length across the 0.2 scope, plus 10 untimed warm-up rows) as one process sending one request at a time on 1 x NVIDIA RTX PRO 6000, using the fastest inference path the model or its code supports. The figure is the wall time of one row, request in to all answers out, measured the same way for in-process and HTTP engines; board medians and p95s come from this run, not from the scoring runs.","aggregate":"Median and p95 over the timed rows that returned an answer; per-benchmark medians are shown only where the sample holds at least 5 rows of that benchmark. A bootstrap on the full single-process runs of ten entrants put the sample median within 5% and the p95 within 10% of a full run at 95% confidence. Earlier board figures came from the scoring runs, some timed with 6 to 12 processes sharing a card or with queue waits removed; they are no longer shown.","protocol":{"protocol":"latency-v1","frozen_utc":"2026-09-26","seed":20260926,"sample":{"rows":"selected-rows.jsonl","sha256":"d6666eb50235b0cd77a5b07391b7ac551dea5a3e89f2802ccf92620f41a63235","timed":750,"warmup":10,"design":"benchmark+length: systematic sample (random start, seed 20260926) over the 151,024-row live scope sorted by benchmark then payload length, so each benchmark gets rows in proportion to its share of the scope and each benchmark is spread across its length range","scope":"0.2 live scope: release-v2 rows (ACOS subset) minus SimpleBench and excluded questions, plus the 7 added catalogs","per_catalog":{"1":9,"2":5,"3":2,"4":16,"5":27,"6":50,"9":0,"10":13,"11":0,"12":16,"20":25,"21":13,"22":10,"23":25,"24":70,"25":1,"26":11,"27":6,"28":7,"29":49,"30":13,"31":25,"32":4,"33":8,"36":3,"37":25,"38":8,"39":4,"40":23,"41":15,"42":28,"43":2,"44":25,"45":3,"48":50,"50":8,"56":10,"57":60,"58":27,"59":14,"61":20,"62":18,"64":2},"refusals":"every entrant runs the same 760 rows; rows it refuses on its board run are recorded but not timed"},"load":"one process, one request at a time, no concurrency, on exactly one RTX PRO 6000 (96 GB); no multi-GPU entrants","path":"fastest inference path the model or its code supports (CUDA graphs, fused/fla kernels, flash attention, compiled paths, vLLM/SGLang, prefix/KV reuse, batching questions inside a row); rows that do not fit the fastest path fall back to the next-fastest and are recorded; the path is listed per entrant","timing":"end-to-end wall time of one row (request in, all answers out), total_wall_ms from run.py --suite latency-v1 (CUDA-synchronized in-process, or one sequential HTTP client against the author server); job/server startup and model load excluded","report":"median and p95 over the 750 timed rows, plus per-benchmark medians (catalogs with fewer than 5 sampled rows are shown as too few)","parity":"answers compared with the entrant board rows: choice flips and max |dp|; informational only, a faster path is not rejected for drift","jev":"network API latency with one sequential client if the jev-release-v1 credentials still work; otherwise the old figure, flagged","evidence":{"method":"bootstrap (B=4000) of the design at each n on per-row wall times from full runs that were single-process (one runner per card or per HF Job; in-file row concurrency 1, AutoJev one shard per card); error = 95th percentile of |sample stat / full-run stat - 1|; criterion median within 5% and p95 within 10% for every probe","probes":["lfm350-sdpa","laya-1e28ac20","decision-eos-0.8b","harsha","this-that-1.2","mini-jev","kev-4b-raised","winnow-12b-q8","openvons","pplx-decider-v1-27b"],"reference":{"lfm350-sdpa":{"median":25.9357,"p95":172.8058,"rows":151024},"laya-1e28ac20":{"median":18.2852,"p95":49.7916,"rows":151007},"decision-eos-0.8b":{"median":36.0447,"p95":76.2549,"rows":151021},"harsha":{"median":41.8783,"p95":73.3379,"rows":151024},"this-that-1.2":{"median":38.5695,"p95":64.3355,"rows":151024},"mini-jev":{"median":63.8445,"p95":162.631,"rows":136524},"kev-4b-raised":{"median":52.1934,"p95":144.7569,"rows":151024},"winnow-12b-q8":{"median":67.0422,"p95":355.6466,"rows":151024},"openvons":{"median":17.0598,"p95":104.556,"rows":151024},"autojev-27b":{"median":96.9983,"p95":1123.6942,"rows":151024}},"table":[{"design":"benchmark+length","n":100,"worst_median_err95":0.0806,"worst_p95_err95":0.3214,"worst_median_entrant":"pplx-decider-v1-27b","worst_p95_entrant":"mini-jev","passes":false},{"design":"benchmark+length","n":200,"worst_median_err95":0.0739,"worst_p95_err95":0.2846,"worst_median_entrant":"pplx-decider-v1-27b","worst_p95_entrant":"decision-eos-0.8b","passes":false},{"design":"benchmark+length","n":300,"worst_median_err95":0.0595,"worst_p95_err95":0.1142,"worst_median_entrant":"pplx-decider-v1-27b","worst_p95_entrant":"this-that-1.2","passes":false},{"design":"benchmark+length","n":400,"worst_median_err95":0.0554,"worst_p95_err95":0.1135,"worst_median_entrant":"pplx-decider-v1-27b","worst_p95_entrant":"this-that-1.2","passes":false},{"design":"benchmark+length","n":500,"worst_median_err95":0.0518,"worst_p95_err95":0.0904,"worst_median_entrant":"pplx-decider-v1-27b","worst_p95_entrant":"this-that-1.2","passes":false},{"design":"benchmark+length","n":600,"worst_median_err95":0.0519,"worst_p95_err95":0.0746,"worst_median_entrant":"pplx-decider-v1-27b","worst_p95_entrant":"this-that-1.2","passes":false},{"design":"benchmark+length","n":750,"worst_median_err95":0.0493,"worst_p95_err95":0.0677,"worst_median_entrant":"pplx-decider-v1-27b","worst_p95_entrant":"mini-jev","passes":true},{"design":"benchmark+length","n":1000,"worst_median_err95":0.0454,"worst_p95_err95":0.0467,"worst_median_entrant":"pplx-decider-v1-27b","worst_p95_entrant":"mini-jev","passes":true},{"design":"length","n":100,"worst_median_err95":0.0924,"worst_p95_err95":0.3048,"worst_median_entrant":"winnow-12b-q8","worst_p95_entrant":"decision-eos-0.8b","passes":false},{"design":"length","n":200,"worst_median_err95":0.0736,"worst_p95_err95":0.2538,"worst_median_entrant":"pplx-decider-v1-27b","worst_p95_entrant":"decision-eos-0.8b","passes":false},{"design":"length","n":300,"worst_median_err95":0.0721,"worst_p95_err95":0.1758,"worst_median_entrant":"pplx-decider-v1-27b","worst_p95_entrant":"decision-eos-0.8b","passes":false},{"design":"length","n":400,"worst_median_err95":0.0629,"worst_p95_err95":0.1674,"worst_median_entrant":"pplx-decider-v1-27b","worst_p95_entrant":"decision-eos-0.8b","passes":false},{"design":"length","n":500,"worst_median_err95":0.0596,"worst_p95_err95":0.1513,"worst_median_entrant":"pplx-decider-v1-27b","worst_p95_entrant":"decision-eos-0.8b","passes":false},{"design":"length","n":600,"worst_median_err95":0.0569,"worst_p95_err95":0.1554,"worst_median_entrant":"pplx-decider-v1-27b","worst_p95_entrant":"decision-eos-0.8b","passes":false},{"design":"length","n":750,"worst_median_err95":0.054,"worst_p95_err95":0.1205,"worst_median_entrant":"pplx-decider-v1-27b","worst_p95_entrant":"decision-eos-0.8b","passes":false},{"design":"length","n":1000,"worst_median_err95":0.0531,"worst_p95_err95":0.1029,"worst_median_entrant":"pplx-decider-v1-27b","worst_p95_entrant":"decision-eos-0.8b","passes":false}],"chosen_n":750,"median_only_n":{"all probes":750,"all probes except AutoJev-27B (bimodal, two-card shards)":200},"rejected":{"eligible-rows pool (rows every entrant answered)":"biased: p95 low by 6-12% and pplx-decider-v1-27b median high by 7% at any n, because refusal-driven eligibility drops the longest rows; see bootstrap-eligible.json","benchmark-only stratification":"needs n=2000; see bootstrap.json","length-only stratification":"fails p95 at n=1000"}}},"methods":[{"method":"latency-v1 single process: GLiNER2 Classifier.classify in bf16 (board fp32; bf16 is equal on median and lower on p95), fleet venv torch 2.14.0+cu130, board guard lifts","engines":["gliner-base","gliner-multi","gliner-small"]},{"method":"latency-v1 single process: author DecisionEngine.predict on the whole row, SDPA, max_length 262144 and the board lifts; transformers fast path enabled with causal-conv1d 1.7.0 (fla 0.4.2 gated delta + CUDA conv). The board ran the torch fallback because causal-conv1d was absent; that run is kept as latency-v1-full.torch-fallback. venv-intern torch 2.9.1+cu128","engines":["intern-decision-0.8b","intern-decision-2b","intern-decision-4b"]},{"method":"latency-v1 single process: author RegistryRuntime.decide with the row-batched backend (all questions of a row scored together, prompts rendered exactly as model.predict); board used exact per-question mode; bf16, torch 2.14.0+cu130","engines":["bosun-v3.1-0.6b","bosun-v3.1-1.7b"]},{"method":"latency-v1 single process: author DecisionModel.from_pretrained(...).decide (bf16, FLA gated-delta kernels with sm_120 autotune instead of the gfx942 l2norm profile, questions batched in-row), venv-decision torch 2.13.0+cu130 / transformers 5.17.0 / fla 0.5.2; same as board","engines":["decision-nox-0bb83350","decision-sol-0665a411"]},{"method":"latency-v1 single process: native Kev serving at the board checkpoint (raw probabilities, no temperature, as on the board): merged bf16 backbone, fp32 pointer head, sdpa, native prefix cache 4, FLA kernels; venv-kev torch 2.8.0+cu128. A first run on the newer checkpoint (kev-models.json head) was set aside.","engines":["kev-4b-raised","kev-9b-raised"]},{"method":"latency-v1 single process: author model.decide, all questions in one forward, bf16, sdpa backbone (outer attention eager as shipped), torch 2.14.0+cu130","engines":["lumma-fev-0.1b","lumma-fev-0.6b"]},{"method":"latency-v1 single process: in-process vLLM 0.29.0 LLM.generate with all questions of the row in one call, CUDA graphs, gpu util 0.9 (single process)","engines":["tev1-0.8b","tev1-4b"]},{"method":"latency-v1 single process: author infer.score, fp32 (the only precision infer.load_trained_model offers), one padded batch of all options per question, SDPA default; max_len 32768 lift as board; OOM retry sub-batches of 8","engines":["akash-gemma"]},{"method":"latency-v1 single process: author DecisionModel forward (bf16, SDPA, one pass per question over the shared state) on one card; latency definition: In-process wall time of the DecisionModel forward: one pass per question over the shared state, CUDA-synchronized.","engines":["pplx-decider-v1-27b"]},{"method":"latency-v1 single process: clm.Engine.answer against the owned vLLM pooling server with CUDA graphs (JEV_CLM_GRAPHS=1; the author script uses --enforce-eager: eager measured 58.5 / 133.2 ms vs graphs 46.8 / 130.9), prefix caching on, gpu util 0.85; eager run kept as latency-v1-full.eager","engines":["clm-v0.1-8b"]},{"method":"latency-v1 single process: author decider.serve through decider_memory_app (DECIDER_COMPILE=1, DECIDER_FP8=1, CUDA graphs, DECIDER_MAX_FWD_TOKENS 65536), same server as board; one sequential HTTP client (run_decider_http_suite.py --concurrency 1) instead of concurrency 8","engines":["decider-2b-fp8-http"]},{"method":"latency-v1 single process: in-process vLLM 0.29.0 LLM, NVFP4 MoE, CUDA graphs, prefix caching off (board config; a prefix-cache run was slower and flipped answers), gpu util 0.9, venv-vllm torch 2.13.0+cu130","engines":["decider-35b-nvfp4"]},{"method":"latency-v1 single process: author decider.serve v1.2.1 with DECIDER_FP8=1 and DECIDER_COMPILE=1 on top of CUDA graphs (89 captured), one sequential HTTP client; the board ran graphs only. FP8+compile: 12.6 / 188.5 ms; graphs only: 13.6 / 172.7 ms (kept as latency-v1-full.stock). Chosen on the median","engines":["decider-4b"]},{"method":"latency-v1 single process: decider2.serve_chat_adaptive over serve_chat_lowmem as board (graph server serve_chat OOMs on 96 GB); one sequential HTTP client; card venv torch 2.14/transformers 5.17 instead of the job's pytorch 2.10 image","engines":["decider-chat-gemma4-31b"]},{"method":"latency-v1 single process: author decider.serve v1.2.1 chat layout (bf16, graphs), raised admission limits as board; one sequential HTTP client (engine-launched server, not attached)","engines":["decider-chat-qwen3.6-27b"]},{"method":"latency-v1 single process: author DecisionModel.decide (bf16, FLA gated-delta kernels, questions of a row batched at physical batch 8), venv-decision torch 2.13.0+cu130 / transformers 5.17.0 / fla 0.5.2; same as board (no causal-conv1d CUDA kernel, as on the board)","engines":["decision-eos-0.8b"]},{"method":"latency-v1 single process: author encoder DecisionModel (Decision 1.0 decision-kai), predict_1k cap lifted (uncapped engine), venv-decision-legacy torch 2.13.0+cu130 / transformers 4.57.6","engines":["decision-kai-7185f514"]},{"method":"latency-v1 single process: author encoder DecisionModel (Decision 1.0 decision-lex), predict_1k cap lifted (uncapped engine), venv-decision-legacy torch 2.13.0+cu130 / transformers 4.57.6","engines":["decision-lex"]},{"method":"latency-v1 single process: author DecisionModel.decide (bf16, FLA gated-delta kernels with sm_120 autotune, questions batched in-row), input limit raised to the backbone window as on the board, venv-decision torch 2.13.0+cu130","engines":["decision-lux-9b"]},{"method":"latency-v1 single process: board DjevLong: djev in-process adapter over pinned vLLM 0.29.1rc1.dev347, prefix caching, max-model-len 262144","engines":["djev"]},{"method":"latency-v1 single process: author AutoExtractor.classify_text with TF32 matmuls on (JEV_DECIDE_TF32=1; 23.3 / 228.6 ms vs 24.0 / 251.3 ms in fp32, kept as latency-v1-full.fp32), encoder attention eager as shipped, venv-q torch 2.14.0+cu130 (+peft)","engines":["gliner-decide"]},{"method":"latency-v1 single process: author engine_torch (bf16, SDPA, shared prefix + batched candidate suffixes), fleet venv torch 2.14.0+cu130; same as board","engines":["harsha"]},{"method":"latency-v1 single process: hopper-decisions 1.1.1 at tag g-1.2.0 with the HopitAI/hopper-g adapter (tournament shortlist for choice questions over 26 options, package default), Decider(cuda_graphs=True) package opt-in CUDA graphs per length bucket, eager outside captured buckets; author pins torch 2.8.0+cu128 / transformers 5.17.0 / fla 0.5.2 / causal-conv1d 1.7.0; eager variant kept as latency-v1-full.eager","engines":["hopper-g-1.2"]},{"method":"latency-v1 single process: author jeff encoder with flash attention, long-context lift as board (compile off in the package); venv-jeff","engines":["jeff-uncapped"]},{"method":"latency-v1 single process: author JevOmni.predict per question, single process (no shared-card lock); torch 2.14.0+cu130 / transformers 5.17.0","engines":["jev-omni"]},{"method":"latency-v1 single process: author DecisionEngine over vLLM 0.29.0 OpenAI server, Qwen3.8-27B-FP8, fp8 KV, prefix caching, CUDA graphs, max-model-len 262144 (board lift), gpu util 0.9","engines":["jevfire-uncapped"]},{"method":"latency-v1 single process: JevK5 0.2.2 lifted (JEVK5_LIFT=1, knockout readout), CUDA graphs on (package default JevK5(graphs=True)), fla kernels; venv torch 2.14.0+cu130 / transformers 5.17.0 / fla 0.5.2","engines":["jevk5"]},{"method":"latency-v1 single process: author jobe.score per question with SDPA attention (published default eager; sdpa is faster: median 53.5 vs 55.9 ms, p95 2910 vs 3088 ms); JOBE_LIFT=1 board lifts; venv-q torch 2.14.0+cu130. The eager run is kept as latency-v1-full.eager","engines":["jobe"]},{"method":"latency-v1 single process: open-jev[inference] diffusiongemma engine, bf16, chunked-expert path as board; venv-joshua","engines":["joshua-diffusion-full"]},{"method":"latency-v1 single process: native Kev serving: merged bf16 backbone, fp32 pointer head, sdpa, native prefix cache 4, FLA kernels, head.pt T=2.406 applied as on the board; venv-kev torch 2.8.0+cu128","engines":["kev-0.8b-raised"]},{"method":"latency-v1 single process: laya Agent.accelerate(use_graphs=True): TileLang fused kernels + CUDA graphs per shape bucket, built with the CUDA 13.0 nvcc from pip (nvidia-cuda-nvcc 13.0) because the card image ships nvcc 12.3 without sm_120a; measured with a warm TileLang JIT cache (a first pass with a cold cache had 72 rows of ~5.3 s kernel compilation, kept as latency-v1-full.fast-coldjit); p95 still includes first-use CUDA graph capture per bucket. Stock path (19.8 / 51.2 ms) kept as latency-v1-full.stock","engines":["laya-1e28ac20"]},{"method":"latency-v1 single process: author pcd Engine.constrained, board guard lifts, fleet venv torch 2.14.0+cu130","engines":["lfm2600"]},{"method":"latency-v1 single process: author rlcd Engine with SDPA attention (board path), fleet venv torch 2.14.0+cu130","engines":["lfm350-sdpa"]},{"method":"latency-v1 single process: author serve.py /v1/systemone handler in-process (Flask test client), fla kernels; torch 2.14.0+cu130 / transformers 5.17.0 / fla 0.5.2","engines":["metask-jev-4b"]},{"method":"latency-v1 single process: author mini-jev engine in-process, bf16, fleet venv torch 2.14.0+cu130 (same as board)","engines":["mini-jev"]},{"method":"latency-v1 single process: mojev.serve.Engine.answer, one packed forward over the state and every question (eager, author Space setting); torch 2.14.0+cu130","engines":["mojev"]},{"method":"latency-v1 single process: ParallelScorer.score (one full-prompt forward per question), author pins torch 2.8.0+cu128 / transformers 5.17.0 / fla 0.5.2 / causal-conv1d 1.7.0, Triton autotune cache from the board HF Job","engines":["nimble-v2"]},{"method":"latency-v1 single process: author client over vLLM 0.29.0 OpenAI server, Qwen3-4B-Instruct-2507 bf16, prefix caching, CUDA graphs, gpu util 0.8","engines":["openvons"]},{"method":"latency-v1 single process: Space cached scorer (shared prefix, chunked branch forwards), v2b merged LoRA bf16, sdpa, uncapped guards as board; fleet venv torch 2.14.0+cu130 (causal-conv1d absent: torch conv fallback)","engines":["pngwn-space-uncapped-full"]},{"method":"latency-v1 single process: openjev 0.4 API over vLLM 0.29.1rc1.dev551 NVFP4, async scheduling, prefix caching, TRITON_ATTN, one read per question group, one sequential HTTP client","engines":["razorback-one-read-3e296f08"]},{"method":"latency-v1 single process: author reflex frontend (serving-source archive 1dfaa87f: d08a8c6 + admission/readout overlays, wide_choice.py) over SGLang 0.5.20 from the pinned image b27fce60, --permutations 2, pressure admission 6, published SGLang flags with the triton attention backend (fa3 is Hopper-only), one sequential client, one RTX PRO 6000 HF Job","engines":["reflex-27b-v2"]},{"method":"latency-v1 single process: surogate v1.5.3 release wheel (pinned rune-v3-audit setup: uv py3.12.14, torch 2.11.0+cu130, noble glibc shim) serving the rune-v3 bf16 safetensors, surogate serve --max-model-len 262144 --kv-capacity 262144 --max-num-seqs 8, POST /v1/decisions with the whole row in one request (shared state prefill), one sequential client via run.py (latency_run_extra.py wrapper, engine latency_rune.RuneV3Latency)","engines":["rune-26b-a4b-v3"]},{"method":"latency-v1 single process: author score_shared with shared prefix, bf16, causal-conv1d 1.7.0 CUDA kernel (JEV_SEMIF_REQUIRE_CUDA_CONV=1), board lifts; fleet venv torch 2.14.0+cu130","engines":["semif"]},{"method":"latency-v1 single process: Solomon v1.1 bf16 serving binding with the encoding-fix path (solomon-v11-bf16-empty-context engine), sdpa (flash-attn optional, absent), venv-solomon torch 2.13.0+cu130 / transformers 5.17.0 / fla 0.5.2 as the binding requires","engines":["solomon-v11-bf16-encoding"]},{"method":"latency-v1 single process: TypedDecider.decide one forward per row, bf16, fla chunk_gated_delta_rule + causal-conv1d 1.7.0 CUDA kernel (board used the torch conv fallback), sdpa full attention, torch 2.14.0+cu130","engines":["this-that-1.2"]},{"method":"latency-v1 single process: author DecisionEngine (ModernBERT 151M) evaluate, max_length 8192 lift as board; venv-vl torch 2.13.0+cu130 from the rerun freeze. Native abstentions are not timed (357 of 750 rows answered)","engines":["verdict-8af2496e"]},{"method":"latency-v1 single process: structured_server.py over vLLM PR 57250 nightly, prefix caching, default attention backend, one sequential HTTP client","engines":["vllm-pr57250"]},{"method":"latency-v1 single process: winnow-inference llama.cpp CUDA server (6c2b3c0 + 255-label lift, WINNOW_LABELS_LIFT=1), Winnow-12B Q8_0 GGUF fully on GPU, n_ctx 131072, batch 2048 / ubatch 1024, one slot, one sequential HTTP client; same build and settings as the board","engines":["winnow-12b-q8"]},{"method":"latency-v1 single process: winnow-inference llama.cpp CUDA server (77d1458 labels-lift build), Winnow-E4B Q8_0 GGUF fully on GPU, n_ctx 131072, batch 2048 / ubatch 1024, one server and one sequential HTTP client (board: 4 servers per card)","engines":["winnow-e4b-q8"]},{"method":"latency-v1 single process: author xor server.py (raised body caps) over SGLang image 6bcaa47d, TP=1 on one RTX PRO 6000 HF Job, OPENJEV_CACHE=0 as board, one sequential client","engines":["xor"]},{"method":"latency-v1 single process: author DecisionEngine.system_one single batched forward, bf16, LEV_LOGITS_KEEP=last (board setting), torch 2.14.0+cu130","engines":["lev"]},{"method":"latency-v1 single process: author hf-server/hf_server.py (bf16, fla kernels, examples_binary policy, max-model-len 65536, batch tokens 65536), one sequential client (JEV_ATTACH); 203 of 760 rows ran out of GPU memory at the server default batch and were answered in-process by the same hf_server service with max_batch_size 32 (simplejev_ladder.py, as on the board for heavy rows); causal-conv1d absent (torch conv fallback, as on the board)","engines":["simple-jev-qwen3.8-27b"]},{"method":"latency-v1 single process: author neohorse_decision 1.0.0 wheel in its own env (py3.12.10, torch 2.8.0+cu128, triton 3.7.1), board guard lifts, one process","engines":["neohorse-jev-4b"]},{"method":"latency-v1 single process: author julia runtime (supersonic-julia 0.1.0, FastEngine) on CUDA with its optional compile_model=True (torch.compile dynamic=True; recompiles on new shapes are inside the timed rows), BF16 autocast, board guard lifts, one process; torch 2.14.0+cu130, transformers 5.0.0. Other variants kept as latency-v1-full.<variant>: eager (median 12.4 / p95 35.0 ms), marker_only_head (12.3 / 39.5), compile+marker_only_head (5.9 / 14.6)","engines":["julia-1"]},{"method":"latency-v1 single process: author-submitted full run, not re-timed by us: kit 19ad28e runner (one process, rows strictly serial) with the PR #8 JebadiahEngine in-process on one RTX PRO 6000 (HF Job), bf16 SDPA + fla 0.5.2 Gated DeltaNet kernels, prefix_cache=true (shared system+state prefix once per row, question suffixes from cached KV/DeltaNet state), batch_tokens=32768, max_batch=64 (questions batched inside a row only); torch 2.8.0+cu128, transformers 5.17.0","engines":["jebadiah-27b"]},{"method":"latency-v1 single process: author-submitted LavoirEngine (di_lavoir_engine.py, native=true) in-process on one RTX PRO 6000 Space (rtx-13): ModernBERT-large encoder with SDPA attention (pinned by lavoir/model.py), bf16 autocast, a row's questions length-sorted and batched (64 questions / 65,536 padded tokens per forward); torch 2.14.0+cu130, transformers 5.17.0, laya 0.3.11, lavoir af79b0a","engines":["lavoir"]},{"method":"latency-v1 single process: llm2jev 0.6.1 (chat prompt, thinking off, T=1.140) over SGLang 0.5.9 with the card flags (--context-length 32768 --mamba-scheduler-strategy extra_buffer; radix cache and CUDA graphs on) and --attention-backend triton (required by SGLang 0.5.9 for Qwen3.5 hybrid GDN on Blackwell); all questions of a row in one /generate call; one sequential client; one RTX PRO 6000 Space; venv torch 2.9.1+cu128, transformers 4.57.1, nvidia-cudnn-cu12 9.16.0.29 (SGLang start-up requirement)","engines":["jpt-0.8b"]},{"method":"latency-v1 single process: llm2jev 0.6.1 (chat prompt, thinking off, T=1.036) over SGLang 0.5.9 with the card flags (--context-length 32768 --mamba-scheduler-strategy extra_buffer; radix cache and CUDA graphs on) and --attention-backend triton (required by SGLang 0.5.9 for Qwen3.5 hybrid GDN on Blackwell); all questions of a row in one /generate call; one sequential client; one RTX PRO 6000 Space; venv torch 2.9.1+cu128, transformers 4.57.1, nvidia-cudnn-cu12 9.16.0.29 (SGLang start-up requirement)","engines":["jpt-4b"]},{"method":"latency-v1 single process: llm2jev 0.6.1 (chat prompt, thinking off, T=1.087) over SGLang 0.5.9 with the card flags (--context-length 32768 --mamba-scheduler-strategy extra_buffer; radix cache and CUDA graphs on) and --attention-backend triton (required by SGLang 0.5.9 for Qwen3.5 hybrid GDN on Blackwell); all questions of a row in one /generate call; one sequential client; one RTX PRO 6000 Space; venv torch 2.9.1+cu128, transformers 4.57.1, nvidia-cudnn-cu12 9.16.0.29 (SGLang start-up requirement)","engines":["jpt-9b"]},{"method":"latency-v1 single process: model repo serve.py 1.2 (caiovicentino1/Eikos-27B-FP8@e40a92f, letter readout, calib.json T=1, max_one_pass 160) over vLLM 0.30.0 with the submission's launch flags (serve_vllm.sh flags, --max-model-len 65536, --enable-prefix-caching --mamba-cache-mode align, --max-num-seqs 64, gpu util 0.85); one sequential client; one RTX PRO 6000 Space; venv torch 2.13.0+cu130, transformers 5.17.0 (fleet/eikos-27b-fp8-freeze.txt)","engines":["eikos-27b-fp8"]},{"method":"latency-v1 single process: kit PR #20 engine submissions/jet/engine.py @21092e8 (sha256 357454d0) unchanged over michaljach/jet@fbc3d2d format.py/runtime.py: BF16, SDPA, FLA chunk gated-delta kernel, PyTorch causal-conv fallback as the runtime specifies, one forward per question, no cache; the only path the published runtime has; venv torch 2.11.0+cu128, transformers 5.17.0, fla 0.5.2, triton 3.6.0 (the model repo requirements.txt); fresh process, cold autotune cache; warm-cache rerun kept as latency-v1-full.rerun (47.1 / 319.0 ms, bit-identical answers)","engines":["jet-v6.2"]}]},"jev":{"source":"artifacts/benchmark-suite/jev-release-v1/results.jsonl","ok_rows":132422,"total_wall_ms":{"median":949.5,"mean":1335.2,"p95":3743.3,"max":151038.9,"n":132422},"http_wall_ms":{"median":252.8,"mean":276.1,"p95":436.6,"max":5434.6,"n":132422},"note":"Remote hosted API measured over HTTP. total_wall_ms includes client retries and network; http_wall_ms is the request itself. Local GPU reproductions have no network leg, so these are not a controlled speed comparison.","summary":"Jev is TypeSafe's hosted API: a network round-trip from our lab with one sequential HTTPS client on the same latency sample, not comparable to on-card latency. Its answers on this sample drifted from the Sep-19 board rows (20 choice flips); Jev's scores are unchanged."},"comparability":"The leaderboard plots the reproductions' single-process time against Jev's HTTPS round trip on one axis for orientation only. The two are not a controlled speed comparison: Jev's number includes the network and shared service scheduling, the reproductions' numbers include none of that."},"answer_gaps":[{"engine":"verdict-8af2496e","name":"Verdict","variant":null,"total":120340,"unanswered":71101,"share":0.5908,"reasons":[{"reason":"chose its own abstain option instead of an answer","count":56045,"share":0.4657},{"reason":"supports only 2–24 answer options per question","count":14733,"share":0.1224},{"reason":"prompt longer than its 8,192-token context window","count":323,"share":0.0027}]},{"engine":"solomon-v11-bf16-encoding","name":"Solomon v1.1","variant":null,"total":120328,"unanswered":27498,"share":0.2285,"reasons":[{"reason":"supports only 2–8 answer options per question","count":27080,"share":0.2251},{"reason":"request shape outside the model's schema","count":415,"share":0.0034},{"reason":"request larger than its native request budget","count":3,"share":2.49e-05}]},{"engine":"semif","name":"SemIf","variant":null,"total":120340,"unanswered":15317,"share":0.1273,"reasons":[{"reason":"supports only 2–16 answer options per question","count":15317,"share":0.1273}]},{"engine":"tev1-0.8b","name":"Tev1-0.8B-experimental","variant":null,"total":119888,"unanswered":14732,"share":0.1229,"reasons":[{"reason":"Tev1 answers with one of the labels A-X (2-24 options); question 'q1' has 151","count":5500,"share":0.0459},{"reason":"Tev1 answers with one of the labels A-X (2-24 options); question 'q1' has 77","count":3080,"share":0.0257},{"reason":"Tev1 answers with one of the labels A-X (2-24 options); question 'chord' has 129","count":2000,"share":0.0167},{"reason":"Tev1 answers with one of the labels A-X (2-24 options); question 'tool' has 53","count":508,"share":0.0042}]},{"engine":"tev1-4b","name":"Tev1-4B-experimental","variant":null,"total":119898,"unanswered":14732,"share":0.1229,"reasons":[{"reason":"Tev1 answers with one of the labels A-X (2-24 options); question 'q1' has 151","count":5500,"share":0.0459},{"reason":"Tev1 answers with one of the labels A-X (2-24 options); question 'q1' has 77","count":3080,"share":0.0257},{"reason":"Tev1 answers with one of the labels A-X (2-24 options); question 'chord' has 129","count":2000,"share":0.0167},{"reason":"Tev1 answers with one of the labels A-X (2-24 options); question 'tool' has 53","count":508,"share":0.0042}]},{"engine":"xor","name":"Xor","variant":null,"total":119898,"unanswered":14500,"share":0.1209,"reasons":[{"reason":"xor server 422: q1: 2..26 options, got 151","count":5500,"share":0.0459},{"reason":"xor server 422: q1: 2..26 options, got 77","count":3080,"share":0.0257},{"reason":"xor server 422: chord: 2..26 options, got 129","count":2000,"share":0.0167},{"reason":"xor server 422: tool: 2..26 options, got 53","count":508,"share":0.0042}]},{"engine":"vllm-pr57250","name":"mmastrac diffusiongemma (vLLM PR 57250)","variant":null,"total":120340,"unanswered":14522,"share":0.1207,"reasons":[{"reason":"supports at most 26 answer options per question","count":14500,"share":0.1205},{"reason":"structured_server: question 'q_78d37c943d': label 'C' is not a single token","count":1,"share":8.31e-06},{"reason":"structured_server: question 'q_c3e60b168d': label 'C' is not a single token","count":1,"share":8.31e-06},{"reason":"structured_server: question 'q_dbd1ddcc9b': label 'D' is not a single token","count":1,"share":8.31e-06}]},{"engine":"mini-jev","name":"mini-jev","variant":null,"total":120341,"unanswered":14501,"share":0.1205,"reasons":[{"reason":"supports only 2–26 answer options per question","count":14500,"share":0.1205},{"reason":"ran out of GPU memory on the longest prompts","count":1,"share":8.31e-06}]},{"engine":"metask-jev-4b","name":"Metask-Jev-4B","variant":null,"total":120340,"unanswered":14500,"share":0.1205,"reasons":[{"reason":"metask serve.py HTTP 500: decision: enum choices must be 1–26 nonempty strings.","count":14500,"share":0.1205}]},{"engine":"lumma-fev-0.1b","name":"Lumma-Fev-0.1B","variant":null,"total":119888,"unanswered":2093,"share":0.0175,"reasons":[{"reason":"question longer than its 2,048-token context window","count":2093,"share":0.0175}]},{"engine":"djev","name":"djev","variant":null,"total":120340,"unanswered":1165,"share":0.0097,"reasons":[{"reason":"more questions per request than its answer canvas holds","count":1165,"share":0.0097}]},{"engine":"lfm2600","name":"LFM2.5-2.6B-RLCD","variant":null,"total":120340,"unanswered":533,"share":0.0044,"reasons":[{"reason":"request exceeds conservative GPU memory budget; reduce prompt or branch batch size","count":533,"share":0.0044}]},{"engine":"harsha","name":"Qwen-2.5-1B-RLCD","variant":null,"total":120340,"unanswered":380,"share":0.0032,"reasons":[{"reason":"prompt longer than its context window (no truncation allowed)","count":380,"share":0.0032}]},{"engine":"decision-lex","name":"Decision 1.0 Lex","variant":null,"total":120340,"unanswered":326,"share":0.0027,"reasons":[{"reason":"ran out of GPU memory on the longest prompts","count":319,"share":0.0027},{"reason":"prompt longer than its 8,192-token context window","count":7,"share":5.82e-05}]},{"engine":"lavoir","name":"Lavoir","variant":null,"total":120340,"unanswered":324,"share":0.0027,"reasons":[{"reason":"instructions longer than its native budget (no truncation allowed)","count":318,"share":0.0026},{"reason":"prompt longer than its context window (no truncation allowed)","count":6,"share":4.99e-05}]},{"engine":"laya-1e28ac20","name":"Laya","variant":null,"total":120226,"unanswered":320,"share":0.0027,"reasons":[{"reason":"instructions longer than its native budget (no truncation allowed)","count":314,"share":0.0026},{"reason":"prompt longer than its context window (no truncation allowed)","count":5,"share":4.16e-05},{"reason":"answer options longer than its per-option budget (no truncation allowed)","count":1,"share":8.32e-06}]},{"engine":"decision-kai-7185f514","name":"Decision 1.0 Kai","variant":null,"total":120237,"unanswered":319,"share":0.0027,"reasons":[{"reason":"ran out of GPU memory on the longest prompts","count":315,"share":0.0026},{"reason":"prompt longer than its 8,192-token context window","count":4,"share":3.33e-05}]},{"engine":"jet-v6.2","name":"Jet v6.2","variant":null,"total":120340,"unanswered":243,"share":0.002,"reasons":[{"reason":"Complete prompt has 26996 tokens; declared limit 16384","count":7,"share":5.82e-05},{"reason":"Complete prompt has 26999 tokens; declared limit 16384","count":6,"share":4.99e-05},{"reason":"Complete prompt has 27017 tokens; declared limit 16384","count":5,"share":4.15e-05},{"reason":"Complete prompt has 27003 tokens; declared limit 16384","count":5,"share":4.15e-05}]},{"engine":"julia-1","name":"Julia 1","variant":null,"total":119888,"unanswered":10,"share":8.34e-05,"reasons":[{"reason":"prompt longer than its 8,192-token context window (no truncation allowed)","count":10,"share":8.34e-05}]},{"engine":"gliner-base","name":"GLiNER 2.5 base","variant":null,"total":120340,"unanswered":9,"share":7.48e-05,"reasons":[{"reason":"prompt text with parentheses, which the native GLiNER prompt syntax cannot carry","count":9,"share":7.48e-05}]},{"engine":"gliner-multi","name":"GLiNER 2.5 multilingual","variant":null,"total":120340,"unanswered":9,"share":7.48e-05,"reasons":[{"reason":"prompt text with parentheses, which the native GLiNER prompt syntax cannot carry","count":9,"share":7.48e-05}]},{"engine":"gliner-small","name":"GLiNER 2.5 small","variant":null,"total":120340,"unanswered":9,"share":7.48e-05,"reasons":[{"reason":"prompt text with parentheses, which the native GLiNER prompt syntax cannot carry","count":9,"share":7.48e-05}]},{"engine":"lumma-fev-0.6b","name":"Lumma-Fev-0.6B","variant":null,"total":119888,"unanswered":4,"share":3.34e-05,"reasons":[{"reason":"question longer than its 12,288-token context window","count":4,"share":3.34e-05}]},{"engine":"decider-2b-fp8-http","name":"Decider 2B","variant":"FP8","total":120344,"unanswered":4,"share":3.32e-05,"reasons":[{"reason":"runtime error","count":4,"share":3.32e-05}]},{"engine":"decision-eos-0.8b","name":"Decision 1.0 Eos","variant":null,"total":119898,"unanswered":2,"share":1.67e-05,"reasons":[{"reason":"prompt longer than its 16,384-token context window","count":2,"share":1.67e-05}]},{"engine":"rune-26b-a4b-v3","name":"Surogate Rune 26B-A4B v3","variant":"bf16","total":119898,"unanswered":1,"share":8.34e-06,"reasons":[{"reason":"runtime error","count":1,"share":8.34e-06}]},{"engine":"akash-gemma","name":"system-one-gemma","variant":null,"total":120237,"unanswered":0,"share":0,"reasons":[]},{"engine":"autojev-27b","name":"pplx-decider-v1-27b","variant":null,"total":120340,"unanswered":0,"share":0,"reasons":[]},{"engine":"bosun-v3.1-0.6b","name":"Bosun v3.1 0.6B","variant":null,"total":119898,"unanswered":0,"share":0,"reasons":[]},{"engine":"bosun-v3.1-1.7b","name":"Bosun v3.1 1.7B","variant":null,"total":119898,"unanswered":0,"share":0,"reasons":[]},{"engine":"clm-v0.1-8b","name":"CLM-v0.1-8B","variant":null,"total":119898,"unanswered":0,"share":0,"reasons":[]},{"engine":"decider-35b-nvfp4","name":"Decider 35B-A3B","variant":"NVFP4","total":120340,"unanswered":0,"share":0,"reasons":[]},{"engine":"decider-4b","name":"Decider 4B","variant":null,"total":119898,"unanswered":0,"share":0,"reasons":[]},{"engine":"decider-chat-gemma4-31b","name":"Decider chat · Gemma-4-31B","variant":null,"total":119888,"unanswered":0,"share":0,"reasons":[]},{"engine":"decider-chat-qwen3.6-27b","name":"Decider chat · Qwen3.6-27B","variant":null,"total":119898,"unanswered":0,"share":0,"reasons":[]},{"engine":"decision-lux-9b","name":"Decision 1.0 Lux","variant":null,"total":119898,"unanswered":0,"share":0,"reasons":[]},{"engine":"decision-nox-0bb83350","name":"Decision 1.0 Nox","variant":null,"total":120171,"unanswered":0,"share":0,"reasons":[]},{"engine":"decision-sol-0665a411","name":"Decision 1.0 Sol","variant":null,"total":120226,"unanswered":0,"share":0,"reasons":[]},{"engine":"gliner-decide","name":"GLiNER2.5-Decide","variant":null,"total":119898,"unanswered":0,"share":0,"reasons":[]},{"engine":"hopper-g-1.2","name":"Hopper (G) 1.2","variant":null,"total":119898,"unanswered":0,"share":0,"reasons":[]},{"engine":"jeff-uncapped","name":"jeff","variant":null,"total":120340,"unanswered":0,"share":0,"reasons":[]},{"engine":"jev-omni","name":"Jev-Omni","variant":null,"total":119888,"unanswered":0,"share":0,"reasons":[]},{"engine":"jevfire-uncapped","name":"Jevfire","variant":null,"total":120340,"unanswered":0,"share":0,"reasons":[]},{"engine":"jevk5","name":"JevK5","variant":null,"total":120340,"unanswered":0,"share":0,"reasons":[]},{"engine":"jobe","name":"Jobe","variant":null,"total":120340,"unanswered":0,"share":0,"reasons":[]},{"engine":"joshua-diffusion-full","name":"open-jev (JoshuaSP)","variant":null,"total":120025,"unanswered":0,"share":0,"reasons":[]},{"engine":"kev-0.8b-raised","name":"Kev 0.8B","variant":null,"total":120340,"unanswered":0,"share":0,"reasons":[]},{"engine":"kev-4b-raised","name":"Kev 4B","variant":null,"total":120340,"unanswered":0,"share":0,"reasons":[]},{"engine":"kev-9b-raised","name":"Kev 9B","variant":null,"total":120340,"unanswered":0,"share":0,"reasons":[]},{"engine":"lfm350-sdpa","name":"LFM2.5-350M-RLCD","variant":null,"total":120340,"unanswered":0,"share":0,"reasons":[]},{"engine":"mojev","name":"MoJev","variant":null,"total":119898,"unanswered":0,"share":0,"reasons":[]},{"engine":"nimble-v2","name":"Bespoke Nimble 9B v2","variant":null,"total":119898,"unanswered":0,"share":0,"reasons":[]},{"engine":"openvons","name":"openvons","variant":null,"total":120340,"unanswered":0,"share":0,"reasons":[]},{"engine":"pngwn-space-uncapped-full","name":"open-jev (pngwn)","variant":null,"total":120139,"unanswered":0,"share":0,"reasons":[]},{"engine":"razorback-one-read-3e296f08","name":"openjev (razorback16)","variant":"one read, NVFP4","total":119898,"unanswered":0,"share":0,"reasons":[]},{"engine":"reflex-27b-v2","name":"reflex 27B","variant":"FP8 · wide choice","total":120340,"unanswered":0,"share":0,"reasons":[]},{"engine":"this-that-1.2","name":"this-that 1.2","variant":null,"total":119898,"unanswered":0,"share":0,"reasons":[]},{"engine":"winnow-12b-q8","name":"Winnow-12B","variant":"Q8_0","total":120340,"unanswered":0,"share":0,"reasons":[]},{"engine":"winnow-e4b-q8","name":"Winnow-E4B","variant":"Q8_0","total":119898,"unanswered":0,"share":0,"reasons":[]},{"engine":"lev","name":"lev","variant":null,"total":119888,"unanswered":0,"share":0,"reasons":[]},{"engine":"intern-decision-0.8b","name":"Intern-Decision-0.8B","variant":null,"total":119888,"unanswered":0,"share":0,"reasons":[]},{"engine":"intern-decision-2b","name":"Intern-Decision-2B","variant":null,"total":119888,"unanswered":0,"share":0,"reasons":[]},{"engine":"intern-decision-4b","name":"Intern-Decision-4B","variant":null,"total":119888,"unanswered":0,"share":0,"reasons":[]},{"engine":"simple-jev-qwen3.8-27b","name":"simple-jev · Qwen3.8-27B (featherless)","variant":null,"total":119888,"unanswered":0,"share":0,"reasons":[]},{"engine":"neohorse-jev-4b","name":"NeoHorse-Jev-4B","variant":null,"total":119888,"unanswered":0,"share":0,"reasons":[]},{"engine":"jebadiah-27b","name":"Jebadiah 27B","variant":null,"total":120340,"unanswered":0,"share":0,"reasons":[]},{"engine":"jpt-0.8b","name":"JPT-0.8B","variant":null,"total":120340,"unanswered":0,"share":0,"reasons":[]},{"engine":"jpt-4b","name":"JPT-4B","variant":null,"total":120340,"unanswered":0,"share":0,"reasons":[]},{"engine":"jpt-9b","name":"JPT-9B","variant":null,"total":120340,"unanswered":0,"share":0,"reasons":[]},{"engine":"eikos-27b-fp8","name":"Eikos-27B","variant":"FP8","total":120340,"unanswered":0,"share":0,"reasons":[]}],"changelog":[{"edition":"v0.4.1","date":"2026-10-21","notes":["We expanded and refreshed the private new-domain tests to cover a wider range of real decisions and measure them more accurately. The added tasks are harder, so new-domain scores sit lower than before."]},{"edition":"v0.4","date":"2026-10-19","notes":["Full score = 0.60 × private tests of the same skills + 0.40 × private tasks from new domains, both mapped onto the public scale using untuned stock models.","Public benchmarks are still run and shown next to every model, but they no longer count toward the Full score: the ranking rests entirely on tests nobody can train on.","The new-domain part uses a revised set of private tasks, kept to the ones that rank untuned stock models the same way the rest of the index does.","Ties are unchanged: models within 0.9 points of the first model in their group share a rank."]},{"edition":"v0.3.2","date":"2026-10-12","notes":["Scoring bug fix: three same-skill sets (the GPQA, MMLU-Pro and BBH skill sets) were left out of the same-skill score for 37 models scored since the 0.3 launch. Every model now is assessed against all \"same skill\" sets","New chart: public vs private score, model by model, under How the Full score works."]},{"edition":"v0.3.1","date":"2026-10-09","notes":["Vision board: every model now reads images at up to 1.6 MP (1,638,400 pixels). Models whose own image path shrank images further (to 262k pixels, or 280 image tokens) were re-run with that cap lifted; Gemma-4 models use 560 image tokens (about 1.3 MP), the nearest setting below 1.6 MP. Engines that cannot be set this way keep their own budget, shown per model.","The text board is unchanged."]},{"edition":"v0.3","date":"2026-10-06","notes":["Full score = 0.20 × public + 0.50 × private tests of the same skills + 0.30 × private tasks from new domains. The private parts are mapped onto the public scale using untuned stock models.","Most of the score comes from tests nobody can train on, so a public-only advantage counts for little.","Models within 0.9 points of the first model in their group share a rank: a gap that small could come from which questions happened to be sampled. Tied models share a rank; the next group takes the next number, so ranks count groups, not models.","Private tests of equivalent competence for most public tasks, built from fresh data and kept private, plus a set of private decision tasks from new domains.","Public index rebuilt: GSM8K rebuilt, ForecastBench retired, WinoGrande moved to Knowledge & Reasoning.","Vision board added: 0.5 × public vision benchmarks + 0.5 × private tests of the same skills.","Superseded versions leave the board when a newer version of the same model is scored: pplx-decider v1 → v1.1, RSI-Jev v4.0 → v5.0, EXAONE-4.0-1.2B-JEV v0.1 and v0.2 → v0.3, among others."]},{"edition":"v0.2.1","date":"2026-09-28","notes":["New area weights: Arts & Human Taste is fixed at 10%; the other four areas share the rest in proportion to the square root of their benchmark count: Tools & Automation 18.3%, Retrieval & Classification 20.0%, Knowledge & Reasoning 25.8%, Language Understanding 25.8%.","Gold ★ benchmarks weigh 1.2 inside their area, the rest 1.0: MMLU-Pro, BBH, GPQA Diamond, HLE, ANLI, WinoGrande, HellaSwag, BANKING77, CLINC150, BRIGHT, BFCL, API-Bank, ForecastBench.","SGD leaves the index until it is re-run with a fixed builder, and RouterBench leaves because its prompt gives away the best route. Both stay on the board.","ACOS is now scored with F1 per review instead of all-or-nothing, and stays in the index.","ToolRet and BRIGHT count only queries with at least one relevant candidate among the 32 (685 of 1,000 and 220 of 550); chance is recomputed on the same queries.","RAGTruth chance is now always answering \"hallucinated\" (F1 0.518) instead of a fair coin (0.411).","Home appliances drops 24 duplicate test rows and 48 rows identical to published dev rows (88 of 160 kept).","Surogate Rune 26B-A4B v3 replaces v1 on the board: the author's submitted run, adopted after our 3,000-row reproduction matched it exactly.","reflex 27B's full-coverage rerun replaces its 26-option run on the board: the author's submitted run with wide-choice labels, adopted after our single-card latency run agreed with it within reflex's own run-to-run noise.","Hopper (G) 1.2 replaces Hopper 1.1.1 on the board: the author's submitted run of the next Hopper adapter (same base and serving code), adopted after our single-card latency run agreed with it within the A10G-vs-RTX noise."]},{"edition":"v0.2","date":"2026-09-27","notes":["Scores are now chance-corrected: 0 means random guessing and 100 means perfect, so yes/no questions no longer give free points.","30 new models on the board, plus newer versions of 14 existing ones.","The index now counts 40 benchmarks, up from 19: 7 brand-new ones (MMLU-Pro, BBH, When2Call, RAGTruth, HoVer, New Yorker caption matching, PhishNChips), plus 15 we already ran that now count, such as HLE, ANLI, WinoGrande, HellaSwag, BANKING77 and CLINC150.","MMLU left the index: top models have nearly maxed it out, and MMLU-Pro takes its place. ARC-Easy and ARC-Challenge are still shown but don't count.","ToolRet and BRIGHT are run on a subset of their data that doesn't skew results but is faster and cheaper to run."]},{"edition":"v0.1","date":"2026-09-22","notes":["First public edition: 31 open reproductions plus Jev jev-1.13.0 on the frozen 132,422-request suite.","442 defective or out-of-reach questions excluded for every engine alike; 131,980 scoreable rows.","Panel: core25 with the six interactive environments left out (19 static benchmarks, five equal-weight areas); ChessBench folded into Knowledge & Reasoning; Language & Retrieval split into Language Understanding and Retrieval & Classification.","iSarcasmEval headline is track A English sarcasm F1; other tracks listed under each result.","Only the balanced raw index is shown; chance-normalized and breadth variants are computed but hidden.","Point estimates only; paired cluster-bootstrap intervals not yet calculated."]}],"added_benchmarks":[{"id":56,"name":"PhishNChips","dataset":"PhishNChips phishing decisions","area":"retrieval","requests":2000,"metric":"accuracy","chance":0.5,"chance_note":"uniform random choice, mean of 1/options","source":null,"suite":"calibration-extension-v1","framing":"Says whether an email and its link are phishing or legitimate.","licence":"see calibration-extension-v1 README","jev":0.8375,"entrants_scored":70,"references":[{"kind":"dataset","url":"https://huggingface.co/datasets/AreLit/PhishNChips"},{"kind":"code","url":"https://github.com/anisselbd/jev-phishing-bench"}],"explainer":"Says whether an email and its link are phishing or legitimate.","in_panel":true},{"id":57,"name":"MMLU-Pro","dataset":"MMLU-Pro","area":"knowledge","requests":12032,"metric":"accuracy","chance":0.1109,"chance_note":"uniform random choice, mean of 1/options","source":{"huggingface":{"repo":"TIGER-Lab/MMLU-Pro","revision":"b189ec765aa7ed75c8acfea42df31fdae71f97be"}},"suite":"candidates-v3/batch-1","framing":"state = question; instructions = `CHOICE`; criteria = letters `A`.. mapped to the published options in order (3–10 options; 9,981 items have 10). Gold = published `answer`. One item has duplicate option text at the source and is kept as published.","licence":"MIT","jev":0.8213,"entrants_scored":70,"references":[{"kind":"paper","url":"https://arxiv.org/abs/2406.01574"},{"kind":"dataset","url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro"}],"explainer":"Harder MMLU successor: questions across 14 subjects with up to ten options.","in_panel":true},{"id":58,"name":"BBH","dataset":"BBH fixed-option tasks","area":"knowledge","requests":5507,"metric":"accuracy","chance":0.3101,"chance_note":"uniform random choice, mean of 1/options","source":{"git":{"url":"https://github.com/suzgunmirac/BIG-Bench-Hard","commit":"9ee07bd481feebf959a6b59d61ea57bdcf30964d"}},"suite":"candidates-v3/batch-1","framing":"The 27 BBH task files minus the four with free-form answers (`dyck_languages`, `multistep_arithmetic_two`, `object_counting`, `word_sorting`) leaves 23 files = 19 task families (the three `logical_deduction_*` and three `tracking_shuffled_objects_*` files are one family each). state = the task input verbatim, including its own `Options:` block; instructions = `CHOICE`. Criteria: the `(A) text` lines parsed from that block (key `(A)`, the published target form), or the `- Yes / - No`, `- valid / - invalid` lines, or for the three tasks that list no options the task's fixed answer set (`boolean_expressions` True/False, `sports_understanding` yes/no, `web_of_lies` Yes/No). No option is invented. Four source items are malformed and excluded before any call, so 5,507 of 5,511 remain: `movie_recommendation:163` (target \"Monsters, Inc\" was split across options A and B), `ruin_names:99` and `ruin_names:144` (targets containing commas were split into several options, so the gold is not an option), `snarks:88` (input truncated to a single option).","licence":"MIT (repository LICENSE); carries the BIG-bench canary, never place in training corpora","jev":0.9319,"entrants_scored":70,"references":[{"kind":"paper","url":"https://arxiv.org/abs/2210.09261"},{"kind":"dataset","url":"https://github.com/suzgunmirac/BIG-Bench-Hard"}],"explainer":"23 tricky reasoning tasks from BIG-Bench (logic, dates, tracking objects, fallacies), answered with no working.","in_panel":true},{"id":59,"name":"RAGTruth","dataset":"RAGTruth response-level hallucination","area":"language","requests":2700,"metric":"F1 on hallucinated class","chance":0.5177,"chance_note":"Always \"hallucinated\" · F1","source":{"git":{"url":"https://github.com/ParticleMedia/RAGTruth","commit":"c103204b9ce28d6bbad859304bf30de72b8ed8fe"}},"suite":"candidates-v3/batch-2","framing":"All 2,700 test responses (900 each of QA, Summary, Data2txt; six responder models). state = `{prompt, response}`, where `prompt` is the exact generation prompt from `source_info.jsonl` (it carries the passages, article or structured data) and `response` is the evaluated response. Question is a `noul`: instructions `The response contains content that is not supported by the context in the prompt.`; criteria true `The response contains content that is not supported by the context in the prompt.`, false `All content of the response is supported by the context in the prompt.` Gold = hallucinated iff the response has at least one annotated span (`len(labels) > 0`, exactly the RAGTruth baseline's `is_halu`); 943 positives. Spans flagged `implicit_true` or `due_to_null` count, as in the baseline. Responses marked `incorrect_refusal`/`truncated` are kept. Headline metric: F1 on the hallucinated class (prediction = P(yes) ≥ 0.5), plus accuracy.","licence":"MIT annotations and code; contexts include CNN/Daily Mail, MS MARCO (non-commercial) and the Yelp Open Dataset (redistribution forbidden): built locally, rows not republished","jev":0.7512,"entrants_scored":70,"references":[{"kind":"dataset","url":"https://github.com/ParticleMedia/RAGTruth"}],"explainer":"Says yes or no: does an AI-written answer contain anything its source text doesn't support? Scored with F1.","in_panel":true},{"id":61,"name":"HoVer","dataset":"HoVer claim verification","area":"retrieval","requests":4000,"metric":"accuracy","chance":0.5,"chance_note":"uniform random choice, mean of 1/options","source":{"git":{"url":"https://github.com/hover-nlp/hover","commit":"39b84697f196308f398a251a7aea9b82ae0f0562"},"hover_wiki_db":{"url":"https://nlp.cs.unc.edu/data/hover/wiki_wo_links.db","sha256":"c37ee397916ec0bffacfe8902db454a5cda88a7a188409217b2e15231fe5ee2f"}},"suite":"candidates-v3/batch-3","framing":"All 4,000 dev claims (2,000 SUPPORTED, 2,000 NOT_SUPPORTED). The dataset provides evidence as `supporting_facts` = (Wikipedia title, sentence id) pairs into the HoVer authors' own Wikipedia introduction database (`wiki_wo_links.db`, distributed by the HoVer authors and pinned by sha256). Sentence ids refer to a CoreNLP split that is not distributed, so rather than re-splitting, state carries the **complete introduction of every document named in `supporting_facts`**: `{claim, evidence: [{title, text}]}`, documents in first-mention order. Only dataset-provided text is used; no retrieval, no external text, and NOT_SUPPORTED claims get their own annotated documents just like SUPPORTED ones. This is the oracle-document setting, easier than HoVer's open retrieval setting. instructions `Is the claim supported by the evidence?`; criteria `SUPPORTED` → `The evidence supports the claim.`, `NOT_SUPPORTED` → `The evidence does not support the claim.` (the original label strings as keys).","licence":"CC BY-SA 4.0 data, MIT code, Wikipedia text CC BY-SA: share-alike applies to republished rows","jev":0.7675,"entrants_scored":70,"references":[{"kind":"paper","url":"https://arxiv.org/abs/2011.03088"},{"kind":"dataset","url":"https://github.com/hover-nlp/hover"}],"explainer":"Decides whether a claim linking two to four Wikipedia articles is supported by their introductions.","in_panel":true},{"id":62,"name":"When2Call","dataset":"When2Call MCQ","area":"tools","requests":3652,"metric":"accuracy","chance":0.25,"chance_note":"uniform random choice, mean of 1/options","source":{"huggingface":{"repo":"nvidia/When2Call","revision":"0582f7749df63a96fdc3070932e83e72396ace53"}},"suite":"candidates-v3/batch-1","framing":"All 3,652 `mcq` test items. state = `{tools, question}` with `tools` = the item's `tools` field parsed from JSON strings (not `orig_tools`; 258 items have no tools). instructions `Which response should the assistant give to the user's question, given the available tools?`; criteria `A`..`D` = the four published answers in published order (direct, tool_call, request_for_info, cannot_answer). Category names are not on the wire. Gold is never `A` (direct) by the dataset's construction. Reported: accuracy, macro-F1 over the three gold categories, and tool-call rate when no tools are supplied.","licence":"CC BY 4.0","jev":0.8165,"entrants_scored":70,"references":[{"kind":"paper","url":"https://aclanthology.org/2025.naacl-long.174"},{"kind":"dataset","url":"https://huggingface.co/datasets/nvidia/When2Call"}],"explainer":"Given a question and an assistant's tools, picks the right move: answer, call a tool, ask for details, or decline.","in_panel":true},{"id":64,"name":"New Yorker","dataset":"New Yorker caption matching","area":"arts","requests":528,"metric":"accuracy","chance":0.2,"chance_note":"uniform random choice, mean of 1/options","source":{"huggingface":{"repo":"jmhessel/newyorker_caption_contest","revision":"d81cbab7d0392708d5371d3a4960e69261824db4"}},"suite":"candidates-v3/batch-1","framing":"`matching` config (fold 0) test, 528 items. Text only, no pixels: state = `{scene, description, uncanny_description, entities}` = the dataset's `image_location`, `image_description`, `image_uncanny_description`, `entities` (Wikipedia URLs as published). instructions `Which caption was written for this cartoon?`; criteria `A`..`E` = `caption_choices` in order.","licence":"CC BY 4.0 annotations; cartoons not used, text descriptions only","jev":0.7254,"entrants_scored":70,"references":[{"kind":"paper","url":"https://arxiv.org/abs/2209.06293"},{"kind":"dataset","url":"https://huggingface.co/datasets/jmhessel/newyorker_caption_contest"}],"explainer":"Given a text description of a New Yorker cartoon, picks the caption written for it from five.","in_panel":true}],"reproduce":{"github":"https://github.com/apolinario/decision-index","lab_scripts":["evaluation/reproductions/run.py","evaluation/reproductions/report_model.py","evaluation/reproductions/answer_gaps.py","evaluation/reproductions/compute_indices.py","evaluation/reproductions/entrant_metadata.py","evaluation/reproductions/build_leaderboard.py","evaluation/reproductions/build_methodology.py"],"outputs":["data/index-v0.2.1.json","data/methodology-v0.2.1.json"]},"notes":{"lifts":[{"engine":"winnow-12b-q8","name":"Winnow-12B","lift_class":"labels-255","rows_answered_by_lift":10582,"line":"Winnow-12B: answer-label stop in the author's native engine (native/engine.h) raised from 64 to 255 labels; same label loop and single-token filter, first 64 labels unchanged; 10,582 previously refused rows answered by the lift."},{"engine":"semif","name":"SemIf","lift_class":"direct-fallback-max-tokens","rows_answered_by_lift":2855,"line":"SemIf: max_tokens raised from the 4,096 keyword default to the 262,144-token Qwen3.5-4B window; rows that fail the shared-prefix precondition or run out of memory in the shared batch take the author's direct path; 2,855 previously refused rows answered by the lift."},{"engine":"lfm2600","name":"LFM2.5-2.6B-RLCD","lift_class":"pcd-limits","rows_answered_by_lift":11910,"line":"LFM2.5-2.6B-RLCD: pcd Limits request budgets raised (prompt 122,880 plus value 8,192 tokens inside the 131,072-token window; field, candidate and character budgets opened); batch sizes unchanged; 11,910 previously refused rows answered by the lift."},{"engine":"pngwn-space-uncapped-full","name":"open-jev (pngwn)","lift_class":"stale-question-limit","rows_answered_by_lift":26,"line":"open-jev (pngwn): rows refused under a question limit the uncapped path no longer applies were re-answered, and rows missing from the release run were answered; 26 previously refused rows answered by the lift."},{"engine":"jevfire-uncapped","name":"Jevfire","lift_class":"context-262144","rows_answered_by_lift":292,"line":"Jevfire: context window raised from 16,384 to the 262,144-token Qwen3.8-27B-FP8 window (vLLM max-model-len); 292 previously refused rows answered by the lift."},{"engine":"djev","name":"djev","lift_class":"context-262144","rows_answered_by_lift":388,"line":"djev: max_model_len raised from 32,768 to 262,144, canvas from 128 to the checkpoint's declared 256, question cap from 32 to 256; API character caps opened; 388 previously refused rows answered by the lift."},{"engine":"jeff-uncapped","name":"jeff","lift_class":"tokenizer-budget-lean-encoder","rows_answered_by_lift":564,"line":"jeff: tokenizer max_len of 16,384 lifted (the encoder has no absolute position limit); rows that ran out of GPU memory re-answered with a lean encoder path that skips two unused n x n tensors; 564 previously refused rows answered by the lift."},{"engine":"gliner-base","name":"GLiNER 2.5 base","lift_class":"maxlen-paren-chunked-attention","rows_answered_by_lift":19525,"line":"GLiNER 2.5 base: harness max_len guard removed (upstream sets none); \"(\" and \")\" removed from GLiNER2's reserved characters (markers are routed by position; rows without parentheses bit-identical); memory-bound rows re-answered with query-chunked DeBERTa attention; 19,525 previously refused rows answered by the lift."},{"engine":"gliner-small","name":"GLiNER 2.5 small","lift_class":"maxlen-paren-chunked-attention","rows_answered_by_lift":19525,"line":"GLiNER 2.5 small: harness max_len guard removed (upstream sets none); \"(\" and \")\" removed from GLiNER2's reserved characters (markers are routed by position; rows without parentheses bit-identical); memory-bound rows re-answered with query-chunked DeBERTa attention; 19,525 previously refused rows answered by the lift."},{"engine":"gliner-multi","name":"GLiNER 2.5 multilingual","lift_class":"maxlen-paren-chunked-attention","rows_answered_by_lift":19852,"line":"GLiNER 2.5 multilingual: harness max_len guard removed (upstream sets none); \"(\" and \")\" removed from GLiNER2's reserved characters (markers are routed by position; rows without parentheses bit-identical); memory-bound rows re-answered with query-chunked DeBERTa attention; 19,852 previously refused rows answered by the lift."},{"engine":"akash-gemma","name":"system-one-gemma","lift_class":"max-len-32768-full","rows_answered_by_lift":41425,"line":"system-one-gemma: max_len raised from the 384 keyword default to the 32,768-token gemma-3-270m window; out-of-memory retries score options in sub-batches of 8 (0 flips, max |dp| 2.4e-6 on 30 rows); 41,425 previously refused rows answered by the lift."},{"engine":"jobe","name":"Jobe","lift_class":null,"rows_answered_by_lift":15504,"line":"Jobe: questions over 16 options answered through the author's score_text path; rows that ran out of GPU memory re-run with query-chunked eager attention (0 flips, max |dp| 0.061 on 30 rows); 15,504 previously refused rows answered by the lift."},{"engine":"hopper","name":"Hopper","lift_class":null,"rows_answered_by_lift":14500,"line":"Hopper: questions over 26 options answered by the author's v1.1.1 runtime (same weights and map; discussion #14): a tournament deals the menu into ceil(n/26) chunks with a fixed seed, keeps the 10 best, decides among them with one more pass, and spreads 5% of the mass uniformly over the whole menu; on this card v1.1.1 and v1.1.0 are bit-identical on 106 in-range rows (both 51/106 bit-identical to the rtx-3 release rows, max |dp| 0.078, 2 flips, a card/autotune effect), and in-range rows stay verbatim from the release run; 14,500 previously refused rows answered by the lift."},{"engine":"jevk5","name":"JevK5","lift_class":null,"rows_answered_by_lift":15317,"line":"JevK5: guard lifted; 15,317 previously refused rows answered by the lift."},{"engine":"decision-kai-7185f514","name":"Decision 1.0 Kai","lift_class":null,"rows_answered_by_lift":18969,"line":"Decision 1.0 Kai: rows refused at the 1,024-token predict_1k product cap answered with the bundle's native 8,192-token packing (the author points users to predict_1k); empty-state check reduced to a type check; 18,969 previously refused rows answered by the lift."},{"engine":"decision-lex","name":"Decision 1.0 Lex","lift_class":null,"rows_answered_by_lift":20791,"line":"Decision 1.0 Lex: rows refused at the 1,024-token predict_1k product cap answered with the bundle's native 8,192-token packing (the author points users to predict_1k); empty-state check reduced to a type check; 20,791 previously refused rows answered by the lift."},{"engine":"neohorse-jev-4b","name":"NeoHorse-Jev-4B","lift_class":"in-run","rows_answered_by_lift":null,"line":"NeoHorse-Jev-4B: four serving guards of the neohorse_decision runtime shipped in the bundle lifted (DecisionEngine documents them as deployment protection limits): max_state 2,048 and max_branch 8,192 raised to the 262,144-token backbone window (strict encode kept, so nothing is truncated), the 16-questions-per-request cap and the 32,768-token expanded-request budget opened. 9,388 of 151,024 rows need a lift (6,038 over the state cap, mostly RouterBench, POP909-CL and API-Bank; 3,401 over 16 questions: ACOS, ToolRet, BRIGHT, Home appliance, ContractNLI and 2 others; 183 over the token budget; 98 over the branch cap; longest row 22,163 tokens). Every in-range row checked is bit-identical to the stock defaults (74 smoke rows, 200 sampled full-run rows). The author's single padded forward is always tried first; 69 rows that ran out of GPU memory, or hit the torch conv1d 32-bit index limit (one 32-question ToolRet row at 22,163 tokens), were re-run with their question rows split in halves at the same padded width."},{"engine":"simple-jev-qwen3.8-27b","name":"simple-jev · Qwen3.8-27B (featherless)","lift_class":"in-run","rows_answered_by_lift":null,"line":"simple-jev · Qwen3.8-27B (featherless): served exactly as the user specified (examples_binary, --max-model-len 65,536, --max-batch-tokens 65,536, 256 branches, 255 options); 2 ToolRet rows whose question branch exceeds 65,536 tokens were re-answered with both limits raised to the 262,144-token Qwen3.8-27B window (the flag is an admission check, nothing is truncated either way). The server batches up to 32 question branches per forward and does not fit that next to the 27B weights on ToolRet/BRIGHT rows, so those 1,550 rows ran through the same service in-process, stepping the batch size down (32, 16, 8 ...) only after an out-of-memory; batch size moves probabilities by up to 0.05 with no choice flips on 7 smoke rows (208 questions)."},{"engine":"mojev","name":"MoJev","lift_class":"in-run","rows_answered_by_lift":null,"line":"MoJev: keeps its field_tokens_max lift: per-question prompt cap raised from the 32-token training default to the 262,144-token backbone window (the collate would otherwise cut instructions), context_tokens from 16,384 to 262,144."},{"engine":"winnow-e4b-q8","name":"Winnow-E4B","lift_class":"in-run","rows_answered_by_lift":null,"line":"Winnow-E4B: runs on the Winnow-12B lift: answer-label stop in the author's native engine (native/engine.h) raised from 64 to 255 labels; same label loop and single-token filter, first 64 labels unchanged; 10,582 release rows with more than 64 options (CLINC150+OOS, BANKING77, POP909-CL, ChessBench) answered, and 732 of 732 in-range rows re-run on the pinned build are bit-identical."},{"engine":"lumma-fev-0.6b","name":"Lumma-Fev-0.6B","lift_class":"in-run","rows_answered_by_lift":null,"line":"Lumma-Fev-0.6B: MAX_ROW in the author's modeling_fev.py raised from 8,192 to above the 12,288-token backbone window, so the author's own limits() lowers it to the window (a state is cut at 11,648 tokens by its QUESTION_RESERVE rule and each question row keeps the rest); every in-range smoke row is bit-identical to the stock limits."},{"engine":"tev1-0.8b","name":"Tev1-0.8B-experimental","lift_class":"in-run","rows_answered_by_lift":null,"line":"Tev1-0.8B-experimental: lifts of the Tev1-4B engine: an empty question is sent as \"\" (decide.py's nonempty-question check is client validation), noul questions are sent as two options A \"yes\" / B \"no\", and the context is the 262,144-token backbone window (no serving cap is published); questions over 24 options stay refused because the A-X alphabet is the trained answer format."},{"engine":"intern-decision-0.8b","name":"Intern-Decision-0.8B","lift_class":"in-run","rows_answered_by_lift":null,"line":"Intern-Decision-0.8B: three serving guards of the inference.py shipped with the checkpoint lifted: DecisionEngine max_length raised from 8,192 to the 262,144-token backbone window; the 1-16 questions check in validate_request (the HTTP service limit, per the release's docs/DATA.md) opened by running the author's own validation over chunks of 16, so the whole row is still one skeleton and one forward pass; the 62-symbol answer alphabet (A-Z, a-z, 0-9) extended with the 562 two-letter labels AA-ZZ that stay one token (the Winnow and razorback order and filter; the model was trained on the 62). 14,493 of 151,024 rows need a lift (CLINC150+OOS, BANKING77, POP909-CL and 3 ChessBench rows over 62 options; ACOS, ToolRet, BRIGHT, Home appliance, ContractNLI and 3 others over 16 questions; 1,681 rows over 8,192 tokens, longest 33,135); every in-range smoke row is bit-identical to the stock module."},{"engine":"intern-decision-2b","name":"Intern-Decision-2B","lift_class":"in-run","rows_answered_by_lift":null,"line":"Intern-Decision-2B: three serving guards of the inference.py shipped with the checkpoint lifted: DecisionEngine max_length raised from 8,192 to the 262,144-token backbone window; the 1-16 questions check in validate_request (the HTTP service limit, per the release's docs/DATA.md) opened by running the author's own validation over chunks of 16, so the whole row is still one skeleton and one forward pass; the 62-symbol answer alphabet (A-Z, a-z, 0-9) extended with the 562 two-letter labels AA-ZZ that stay one token (the Winnow and razorback order and filter; the model was trained on the 62). 14,493 of 151,024 rows need a lift (CLINC150+OOS, BANKING77, POP909-CL and 3 ChessBench rows over 62 options; ACOS, ToolRet, BRIGHT, Home appliance, ContractNLI and 3 others over 16 questions; 1,681 rows over 8,192 tokens, longest 33,135); every in-range smoke row is bit-identical to the stock module."},{"engine":"intern-decision-4b","name":"Intern-Decision-4B","lift_class":"in-run","rows_answered_by_lift":null,"line":"Intern-Decision-4B: three serving guards of the inference.py shipped with the checkpoint lifted: DecisionEngine max_length raised from 8,192 to the 262,144-token backbone window; the 1-16 questions check in validate_request (the HTTP service limit, per the release's docs/DATA.md) opened by running the author's own validation over chunks of 16, so the whole row is still one skeleton and one forward pass; the 62-symbol answer alphabet (A-Z, a-z, 0-9) extended with the 562 two-letter labels AA-ZZ that stay one token (the Winnow and razorback order and filter; the model was trained on the 62). 14,493 of 151,024 rows need a lift (CLINC150+OOS, BANKING77, POP909-CL and 3 ChessBench rows over 62 options; ACOS, ToolRet, BRIGHT, Home appliance, ContractNLI and 3 others over 16 questions; 1,681 rows over 8,192 tokens, longest 33,135); every in-range smoke row is bit-identical to the stock module."},{"engine":"julia-1","name":"Julia 1","lift_class":"in-run","rows_answered_by_lift":null,"line":"Julia 1: four serving guards of the julia runtime shipped in the model repo lifted, keeping its strict (never-truncate) encoding: the 2-20 options check in validate_row opened (the head scores each [MASK] option marker independently and has no per-option parameters), the 48-token option clip and its strict refusal removed, the question-plus-options budget raised from the card's head_length 512 to 8,187 (max_length - 5, the largest the runtime accepts; a budget, not a reserved region), and the strict refusal on a literal mask token replaced by the runtime's own clean(). Both functions are recompiled from the pinned source with only those edits. 24,706 of 151,024 rows need a lift (15,060 over 20 options: CLINC150+OOS, BANKING77, ChessBench, POP909-CL, API-Bank, HLE; 7,708 with an option over 48 tokens: HellaSwag, Habermas, When2Call, BPoMP and 6 others; 1,954 over the 512 head budget: MuSR, ToolRet, BRIGHT, MMLU, HLE, GPQA; 2 others). The 8,192-token encoder window (max_position_embeddings) is kept, so 18 rows over it are refused, never cut (11 ToolRet, 3 BRIGHT, 3 POP909-CL, 1 HLE). Every in-range row checked is bit-identical to the card settings (70 smoke rows, 200 sampled full-run rows)."},{"engine":"clm-v0.1-8b","name":"CLM-v0.1-8B","lift_class":"in-run","rows_answered_by_lift":null,"line":"CLM-v0.1-8B: full-text run: max_model_len raised from the script's 2,048 to the 40,960-token Qwen3-8B window with truncation off; over-window rows refused, not cut (the truncated 2,048-token variant is not used)."},{"engine":"rune-26b-a4b","name":"Surogate Rune 26B-A4B","lift_class":"candidate-harness","rows_answered_by_lift":2000,"line":"Surogate Rune 26B-A4B: PhishNChips harness mapping fixed: every row was refused because the surogate decisions endpoint requires a noul question's criteria to be exactly true/false and five of its noul questions carry an empty criteria object, and it also refuses a null option description (verdict_alt_minimal); empty noul criteria are now sent as {\"false\": \"False\", \"true\": \"True\"} (the runner's default noul wording, as Bosun's empty-criteria fix sends the server default) and a null description as its option key; questions without either are sent byte-identical, and 82/82 answered release smoke rows re-ran bit-identical; 2,000 previously refused rows answered by the fix."},{"engine":"decision-kai-7185f514","name":"Decision 1.0 Kai","lift_class":"candidate-harness","rows_answered_by_lift":1676,"line":"Decision 1.0 Kai: new benchmarks (MMLU-Pro, When2Call, RAGTruth): rows refused at the 1,024-token predict_1k product cap answered with the bundle's native 8,192-token packing at 7185f514, as in its 0.2 lift; 30/30 in-range rows re-ran bit-identical; 1,676 previously refused rows answered by the fix."}],"lifts_not_applied":[{"engine":"solomon-v11-bf16-encoding","line":"Solomon v1.1: lift not applied; the lifted run is built on the earlier solomon-v11-bf16 serving path, not on the encoding-fix path the board uses; on release-v2 rows it answers 422 rows the encoding run refused but refuses 4,768 rows the encoding run answered, so it would lower coverage; the release encoding-fix run stays until a lift over that path exists."}],"run_variation":[{"engine":"decider-35b-nvfp4","line":"Decider 35B-A3B: the release run and a rerun in the rebuilt release environment agree on 97.7% of choices (420 of 430; 9 of the 10 flips are near-ties); two reruns on one card are bit-identical in choice; the perturbation-suite control rows agree with the scored run on 96.9% (1,642 of 1,695)."},{"engine":"razorback-one-read-3e296f08","line":"openjev (razorback16): the openjev 0.4.0 build is not deterministic on the author's default flags; two smoke passes agree on 85.3% of choices (367 of 430), 95.6% with vLLM batch-invariant mode, which no documented option makes the default; the perturbation-suite control rows agree with the scored run on 92.6% (1,570 of 1,695). The full run uses the author defaults."},{"engine":"jevfire-uncapped","line":"Jevfire: the perturbation-suite control rows (unchanged source rows, same build, vLLM serving on the author's settings) agree with the scored run on 97.6% (1,655 of 1,695) of choices."},{"engine":"vllm-pr57250","line":"mmastrac diffusiongemma (vLLM PR 57250): the perturbation-suite control rows (unchanged source rows, same build, vLLM PR 57250 serving on the author's settings) agree with the scored run on 99.6% (1,688 of 1,695) of choices."},{"engine":"clm-v0.1-8b","line":"CLM-v0.1-8B: a rerun of 17 sampled rows on the scored serving settings reproduces 62 of 62 labels (max |dp| 0.011)."},{"engine":"tev1-4b","line":"Tev1-4B-experimental: two smoke passes on the scored vLLM settings agree on 100.0% (421 of 421) of labels."},{"engine":"tev1-0.8b","line":"Tev1-0.8B-experimental: two smoke passes on the scored vLLM settings agree on 100.0% (421 of 421) of labels and are bit-identical; further single-process reruns, a second card and five concurrent processes per card (the scored setting) differ from the first pass on at most 2 of 77 rows, each by one bf16 step in a label logit (max |dp| 0.031, no label changes)."},{"engine":"gliner-decide","line":"GLiNER2.5-Decide: two smoke passes agree on 100.0% (430 of 430) of labels, and every probability is bit-identical between the passes and across the two cards the run was split over."},{"engine":"winnow-e4b-q8","line":"Winnow-E4B: two smoke passes agree on 100.0% (424 of 424) of labels, and every probability is bit-identical between the passes, across cards, with four server processes per card, and between the pinned and label-lifted builds on every row with at most 64 options."},{"engine":"lumma-fev-0.1b","line":"Lumma-Fev-0.1B: two smoke passes agree on 100.0% (428 of 428) of labels, and every probability is bit-identical between the passes, with six concurrent processes on one card, and against the author's stock row limits on every row those limits accept."},{"engine":"lumma-fev-0.6b","line":"Lumma-Fev-0.6B: two smoke passes agree on 100.0% (430 of 430) of labels, and every probability is bit-identical between the passes, with six concurrent processes on one card, and against the author's stock row limits on every row those limits accept."},{"engine":"simple-jev-qwen3.8-27b","line":"simple-jev · Qwen3.8-27B (featherless): two smoke passes agree on 100.0% (302 of 302) of labels and every response is bit-identical; 1, 4 and 8 concurrent clients on one server return bit-identical answers on 164 release rows, all six cards reproduce 20 stored smoke rows bit for bit, and plain curl requests return the stored responses byte for byte."},{"engine":"lev","line":"lev: two smoke passes agree on 100.0% (430 of 430) of labels and are bit-identical. The run used three processes per card; rows that ran out of GPU memory in the author's single batched forward were re-run in smaller chunks, which is not bit-identical (on 60 smoke rows forced through that path: max |dp| 0.052, 2 label flips)."},{"engine":"jev-omni","line":"Jev-Omni: two smoke passes agree on 100.0% (430 of 430) of labels, and every probability is bit-identical between the passes, with three concurrent processes on one card, across the two cards, and against the author's unmodified load_jev_omni() on the card's four verification cases and 68 suite questions."},{"engine":"intern-decision-0.8b","line":"Intern-Decision-0.8B: two smoke passes agree on 100.0% (430 of 430) of labels, and every probability is bit-identical between the passes, with 10 concurrent processes on one card, and against the published guards on every row they accept."},{"engine":"intern-decision-2b","line":"Intern-Decision-2B: two smoke passes agree on 100.0% (430 of 430) of labels, and every probability is bit-identical between the passes, with 8 concurrent processes on one card, and against the published guards on every row they accept."},{"engine":"intern-decision-4b","line":"Intern-Decision-4B: two smoke passes agree on 100.0% (430 of 430) of labels, and every probability is bit-identical between the passes, with 6 concurrent processes on one card, and against the published guards on every row they accept."},{"engine":"neohorse-jev-4b","line":"NeoHorse-Jev-4B: two smoke passes agree on 100.0% (430 of 430) of labels and every probability is bit-identical between the passes; with four concurrent processes per card, 200 sampled in-range full-run rows are bit-identical to a fresh single process of the stock runtime."},{"engine":"julia-1","line":"Julia 1: two smoke passes agree on 100.0% (430 of 430) of labels and every logit is bit-identical between the passes, across 8 concurrent processes on one card, and against the card settings on every row they accept; with 16 processes per card, 200 sampled in-range full-run rows are bit-identical to a fresh single process of the stock runtime."},{"engine":"xor","line":"Xor: no repeat of the single-GPU run on the same settings is on file, so its run-to-run variation is not measured."}],"contamination":["reflex 4B (removed from the board): confirmed leakage. Its published LoRA adapter trained on 800 MMLU-Pro test items; on those items it scores 65.8% against 52.1% on held-out items, a gap of 13.7 points (95% CI 9.8 to 17.5). The author's recommended configuration is the frozen Qwen3.5-4B base without that adapter, so the adapter entry is not shown. reflex 27B stays.","Eikos-27B: 979 FinEntity test items it was trained on (author-disclosed) count as wrong answers; its FinEntity score is still computed over the full benchmark, and its area and index scores follow. caiovicentino1/eikos-decisions (declared on the Eikos-27B card) trains on FinEntity with the human entity labels (1,994 rows, used_in eikos-27b). All 979 suite FinEntity documents appear verbatim as training states (966 in the train split, 13 in the validation split), and every one of their 2,129 entity questions has a training question on the same entity. User decision 2026-09-27: every FinEntity row counts as wrong for this entrant.","Jet v6.2: every training file of its weight lineage (V6.0 train_v5_r2, V6.1 targeted, V6.2 focused) was rebuilt byte-identical to the hashes in github michaljach/jet with the author's own builders; 5 of our test items appear in it with the same label (2 BANKING77, 3 CLINC150), train-split duplicates of test utterances that passed the author's exclusion filter. Those items count as wrong answers; each benchmark is still computed over the full set. 4 RouterBench queries also occur in its V6.1 data (GSM8K train questions inside UltraFeedback prompts), but RouterBench's label is a routing choice that the training text does not carry, so they are not counted. Details: contamination/jet-v6.2-training-overlap.json.","JevK5: 980 MMLU-Pro test items it was trained on (author-disclosed) count as wrong answers; its MMLU-Pro score is still computed over the full benchmark, and its area and index scores follow. Union of the author-published training ids for JevK5 v0.2 (980) and v0.1 (480), mapped to catalog-57 run ids in contamination/jevk5-mmlu-pro-items.json; v0.2 descends from v0.1 training, so both count as trained-on.","lev: its declared training data (29 sources on 26 public datasets in Abhinavexists/lev sources.py) was rebuilt with the author's own builder; 45 of our test items appear in its 200,000 training records (22 BANKING77, 12 CLINC150+OOS, 1 ANLI, 9 ARC-Easy, 1 HellaSwag), mostly exact duplicates between the public train and test splits. Those items count as wrong answers; each benchmark is still computed over the full set. 42 RouterBench queries also occur in its training data (GSM8K train questions via UltraFeedback), but RouterBench's label is a routing choice that the training text does not carry, so they are not counted. Details: contamination/lev-training-overlap.json.","Lumma-Fev-0.1B: its model card declares no training data, but FrontiersMind's own decision datasets (new_kevv_dataset, v2_kevv_dataset) contain 3,077 of our 3,080 BANKING77 test items with their labels, and the card reports Banking77 among its four headline benchmarks. Those items count as wrong answers in its BANKING77 score, still computed over the full benchmark. Details: contamination/lumma-fev-*-overlap.json.","Lumma-Fev-0.6B: its model card declares no training data, but FrontiersMind's own decision datasets (new_kevv_dataset, v2_kevv_dataset) contain 3,077 of our 3,080 BANKING77 test items with their labels, and the card reports Banking77 among its four headline benchmarks. Those items count as wrong answers in its BANKING77 score, still computed over the full benchmark. Details: contamination/lumma-fev-*-overlap.json.","open-jev (pngwn): its training set, pngwn/typed-decisions-v2-system-one, was built from the MMLU-Pro test split, so 9,542 of our 12,032 MMLU-Pro items were trained on for two epochs, and 5,308 MMLU test questions reappear among them. It scores 0.929 on the trained MMLU-Pro items and 0.614 on the 2,490 it never saw, a gap of +31 points that no other entrant shows. Those contaminated items count as wrong answers in its MMLU-Pro and MMLU scores, which are still computed over the full benchmarks. Details: contamination/open-jev-mmlu-pro.md.","Shuffle test (options reordered, keys kept or renamed, on MMLU, GPQA Diamond, ARC-Challenge, HellaSwag and MMLU-Pro): 12 entrants tested, no entrant flagged (no drop of 5 points or more with a 95% CI excluding zero). 6 smaller drops under 5 points have a CI excluding zero: open-jev (pngwn) HellaSwag; AutoJev-27B MMLU-Pro; Winnow-12B MMLU; Winnow-12B HellaSwag; open-jev (JoshuaSP) MMLU-Pro; mmastrac diffusiongemma (vLLM PR 57250) HellaSwag.","Declared overlap, 57 MMLU-Pro: test items in training: jevk5: 980 author-published ids (v0.1 and v0.2 union), counted as wrong in its MMLU-Pro score; open-jev: 9,542 test items (typed-decisions-v2-system-one), counted as wrong in its MMLU-Pro score.","Declared overlap, 24 MMLU: test items in training: open-jev: 5,308 test questions, via MMLU-Pro ori_mmlu items, counted as wrong in its MMLU score. other splits of the same dataset in training: decider family, this-that, hopper, jqv-targeted-v2 (auxiliary_train).","Declared overlap, 26/27 ARC-Easy/Challenge: test items in training: 15 rows (12 Easy, 3 Challenge) duplicated in ARC train and MMLU auxiliary_train: counts for decider family, hopper, jevk5, jqv, this-that. other splits of the same dataset in training: decider family, hopper, jevk5, kev-8b (not on board).","Declared overlap, 29 HellaSwag: other splits of the same dataset in training: decider family, this-that (train split; ours is validation).","Declared overlap, 28 WinoGrande: other splits of the same dataset in training: decider family, this-that. used for checkpoint selection: pplx-decider-v1-27b used WinoGrande rows for checkpoint selection.","Declared overlap, 4 BANKING77: test items in training: 2 rows duplicated in the train split: counts for decider family, kev, jevk5, decision family, reflex. other splits of the same dataset in training: decider family, kev, jevk5, decision family, reflex.","Declared overlap, 5 CLINC150: other splits of the same dataset in training: decider family, hopper, decision family, reflex.","Declared overlap, 58 BBH / 59 RAGTruth: used for checkpoint selection: pplx-decider-v1-27b checkpoint selection used 750 BBH and 1,500 RAGTruth rows."],"other":["Contrastive rankers such as CLM-v0.1-8B do poorly when the option texts are bare labels (yes/no, NotMentioned): scored by similarity, the label carries almost nothing to rank, and CLM picks NotMentioned on all of ContractNLI, yes on 99% of BFCL decisions and no on all of ForecastBench.","Xor runs on a single RTX PRO 6000 with tensor parallelism 1, as every entrant runs on one card for fairness; the author validated the model at tensor parallelism 2 on two cards, and only the tensor-parallel split and the static memory fraction differ from that configuration."],"board_changes":{"replaced":[{"retired":"decision-kai","replaced_by":"decision-kai-7185f514","why":"newer Decision 1.0 Kai revision 7185f514","line":"Decision 1.0 Kai (decision-kai-7185f514) replaces decision-kai: newer Decision 1.0 Kai revision 7185f514."},{"retired":"decision-sol","replaced_by":"decision-sol-0665a411","why":"newer Decision 1.0 Sol revision 0665a411","line":"Decision 1.0 Sol (decision-sol-0665a411) replaces decision-sol: newer Decision 1.0 Sol revision 0665a411."},{"retired":"decision-nox-full","replaced_by":"decision-nox-0bb83350","why":"newer Decision 1.0 Nox revision 0bb83350","line":"Decision 1.0 Nox (decision-nox-0bb83350) replaces decision-nox-full: newer Decision 1.0 Nox revision 0bb83350."},{"retired":"this-that-1.0","replaced_by":"this-that-1.2","why":"newer this-that release","line":"this-that 1.2 (this-that-1.2) replaces this-that-1.0: newer this-that release."},{"retired":"nimble","replaced_by":"nimble-v2","why":"newer Nimble release (v2)","line":"Bespoke Nimble 9B v2 (nimble-v2) replaces nimble: newer Nimble release (v2)."},{"retired":"razorback-one-read","replaced_by":"razorback-one-read-3e296f08","why":"newer openjev build 0.4.0 @3e296f08","line":"openjev (razorback16) (razorback-one-read-3e296f08) replaces razorback-one-read: newer openjev build 0.4.0 @3e296f08."},{"retired":"laya","replaced_by":"laya-1e28ac20","why":"newer laya code revision 1e28ac20 (same weights)","line":"Laya (laya-1e28ac20) replaces laya: newer laya code revision 1e28ac20 (same weights)."},{"retired":"verdict","replaced_by":"verdict-8af2496e","why":"newer Verdict weights 8af2496e and code 30f15564","line":"Verdict (verdict-8af2496e) replaces verdict: newer Verdict weights 8af2496e and code 30f15564."},{"retired":"rune-26b-a4b","replaced_by":"rune-26b-a4b-v3","why":"author-submitted v3 checkpoint (kit PR #9), adopted after a 3,000-row reproduction matched it bit for bit; v1 stays on the 0.2 board","line":"Surogate Rune 26B-A4B v3 (rune-26b-a4b-v3) replaces rune-26b-a4b: author-submitted v3 checkpoint (kit PR #9), adopted after a 3,000-row reproduction matched it bit for bit; v1 stays on the 0.2 board."},{"retired":"reflex-27b","replaced_by":"reflex-27b-v2","why":"author-submitted full-coverage rerun with wide-choice one-token labels (kit PR #13), adopted after our latency-v1 run agreed with it within reflex's own run-to-run noise; v1 stays on the 0.2 board","line":"reflex 27B (reflex-27b-v2) replaces reflex-27b: author-submitted full-coverage rerun with wide-choice one-token labels (kit PR #13), adopted after our latency-v1 run agreed with it within reflex's own run-to-run noise; v1 stays on the 0.2 board."},{"retired":"hopper","replaced_by":"hopper-g-1.2","why":"Hopper (G) 1.2, the author-submitted run of the next Hopper adapter (same Qwen3.5-4B base, continued from Hopper 1.0's adapter, same 1.1.1 serving code and map; kit PR #17), adopted after our latency-v1 run agreed with it within the A10G-vs-RTX noise; 1.1.1 stays on the 0.2 board","line":"Hopper (G) 1.2 (hopper-g-1.2) replaces hopper: Hopper (G) 1.2, the author-submitted run of the next Hopper adapter (same Qwen3.5-4B base, continued from Hopper 1.0's adapter, same 1.1.1 serving code and map; kit PR #17), adopted after our latency-v1 run agreed with it within the A10G-vs-RTX noise; 1.1.1 stays on the 0.2 board."}],"removed":[{"engine":"nanojev-8k","why":"NanoJev retired; removed from the board."},{"engine":"reflex-4b","why":"Removed: the author's recommended configuration is the frozen Qwen3.5-4B base without the published LoRA adapter, and that adapter was trained on 800 MMLU-Pro test items. reflex 27B stays."},{"engine":"jqv-targeted-v2","why":"Parked; not run for 0.2."},{"engine":"bev","why":"Cancelled; not run."},{"engine":"jqv","why":"Left out of 0.2: the new index benchmarks were not run for it; its results stay on disk."}]}}}