Add shell-model comparisons with explicit benchmark protocols

README.md

@@ -39,13 +39,13 @@ The selected research checkpoint is **A3**; all quants here share that trained s
 | Model/checkpoint | Quant | ALFA-updated /300 | Pass rate | Internal /1320 |
 |---|---|---:|---:|---:|
 | **EC 1.5B A3 (this release)** | Q4_K_M | **212/300** | **70.67%** | **1119/1320** |
-| [whatisit / nl2sh-1.5b](https://huggingface.co/ThorOdinson246/nl2sh-1.5b-Q4_K_M) | Q4_K_M | 191/300 | 63.67% | Not measured |
+| [whatisit / nl2sh-1.5b](https://huggingface.co/ThorOdinson246/nl2sh-1.5b-Q4_K_M) | Q4_K_M | 191/300 | 63.67% | 82/1320 |
 | [EC 0.6B A1](https://huggingface.co/dirac-run/ec-0.6b-gguf) | Q8_0 | 174/300 | 58.00% | 922/1320 |
 | [EC 0.6B A1](https://huggingface.co/dirac-run/ec-0.6b-gguf) | Q4_K_M | 165/300 | 55.00% | 911/1320 |
 
 **Benchmark note:** This comparison uses an updated version of the ALFA benchmark,
 [ALFA-updated](https://github.com/dirac-run/ALFA-updated), with documented environment
-and correctness repairs. The GitHub source publication is pending. These are
+and correctness repairs. The benchmark source is public. These are
 locally measured results under the same frozen grading definition, not scores
 from the original ALFA grader. Benchmark errors remain in the 300-task denominator:
 one for each EC Q4 run, two for EC Q8, and five for whatisit.
@@ -54,8 +54,14 @@ Each model uses its own serving profile: EC uses the JSON system prompt below
 and a 256-token limit; whatisit uses its native plain-command prompt, a 64-token
 limit and its published repetition settings. Its saved predictions were regraded
 without changing the commands. This compares deployed configurations, not an
-experiment holding prompt and decoding settings constant. Internal results were
-not measured for whatisit.
+experiment holding prompt and decoding settings constant. The complete whatisit
+internal measurement is **82/1320 (6.21%)** across all six
+current panels. We reused 1,160 saved responses after exact ID/request and native
+extraction checks, generated the missing transfer160 responses, and graded all
+1,320 with the current accepted fixture policies. No older panel scores were pooled.
+The internal run had 1 benchmark error, retained as failed cases in the 1,320 denominator.
+The native plaintext responses were admitted through a model-specific adapter;
+they were not required to satisfy EC's JSON generation protocol.
 
 ### Published original-ALFA results (different protocol)
 
@@ -88,9 +94,30 @@ separate measurements and should not be substituted for one another.
 
 This is the A3 repair checkpoint, not the earlier parent or the find specialist. A3 Q4 scored 212/300 ALFA and 1119/1320 internal. The parent's latest repeat scored 208/1115; its earlier 210 ALFA and 1127 internal results are separate measurements, with 1127 using an older prompt. The find specialist scored 217/1056 and was not selected for this release. The differences over the parent are modest; these runs do not establish a statistically reliable improvement.
 
-| Quant | Commands /244 | Quoting /256 | Operands /512 | Time /84 | Transfers /160 | English /64 |
+| Model / quant | Commands /244 | Quoting /256 | Operands /512 | Time /84 | Transfers /160 | English /64 |
 |---|---:|---:|---:|---:|---:|---:|
-| Q4_K_M | 232 | 237 | 417 | 80 | 89 | 64 |
+| EC 1.5B A3 / Q4_K_M | 232 | 237 | 417 | 80 | 89 | 64 |
+| whatisit 1.5B / Q4_K_M | 80 | 0 | 0 | 2 | 0 | 0 |
+
+The transfer panel measures existing operation/source-versus-runtime diagnostics.
+It is the current transfer160 panel, not a sum of historical pair64/fresh96 panels.
+Whatisit inference used the pinned Q4 model and llama-server `0.4.1-dev`
+(`161755f29`), eight CPU threads, context 4096 and one slot for the new responses,
+matching the recorded historical serving profile. The time observer error was
+caught at the observer boundary so remaining cases could be measured; its case
+remains a failure. Frozen fixture functions and expected outputs were unchanged.
+
+**Suite composition:** Quoting256, operands512 and English64 account for
+**832/1320 (63.0%)** cases and test closely related literal-search/output-mode
+contracts. Transfer160 covers ten text-processing families in source-text and
+runtime-output variants; time84 tests age predicates. These are concentrated
+retention/stress diagnostics, not broad shell-task accuracy. Separate
+known-correct native plaintext controls passed 12/12 across the four zero panels,
+confirming that their commands can pass the same extraction and frozen fixtures
+without generating JSON. An actual native-template check preserved all 1,320
+requests. Four of 14 separate native replays differed from the saved commands;
+saved and newly generated responses are separately bound, and deterministic
+replay equivalence is not assumed.
 
 ALFA scores use [ALFA-updated](https://github.com/dirac-run/ALFA-updated), a documented
 variant with grading definition SHA256 `6f50e4fea37c125f73f93e5f785e95e91fda448b4bf347a9a2986db4d83b48dc`. They are not original ALFA
@@ -103,7 +130,7 @@ Only the published quants' measured results are shown; they do not imply BF16 pe
 
 ## Use with ec
 
-After the separate CLI source release, install the [EasyCommand application](https://github.com/dirac-run/ec), then download
+Install the [EasyCommand application](https://github.com/dirac-run/ec), then download
 the Q4 model from that checkout using its checksum-verifying helper:
 
 ```sh
@@ -117,7 +144,6 @@ ec --model ec-1.5b.Q4_K_M.gguf list the last five commits in this repository
 
 `--preview` prints COMMAND JSON without executing. Ordinary `ec` usage extracts
 the command, displays it and asks for confirmation before running it.
-The download paths refer to this repository after publication.
 
 ## Exact inference profile
 

SHA256SUMS

@@ -1,8 +1,8 @@
 832dd9e00a68dd83b3c3fb9f5588dad7dcf337a0db50f7d9483f310cd292e92e  LICENSE
 dbdafca8d2532be1ecdf749154f589960e5a60cdb9cf228c2e4237f8f46e732e  NOTICE
-6f5417592721fcff98c867b4522d2a834e9346cd70cf24d365eef3f1d0b2a5c9  README.md
+b06af128921e36459bfa341a9fc23b242ccecb8aabe8f6cd66712b1284d80fe8  README.md
 135c8ec1a7ad6c24ba25971ee01cb3659130676031e45d876bd1822185c9e044  ec-1.5b.Q4_K_M.gguf
-bcc6925fe032e35de2d798f566eec1cb3e16d81ecccfec83ed716078f98a4a7a  evaluation.json
+ddc69a77d61e4c8183574f07923cf38406184996d1e2c12de6a76fc2a70581a0  evaluation.json
 13cb45c431d2e38087c5ef2038fb564646e5a338fb3181c880edc9bbe5e72fd9  inference.json
 bfe4c6c58d9955bbadd83022a13fc87c978e792073df6968608fcd214ba1027f  manifest.json
 3a9028d5aebb73c3ed7363e63eb3e751689218775ea5572ab0dc6807e522238b  system-prompt.txt

evaluation.json

@@ -131,7 +131,132 @@
             "benchmark_errors": 5,
             "source_summary_sha256": "8c18864c69d8b0f9128276c92ebae31b898397e006abbbc5b81505ecc0fe7723"
           },
-          "internal": null,
+          "internal": {
+            "passed": 82,
+            "cases": 1320,
+            "benchmark_errors": 1,
+            "complete": true,
+            "panels_complete": 6,
+            "errors_remain_in_denominator": true,
+            "source_summary_sha256": "4b3bf77f6c73bf456d9b59895003e91bddf90b8a5fc29c851374becf2c8493cd",
+            "case_results_sha256": "e9cf47ea089d0e8a10699f7bfa67cae7c388116333b715ebee396ce86fe333d5",
+            "source_report": "reports/whatisit-internal-full-suite-20261005",
+            "canonical_assets_manifest_sha256": "d837b2912826106e46b99678f0fe5775a04ee1e484be1f8ebece0eedb39e16a9",
+            "historical_predictions_reused": 1160,
+            "current_predictions_generated": 160,
+            "all_1320_regraded_with_current_accepted_fixture_policies": true,
+            "historical_scores_pooled": false,
+            "panels": {
+              "main244": {
+                "passed": 80,
+                "cases": 244,
+                "benchmark_errors": 0,
+                "source_summary_sha256": "40e7cf877876a1f7393e0620ffb71d35d25e2d30213258dcb74738d45e257985",
+                "case_results_sha256": "d44781f2011877485bc0136efa24f73d0b107354ec89933018ba080ea797d2c7",
+                "requests_sha256": "7cca0a313c12da3ee15ea3edcffda229132e78736eb371813fc2be8d5419b2c2",
+                "predictions_sha256": "d1b2e2cfb8af64112dc4a6877009c88b41d14a07a6073efb31fc21ecda08d21e",
+                "frozen_grader_sha256": "ceb746610a02075788b14d32b96abfb0d7908accd8f9891d79bd9e4031f5a267"
+              },
+              "quotes256": {
+                "passed": 0,
+                "cases": 256,
+                "benchmark_errors": 0,
+                "source_summary_sha256": "e3cd67b428fa8a446c298a08ed3951a2044f83ddb914507aa1a72bbadb56bb77",
+                "case_results_sha256": "c9ae61b4434cfcea0b0d0a688913fe6a11c9a81c485687d1f7df4e6f8fd1dd95",
+                "requests_sha256": "ff775d6acb0b6993e948558580e7a955b451f5a5a7d9c44022f3b910f3c5f066",
+                "predictions_sha256": "d376dee15bff2d532b8518f7a25cd1ef6692958ceef6f1b098ba6f8411c762a4",
+                "frozen_grader_sha256": "53b0b0bf6ca34adb5e031fcf810e6708bd8c45055c545b12cdf1550b5017a383"
+              },
+              "operands512": {
+                "passed": 0,
+                "cases": 512,
+                "benchmark_errors": 0,
+                "source_summary_sha256": "04c42f73f8277e689869d348776088aeac31dde3d2361bdffe40498cb50b3d78",
+                "case_results_sha256": "80f3859471f056ef355a9bc9a9706ee889f9eba59e99631802cf802a7daddfb2",
+                "requests_sha256": "11854cf537bacab00eb55d8b4b9199af0ad51b4343a3c9bc2b1bf29433e02d86",
+                "predictions_sha256": "f61ab81e40a1ab46bed9d42e93b35929f9f5cd92852b8856e6c37c773d516606",
+                "frozen_grader_sha256": "e4d8ec88c6f5dc84ddea0f31b3c54deaa5624ef6a047f7562d622de791716731"
+              },
+              "time84": {
+                "passed": 2,
+                "cases": 84,
+                "benchmark_errors": 1,
+                "source_summary_sha256": "c04af1d6c3344a2dff5cf8833bd61e7251d1344b23725379c2ecec38cd4e52c6",
+                "case_results_sha256": "a97396636fbe7e876973a484d7ddce99386dcbc449acd242b79e6d07e33bf83b",
+                "requests_sha256": "97dbc621e241b41cecdc829c78a86804cd7be11b9210e63b1adf6c50343d3985",
+                "predictions_sha256": "f3c3106d5db6a442b49f22a72fa11f255432f631b83eeffb422f91c80cb6ca53",
+                "frozen_grader_sha256": "a34436d8d2d687620b52ce3dbc0126a54be9bd9659afa6b879b1b80cedc9790c"
+              },
+              "transfer160": {
+                "passed": 0,
+                "cases": 160,
+                "benchmark_errors": 0,
+                "source_summary_sha256": "3a80e73ca0a60e58348885064a63b13ea21afd740cb370ccaaed23a1b828ac2b",
+                "case_results_sha256": "5c9b2263f84d7b2d995bdb1f1db475df98c40900438442fdcf83d4d141cd40fb",
+                "requests_sha256": "8cafbf2e3903a6a6ae805db0fcd3d0bdb19ac52df98796f259696d6663083dfd",
+                "predictions_sha256": "2d3bb2ac389a84c3116482bbc4e1d9bcd979f84cb615b222806bb7116b7d8beb",
+                "frozen_grader_sha256": "07db981ca4d9c828c5d42c6bd33ba31a84aa7fdebae16600b23b57edc4057d57"
+              },
+              "English64": {
+                "passed": 0,
+                "cases": 64,
+                "benchmark_errors": 0,
+                "source_summary_sha256": "4a916862d6b6487a85e443db58d3c7b75588e72b8404ab724ffc1c906c4d4f8b",
+                "case_results_sha256": "28fd20dcd53e247ba22698c869bfdfce148ab2d8991162e1dc5e4180f5af5a9a",
+                "requests_sha256": "d37d8bbb27b2c11f175ebb549f89a56e48b37138cbfc95e0ab4caf3db826438d",
+                "predictions_sha256": "7de4b0ecc86e470c894e82f83cc31d2ca1c123cf28acf7bbaf1e26bbde3b99af",
+                "frozen_grader_sha256": "a3f0de0e1b5d5ee390a04c4644fc9a22c5bdf2b0fa211f32e9cb8a34ff32852d"
+              }
+            },
+            "admission": "Native plaintext first-line extraction; COMMAND envelope is only a fixture interchange representation, not generated JSON.",
+            "observer_boundary": "Exception counted as failed observation; original qualified fixture/oracle functions and expected outputs unchanged.",
+            "scope": "Six public development retention panels; related templates, not independent unseen-task population accuracy.",
+            "runtime_and_fixture_bindings_sha256": "90a699a4540a3d5782fe03c69eeaa24bacb0fe2099af73d540c5181741e3f235",
+            "runtime": {
+              "backend": "llama-server native GGUF chat completions",
+              "version": "0.4.1-dev",
+              "commit": "161755f29",
+              "server_sha256": "234b05b2138264f8fb263c3205e85f4c290e8afe5067e280a4f6f90cdac5696b",
+              "threads": 8,
+              "context": 4096,
+              "parallel_slots": 1,
+              "CPU": true,
+              "host_context": false,
+              "historical_and_current_generation_separately_bound": true
+            },
+            "suite_composition": {
+              "related_literal_search_panels": [
+                "quotes256",
+                "operands512",
+                "English64"
+              ],
+              "related_literal_search_cases": 832,
+              "total_cases": 1320,
+              "related_literal_search_fraction": 0.6303030303030303,
+              "transfer160": "Ten text-processing families in source-text/runtime-output variants",
+              "time84": "Age predicates",
+              "scope": "Concentrated retention/stress diagnostics; not broad shell-task accuracy"
+            },
+            "separate_plaintext_controls": {
+              "passed": 12,
+              "cases": 12,
+              "model_inference": false,
+              "raw_JSON_answers": false,
+              "source_summary_sha256": "164ca5db3661885decd6ca50289e5b0acbfee0a7dea9216e0e5480c241ead5d6",
+              "original_model_measurement_changed": false
+            },
+            "separate_template_replay_audit": {
+              "native_rendered_requests_checked": 1320,
+              "complete_user_requests_preserved": 1320,
+              "selected_replays": 14,
+              "matching_raw_and_commands": 10,
+              "matching_new_transfer_samples": 3,
+              "new_transfer_samples": 3,
+              "historical_prompt_logs_available": false,
+              "source_summary_sha256": "a7769ce1d03b892ba9658ad23e35f7d053b896c78ce9e05b6dfbfcc953061d6b",
+              "deterministic_replay_equivalence_assumed": false
+            }
+          },
           "serving_profile": {
             "system": "You are a shell command generator. Output exactly one line: a single POSIX/bash command that accomplishes the user's request. No prose, no markdown fences, no explanation.",
             "temperature": 0,