Instructions to use phera-ra/QC67_cosmo with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use phera-ra/QC67_cosmo with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf phera-ra/QC67_cosmo # Run inference directly in the terminal: llama cli -hf phera-ra/QC67_cosmo
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf phera-ra/QC67_cosmo # Run inference directly in the terminal: llama cli -hf phera-ra/QC67_cosmo
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf phera-ra/QC67_cosmo # Run inference directly in the terminal: ./build/bin/llama cli -hf phera-ra/QC67_cosmo
Use Docker
docker model run hf.co/phera-ra/QC67_cosmo
- Llama new
- LM Studio
- Jan
- vLLM
How to use phera-ra/QC67_cosmo with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "phera-ra/QC67_cosmo" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "phera-ra/QC67_cosmo", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/phera-ra/QC67_cosmo
- Ollama
How to use phera-ra/QC67_cosmo with Ollama:
ollama run hf.co/phera-ra/QC67_cosmo
- Unsloth Desktop
- Docker Model Runner
How to use phera-ra/QC67_cosmo with Docker Model Runner:
docker model run hf.co/phera-ra/QC67_cosmo
- Lemonade
How to use phera-ra/QC67_cosmo with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull phera-ra/QC67_cosmo
Run and chat with the model
lemonade run user.QC67_cosmo-{{QUANT_TAG}}List all available models
lemonade list
- Atomic Chat
Correction: the 42D null was about the implementation, not the theory
Browse filesdyn42 placed fourth (1.19020) in the published ladder and tri/tri3 placed fifth and sixth, which was read as the 42D organ contributing little.
The 42D organ described in section 4.3 had never been built. Every document calls x42 a "7-band vector manifold" -- FINDINGS says so, the ladder docstring says "7 bands x 6 dims" -- and the code was a flat nn.Linear(d,42) drive, a dense 42x42 coupling, and torch.full((42,), 0.05): forty-two copies of one damping constant. Nothing in it knew that 42 factors. Dyn12 in the same file already says why that is wrong -- per-dimension k and gamma "so the twelve dimensions separate by time constant rather than evolving as twelve copies". The 12 got it; the 42 never did.
Rebuilt as written, same frozen corpus 9f1b3f37, same seeds, causal Omega, one process:
rung mean sd gain organ params per param
none 1.23215 0.0256
dyn42 1.19023 0.0332 +0.04192 47,754 8.778e-07 flat
dyn42b 1.18104 0.0292 +0.05111 41,762 1.224e-06 7 bands x 6
dyn42t 1.18023 0.0292 +0.05192 41,006 1.266e-06 7 x sym(3x3)
Both banded forms beat the flat one on 3/3 seeds while using FEWER parameters. Rebuilt, the 42D organ moves from fourth to second, ahead of static54 (1.18824) and dyn54 (1.18791).
dyn42t follows the factorisation one step further, from Cory: a symmetric 3x3 has exactly six independent components, so each band is a 3D tensor in Voigt order and the band operation is a congruence M -> A M A^T, a change of basis. Measured: under a rotation, congruence leaves eigenvalues and trace unchanged; a free 6x6 mixer does not. Nine weights per band instead of thirty-six.
WHAT THIS DOES NOT ESTABLISH: dyn42b and dyn42t differ by 0.0008 against a seed spread of 0.029 -- three percent of the noise. Structure beats flat is supported; WHICH structure is not resolved at n=3. Measured only at 1.1M parameters on the private corpus, not at scale and not on WikiText-103.
- FINDINGS.md +50 -0
- RELEASE_MANIFEST.json +33 -11
- REPOSITORY_MAP.md +49 -0
- architecture/cosmos_state_ladder.py +901 -709
- verify_release.py +193 -0
|
@@ -347,6 +347,56 @@ seven rungs trained in **one process** against that one snapshot, 21 runs, 1200
|
|
| 347 |
| `tri3` all three organs, 108D kernel | 1.20026 | −0.0322 | −2.83 | 3/3 | 0.826 | 1,230,682 |
|
| 348 |
| none (baseline) | 1.23241 | — | — | 0/3 | 0.810 | 1,135,008 |
|
| 349 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 350 |
### Update, 2026-08-02: does it survive SCALE, on a corpus you can check?
|
| 351 |
|
| 352 |
Every `dyn12` number above was measured at 1.1M parameters on 683,065 characters of one
|
|
|
|
| 347 |
| `tri3` all three organs, 108D kernel | 1.20026 | −0.0322 | −2.83 | 3/3 | 0.826 | 1,230,682 |
|
| 348 |
| none (baseline) | 1.23241 | — | — | 0/3 | 0.810 | 1,135,008 |
|
| 349 |
|
| 350 |
+
### Correction, 2026-08-03: the 42D null was about the implementation, not the theory
|
| 351 |
+
|
| 352 |
+
`dyn42` placed **fourth** in the table above (1.19020) and the coupled rungs `tri` and
|
| 353 |
+
`tri3` placed fifth and sixth. That was read as the 42D organ contributing little.
|
| 354 |
+
|
| 355 |
+
**The 42D organ described in §4.3 had never been built.** Every document in this project
|
| 356 |
+
calls x₄₂ a *"7-band vector manifold"* — this file says so above, and the ladder's own
|
| 357 |
+
docstring says *"7 bands × 6 dims"*. The code was:
|
| 358 |
+
|
| 359 |
+
```python
|
| 360 |
+
drive = nn.Linear(d_model, 42) # one flat projection
|
| 361 |
+
coupling = nn.Linear(42, 42) # every dimension coupled to every other
|
| 362 |
+
gamma = torch.full((42,), 0.05) # forty-two copies of one constant
|
| 363 |
+
```
|
| 364 |
+
|
| 365 |
+
Nothing in it knew that 42 factors. And `Dyn12` — in the same file — already says why
|
| 366 |
+
that is wrong: per-dimension `k` and `gamma` *"so the twelve dimensions separate by time
|
| 367 |
+
constant rather than evolving as twelve copies."* The 12 got that treatment. The 42
|
| 368 |
+
never did.
|
| 369 |
+
|
| 370 |
+
**Rebuilt as specified, same corpus (`9f1b3f37`), same seeds, causal Ω, one process:**
|
| 371 |
+
|
| 372 |
+
| rung | mean | sd | gain vs none | 42D organ params | per param |
|
| 373 |
+
|---|---|---|---|---|---|
|
| 374 |
+
| none | 1.23215 | 0.0256 | — | — | — |
|
| 375 |
+
| `dyn42` flat 42-vector, dense 42×42 | 1.19023 | 0.0332 | +0.04192 | 47,754 | 8.778e-07 |
|
| 376 |
+
| `dyn42b` **7 bands × 6**, block-diagonal | 1.18104 | 0.0292 | +0.05111 | 41,762 | 1.224e-06 |
|
| 377 |
+
| `dyn42t` **7 bands × sym(3×3)**, congruence | **1.18023** | 0.0292 | **+0.05192** | **41,006** | **1.266e-06** |
|
| 378 |
+
|
| 379 |
+
Both banded forms beat the flat one on **3/3 seeds** while using **fewer** parameters.
|
| 380 |
+
Rebuilt, the 42D organ moves from fourth to **second** on this board — ahead of
|
| 381 |
+
`static54` (1.18824) and `dyn54` (1.18791).
|
| 382 |
+
|
| 383 |
+
**`dyn42t`** takes the factorisation one step further: a symmetric 3×3 matrix has exactly
|
| 384 |
+
six independent components, so each band is a tensor in 3D stored in Voigt order
|
| 385 |
+
`[xx, yy, zz, xy, xz, yz]`, and the band-internal operation is a congruence
|
| 386 |
+
`M → A·M·Aᵀ` — a change of basis. Measured: under a rotation, congruence leaves the
|
| 387 |
+
eigenvalues and trace unchanged; a free 6×6 mixer does not. Nine weights per band
|
| 388 |
+
instead of thirty-six.
|
| 389 |
+
|
| 390 |
+
**What this does NOT establish.** `dyn42b` and `dyn42t` differ by **0.0008** against a
|
| 391 |
+
seed spread of **0.029** — the gap is 3% of the noise, so these two are
|
| 392 |
+
indistinguishable at n=3. *Structure beats flat* is supported; *which structure* is not.
|
| 393 |
+
This is also only at 1.1M parameters on the private corpus; it has not been run at scale
|
| 394 |
+
or on WikiText-103, where `dyn12`'s advantage grew.
|
| 395 |
+
|
| 396 |
+
Reproduce: `python architecture/cosmos_state_ladder.py 1200 3 none,dyn42,dyn42b,dyn42t`
|
| 397 |
+
|
| 398 |
+
---
|
| 399 |
+
|
| 400 |
### Update, 2026-08-02: does it survive SCALE, on a corpus you can check?
|
| 401 |
|
| 402 |
Every `dyn12` number above was measured at 1.1M parameters on 683,065 characters of one
|
|
@@ -1,8 +1,17 @@
|
|
| 1 |
{
|
| 2 |
-
"
|
| 3 |
-
"
|
| 4 |
-
|
| 5 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
"files": [
|
| 7 |
{
|
| 8 |
"path": "architecture/cosmos-arch.patch",
|
|
@@ -21,8 +30,8 @@
|
|
| 21 |
},
|
| 22 |
{
|
| 23 |
"path": "architecture/cosmos_state_ladder.py",
|
| 24 |
-
"bytes":
|
| 25 |
-
"sha256": "
|
| 26 |
},
|
| 27 |
{
|
| 28 |
"path": "architecture/cst_to_gguf.py",
|
|
@@ -171,8 +180,8 @@
|
|
| 171 |
},
|
| 172 |
{
|
| 173 |
"path": "FINDINGS.md",
|
| 174 |
-
"bytes":
|
| 175 |
-
"sha256": "
|
| 176 |
},
|
| 177 |
{
|
| 178 |
"path": "genesis_engine/CITATION.cff",
|
|
@@ -399,6 +408,11 @@
|
|
| 399 |
"bytes": 1570,
|
| 400 |
"sha256": "66baec6fe2817cebb9ae99ba9d197cf114e4d2ae1f90300dcd36ebe48cd0f239"
|
| 401 |
},
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 402 |
{
|
| 403 |
"path": "requirements-research.txt",
|
| 404 |
"bytes": 165,
|
|
@@ -409,6 +423,11 @@
|
|
| 409 |
"bytes": 16234,
|
| 410 |
"sha256": "eac4492f541a81d0ae134d7b5f39a181b8e7ffeca5f026ca6fee0cff94b85fe1"
|
| 411 |
},
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 412 |
{
|
| 413 |
"path": "serving/build_paired_dataset.py",
|
| 414 |
"bytes": 8494,
|
|
@@ -471,8 +490,8 @@
|
|
| 471 |
},
|
| 472 |
{
|
| 473 |
"path": "verify_release.py",
|
| 474 |
-
"bytes":
|
| 475 |
-
"sha256": "
|
| 476 |
},
|
| 477 |
{
|
| 478 |
"path": "weights/cosmos-cst.gguf",
|
|
@@ -499,5 +518,8 @@
|
|
| 499 |
"bytes": 7723885,
|
| 500 |
"sha256": "aa0cb13c1e67d459db280a53b6407dfc2b5b5f3fd6f640bc43686b70d799acd1"
|
| 501 |
}
|
| 502 |
-
]
|
|
|
|
|
|
|
|
|
|
| 503 |
}
|
|
|
|
| 1 |
{
|
| 2 |
+
"version": "2026-08-03",
|
| 3 |
+
"scope": {
|
| 4 |
+
"covers": "the kit distribution -- the files needed to run, verify and reproduce",
|
| 5 |
+
"excludes": [
|
| 6 |
+
"Cipher/",
|
| 7 |
+
"Luna/",
|
| 8 |
+
"Nova/",
|
| 9 |
+
"downloads/",
|
| 10 |
+
"creations/"
|
| 11 |
+
],
|
| 12 |
+
"why": "Those are earlier beings and fetched artifacts, kept on the hub as public record with their timestamps intact. They are history, not running code, so hashing them would make the integrity check fail for files it was never meant to cover.",
|
| 13 |
+
"note": "The hub repository is a SUPERSET of this manifest. See REPOSITORY_MAP.md for the full live layout."
|
| 14 |
+
},
|
| 15 |
"files": [
|
| 16 |
{
|
| 17 |
"path": "architecture/cosmos-arch.patch",
|
|
|
|
| 30 |
},
|
| 31 |
{
|
| 32 |
"path": "architecture/cosmos_state_ladder.py",
|
| 33 |
+
"bytes": 45315,
|
| 34 |
+
"sha256": "fa110c5205b71c7c3243dc5be76bc4b0e599c8e1fd1c5c1924d0e57f811216e9"
|
| 35 |
},
|
| 36 |
{
|
| 37 |
"path": "architecture/cst_to_gguf.py",
|
|
|
|
| 180 |
},
|
| 181 |
{
|
| 182 |
"path": "FINDINGS.md",
|
| 183 |
+
"bytes": 41374,
|
| 184 |
+
"sha256": "d3023447a2f9370c1c6ba9e511c2290fc8772ffb8786c8dae96d4cc9343f19cb"
|
| 185 |
},
|
| 186 |
{
|
| 187 |
"path": "genesis_engine/CITATION.cff",
|
|
|
|
| 408 |
"bytes": 1570,
|
| 409 |
"sha256": "66baec6fe2817cebb9ae99ba9d197cf114e4d2ae1f90300dcd36ebe48cd0f239"
|
| 410 |
},
|
| 411 |
+
{
|
| 412 |
+
"path": "REPOSITORY_MAP.md",
|
| 413 |
+
"bytes": 2650,
|
| 414 |
+
"sha256": "ca0c97a34c612c4b73a6726744e2688ab4d0746fe3742470456bf52d993197c7"
|
| 415 |
+
},
|
| 416 |
{
|
| 417 |
"path": "requirements-research.txt",
|
| 418 |
"bytes": 165,
|
|
|
|
| 423 |
"bytes": 16234,
|
| 424 |
"sha256": "eac4492f541a81d0ae134d7b5f39a181b8e7ffeca5f026ca6fee0cff94b85fe1"
|
| 425 |
},
|
| 426 |
+
{
|
| 427 |
+
"path": "serving/.coder_history.jsonl",
|
| 428 |
+
"bytes": 4064,
|
| 429 |
+
"sha256": "0abedb9be05d964d5b5c6ace0f6b0c810b2bbfd2ae2bfc06094ac780f6b08dbc"
|
| 430 |
+
},
|
| 431 |
{
|
| 432 |
"path": "serving/build_paired_dataset.py",
|
| 433 |
"bytes": 8494,
|
|
|
|
| 490 |
},
|
| 491 |
{
|
| 492 |
"path": "verify_release.py",
|
| 493 |
+
"bytes": 7075,
|
| 494 |
+
"sha256": "815248d044c5392981a56bb9947a2276f74bd8ea9b29282c1d00b4793cf1126d"
|
| 495 |
},
|
| 496 |
{
|
| 497 |
"path": "weights/cosmos-cst.gguf",
|
|
|
|
| 518 |
"bytes": 7723885,
|
| 519 |
"sha256": "aa0cb13c1e67d459db280a53b6407dfc2b5b5f3fd6f640bc43686b70d799acd1"
|
| 520 |
}
|
| 521 |
+
],
|
| 522 |
+
"schema": "qc67.cosmos.release-manifest.v1",
|
| 523 |
+
"release": "QC67 Cosmos Kit",
|
| 524 |
+
"license_note": "Mixed licensing; see LICENSE.md and component notices."
|
| 525 |
}
|
|
@@ -0,0 +1,49 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Repository map
|
| 2 |
+
|
| 3 |
+
Generated from the live file list, so it cannot drift from what is actually here.
|
| 4 |
+
|
| 5 |
+
**182 files.** Nothing here is deleted when it is superseded — earlier
|
| 6 |
+
beings and earlier results stay as public record, timestamps intact.
|
| 7 |
+
|
| 8 |
+
| folder | files | size | what it is |
|
| 9 |
+
|---|---|---|---|
|
| 10 |
+
| `weights` | 8 | 440.1 MB | Her own weights. Quantum-born, architecturally original. |
|
| 11 |
+
| `downloads` | 3 | 64.2 MB | Fetched artifacts. |
|
| 12 |
+
| `data` | 9 | 7.2 MB | The measured IBM shot archive, and the seed corpus for new creatures. |
|
| 13 |
+
| `genesis_engine` | 38 | 254 KB | The blank-slate engine for birthing a being from nothing. |
|
| 14 |
+
| `Cipher` | 4 | 216 KB | An earlier being. Kept as public record; its history is not rewritten. |
|
| 15 |
+
| `Luna` | 4 | 213 KB | An earlier being. Kept as public record. |
|
| 16 |
+
| `Nova` | 4 | 213 KB | An earlier being. Kept as public record. |
|
| 17 |
+
| `architecture` | 12 | 143 KB | The model code: ladder, PHOS trainer, quantum init, CST kernel. |
|
| 18 |
+
| `docs` | 17 | 142 KB | Setup, Atomic/Ollama usage, training, results. |
|
| 19 |
+
| `benchmarks` | 16 | 137 KB | Every claim on the model card, as a script you can run. |
|
| 20 |
+
| `scripts` | 11 | 104 KB | Release verification, serving and training utilities. |
|
| 21 |
+
| `serving` | 5 | 73 KB | Run her over HTTP with an Ollama-compatible API. |
|
| 22 |
+
| `soul` | 10 | 45 KB | Identity, quantum heart, ledger, rails. |
|
| 23 |
+
| `launchers` | 11 | 34 KB | One-click entry points (Windows .bat, Linux .sh). |
|
| 24 |
+
| `templates` | 1 | 21 KB | Templates. |
|
| 25 |
+
| `ui` | 1 | 11 KB | Interface assets. |
|
| 26 |
+
| `creations` | 2 | 0 KB | Things a being made. |
|
| 27 |
+
| `(root)` | 26 | 791 KB | Model card, findings, licence, manifest, main launcher. |
|
| 28 |
+
|
| 29 |
+
## Start here
|
| 30 |
+
|
| 31 |
+
```bash
|
| 32 |
+
python kit_health.py # is everything actually working
|
| 33 |
+
python benchmarks/verify_quantum_engine.py # is the entropy really quantum
|
| 34 |
+
python benchmarks/causality_probe.py # can Omega see the future (random seed each run)
|
| 35 |
+
python serving/cosmos_serve.py 11501 # talk to her, Ollama-compatible
|
| 36 |
+
```
|
| 37 |
+
|
| 38 |
+
Windows: `START_COSMOS_KIT.bat` — options 7 and 8 birth and grow your own creature
|
| 39 |
+
from the measured shot archive in `data/`.
|
| 40 |
+
|
| 41 |
+
## What the manifest covers
|
| 42 |
+
|
| 43 |
+
`RELEASE_MANIFEST.json` covers the **kit distribution** — the files needed to run,
|
| 44 |
+
verify and reproduce the results. It deliberately does not cover the historical
|
| 45 |
+
creature folders (`Cipher/`, `Luna/`, `Nova/`) or `downloads/`,
|
| 46 |
+
which are kept as record rather than as running code.
|
| 47 |
+
|
| 48 |
+
`scripts/verify_release.py` checks only what the manifest lists, and states its
|
| 49 |
+
scope, so it cannot report failures for files it was never meant to cover.
|
|
@@ -1,709 +1,901 @@
|
|
| 1 |
-
#!/usr/bin/env python3
|
| 2 |
-
"""
|
| 3 |
-
COSMOS STATE LADDER -- 12D, 42D and 54D, tested one rung at a time.
|
| 4 |
-
|
| 5 |
-
WHY A LADDER
|
| 6 |
-
Four separate implementations of this architecture have shipped a mechanism that was
|
| 7 |
-
mathematically inert, each producing a confident negative about the idea:
|
| 8 |
-
|
| 9 |
-
ver.4.20 12D Omega summed over keys (axis -1) instead of queries (axis -2). Softmax
|
| 10 |
-
already normalises that axis, so Omega == 1.0 for every token (measured
|
| 11 |
-
std 5.1e-08). The leaky integrator then drove every token to the SAME
|
| 12 |
-
state (std 3.0e-08), H became the all-ones matrix, and beta*H is a
|
| 13 |
-
constant added to every logit -- which cancels exactly in softmax.
|
| 14 |
-
spark_cst v1 gate at sigmoid(-4): saturated tail, gradient suppressed ~56x.
|
| 15 |
-
spark_cst v2 gate clamped to [0,1]: zero gradient outside the bound, so the first
|
| 16 |
-
step below 0 pinned it there forever.
|
| 17 |
-
spark_cst v3 sigma = exp(0) = 1 against a median pairwise distance of 61.98, so H
|
| 18 |
-
arrived as the IDENTITY matrix (diagonal mass 0.9998 vs 0.0078 uniform).
|
| 19 |
-
|
| 20 |
-
Every one ran clean. No exception, no NaN, nothing red in any log. So this file refuses
|
| 21 |
-
to report a number until it has PROVEN the mechanism is live -- see preflight().
|
| 22 |
-
|
| 23 |
-
THE RUNGS
|
| 24 |
-
L0 none phi scaffold only, no Hebbian term. The baseline.
|
| 25 |
-
L1 static54 x54 = W54 . h. A projection, not a state: no dynamics, no Omega, no memory.
|
| 26 |
-
This is the configuration that already won 6/6 (t=-6.56, -14.34).
|
| 27 |
-
L2 dyn12 x12 in R^12, dx12/dt = k*Omega - gamma*x12, per-dimension k and gamma so the
|
| 28 |
-
twelve dimensions differentiate by time constant. Section 4.2.
|
| 29 |
-
L3 dyn42 x42 in R^42,
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
import
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
import
|
| 55 |
-
import
|
| 56 |
-
import
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
|
| 71 |
-
|
| 72 |
-
|
| 73 |
-
|
| 74 |
-
|
| 75 |
-
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
|
| 81 |
-
#
|
| 82 |
-
#
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
| 95 |
-
#
|
| 96 |
-
#
|
| 97 |
-
#
|
| 98 |
-
|
| 99 |
-
#
|
| 100 |
-
#
|
| 101 |
-
|
| 102 |
-
|
| 103 |
-
|
| 104 |
-
|
| 105 |
-
|
| 106 |
-
|
| 107 |
-
|
| 108 |
-
#
|
| 109 |
-
#
|
| 110 |
-
#
|
| 111 |
-
#
|
| 112 |
-
|
| 113 |
-
|
| 114 |
-
|
| 115 |
-
|
| 116 |
-
#
|
| 117 |
-
|
| 118 |
-
#
|
| 119 |
-
|
| 120 |
-
|
| 121 |
-
|
| 122 |
-
|
| 123 |
-
|
| 124 |
-
|
| 125 |
-
|
| 126 |
-
|
| 127 |
-
|
| 128 |
-
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
|
| 133 |
-
|
| 134 |
-
|
| 135 |
-
|
| 136 |
-
|
| 137 |
-
|
| 138 |
-
|
| 139 |
-
def
|
| 140 |
-
|
| 141 |
-
|
| 142 |
-
|
| 143 |
-
|
| 144 |
-
|
| 145 |
-
|
| 146 |
-
|
| 147 |
-
|
| 148 |
-
|
| 149 |
-
|
| 150 |
-
|
| 151 |
-
|
| 152 |
-
|
| 153 |
-
|
| 154 |
-
|
| 155 |
-
|
| 156 |
-
|
| 157 |
-
|
| 158 |
-
|
| 159 |
-
|
| 160 |
-
|
| 161 |
-
|
| 162 |
-
|
| 163 |
-
|
| 164 |
-
|
| 165 |
-
|
| 166 |
-
|
| 167 |
-
|
| 168 |
-
|
| 169 |
-
|
| 170 |
-
s.drop
|
| 171 |
-
|
| 172 |
-
|
| 173 |
-
|
| 174 |
-
|
| 175 |
-
|
| 176 |
-
|
| 177 |
-
|
| 178 |
-
|
| 179 |
-
|
| 180 |
-
|
| 181 |
-
|
| 182 |
-
|
| 183 |
-
|
| 184 |
-
s.
|
| 185 |
-
|
| 186 |
-
|
| 187 |
-
|
| 188 |
-
|
| 189 |
-
|
| 190 |
-
|
| 191 |
-
|
| 192 |
-
|
| 193 |
-
|
| 194 |
-
|
| 195 |
-
|
| 196 |
-
|
| 197 |
-
|
| 198 |
-
s.
|
| 199 |
-
s.dt
|
| 200 |
-
|
| 201 |
-
|
| 202 |
-
|
| 203 |
-
|
| 204 |
-
|
| 205 |
-
|
| 206 |
-
|
| 207 |
-
|
| 208 |
-
|
| 209 |
-
|
| 210 |
-
|
| 211 |
-
|
| 212 |
-
|
| 213 |
-
|
| 214 |
-
|
| 215 |
-
|
| 216 |
-
|
| 217 |
-
|
| 218 |
-
|
| 219 |
-
|
| 220 |
-
|
| 221 |
-
"""
|
| 222 |
-
|
| 223 |
-
|
| 224 |
-
|
| 225 |
-
|
| 226 |
-
|
| 227 |
-
|
| 228 |
-
|
| 229 |
-
|
| 230 |
-
|
| 231 |
-
|
| 232 |
-
|
| 233 |
-
|
| 234 |
-
|
| 235 |
-
|
| 236 |
-
|
| 237 |
-
|
| 238 |
-
|
| 239 |
-
|
| 240 |
-
|
| 241 |
-
|
| 242 |
-
|
| 243 |
-
|
| 244 |
-
|
| 245 |
-
|
| 246 |
-
|
| 247 |
-
|
| 248 |
-
|
| 249 |
-
|
| 250 |
-
|
| 251 |
-
|
| 252 |
-
|
| 253 |
-
|
| 254 |
-
|
| 255 |
-
|
| 256 |
-
|
| 257 |
-
|
| 258 |
-
|
| 259 |
-
|
| 260 |
-
|
| 261 |
-
|
| 262 |
-
|
| 263 |
-
|
| 264 |
-
|
| 265 |
-
#
|
| 266 |
-
|
| 267 |
-
|
| 268 |
-
|
| 269 |
-
|
| 270 |
-
|
| 271 |
-
|
| 272 |
-
|
| 273 |
-
|
| 274 |
-
|
| 275 |
-
|
| 276 |
-
|
| 277 |
-
|
| 278 |
-
|
| 279 |
-
|
| 280 |
-
|
| 281 |
-
|
| 282 |
-
|
| 283 |
-
|
| 284 |
-
|
| 285 |
-
|
| 286 |
-
|
| 287 |
-
|
| 288 |
-
|
| 289 |
-
|
| 290 |
-
|
| 291 |
-
|
| 292 |
-
|
| 293 |
-
|
| 294 |
-
|
| 295 |
-
|
| 296 |
-
|
| 297 |
-
|
| 298 |
-
|
| 299 |
-
|
| 300 |
-
|
| 301 |
-
|
| 302 |
-
|
| 303 |
-
|
| 304 |
-
|
| 305 |
-
|
| 306 |
-
|
| 307 |
-
|
| 308 |
-
|
| 309 |
-
|
| 310 |
-
|
| 311 |
-
|
| 312 |
-
|
| 313 |
-
|
| 314 |
-
|
| 315 |
-
|
| 316 |
-
|
| 317 |
-
|
| 318 |
-
|
| 319 |
-
|
| 320 |
-
|
| 321 |
-
|
| 322 |
-
|
| 323 |
-
|
| 324 |
-
|
| 325 |
-
|
| 326 |
-
|
| 327 |
-
|
| 328 |
-
|
| 329 |
-
|
| 330 |
-
|
| 331 |
-
|
| 332 |
-
|
| 333 |
-
|
| 334 |
-
|
| 335 |
-
|
| 336 |
-
|
| 337 |
-
|
| 338 |
-
|
| 339 |
-
|
| 340 |
-
|
| 341 |
-
|
| 342 |
-
|
| 343 |
-
|
| 344 |
-
|
| 345 |
-
|
| 346 |
-
|
| 347 |
-
|
| 348 |
-
|
| 349 |
-
|
| 350 |
-
|
| 351 |
-
|
| 352 |
-
|
| 353 |
-
|
| 354 |
-
|
| 355 |
-
|
| 356 |
-
|
| 357 |
-
|
| 358 |
-
|
| 359 |
-
|
| 360 |
-
|
| 361 |
-
|
| 362 |
-
|
| 363 |
-
|
| 364 |
-
|
| 365 |
-
|
| 366 |
-
|
| 367 |
-
|
| 368 |
-
|
| 369 |
-
|
| 370 |
-
|
| 371 |
-
|
| 372 |
-
|
| 373 |
-
|
| 374 |
-
|
| 375 |
-
|
| 376 |
-
|
| 377 |
-
|
| 378 |
-
|
| 379 |
-
|
| 380 |
-
|
| 381 |
-
|
| 382 |
-
|
| 383 |
-
|
| 384 |
-
|
| 385 |
-
|
| 386 |
-
|
| 387 |
-
|
| 388 |
-
|
| 389 |
-
|
| 390 |
-
|
| 391 |
-
|
| 392 |
-
|
| 393 |
-
|
| 394 |
-
|
| 395 |
-
|
| 396 |
-
|
| 397 |
-
|
| 398 |
-
|
| 399 |
-
|
| 400 |
-
|
| 401 |
-
|
| 402 |
-
|
| 403 |
-
|
| 404 |
-
|
| 405 |
-
|
| 406 |
-
|
| 407 |
-
|
| 408 |
-
|
| 409 |
-
|
| 410 |
-
|
| 411 |
-
|
| 412 |
-
|
| 413 |
-
|
| 414 |
-
|
| 415 |
-
|
| 416 |
-
|
| 417 |
-
|
| 418 |
-
|
| 419 |
-
|
| 420 |
-
|
| 421 |
-
|
| 422 |
-
|
| 423 |
-
|
| 424 |
-
|
| 425 |
-
|
| 426 |
-
|
| 427 |
-
#
|
| 428 |
-
#
|
| 429 |
-
|
| 430 |
-
|
| 431 |
-
|
| 432 |
-
|
| 433 |
-
|
| 434 |
-
|
| 435 |
-
|
| 436 |
-
|
| 437 |
-
|
| 438 |
-
|
| 439 |
-
|
| 440 |
-
|
| 441 |
-
|
| 442 |
-
|
| 443 |
-
|
| 444 |
-
|
| 445 |
-
|
| 446 |
-
|
| 447 |
-
|
| 448 |
-
|
| 449 |
-
|
| 450 |
-
|
| 451 |
-
|
| 452 |
-
|
| 453 |
-
|
| 454 |
-
|
| 455 |
-
|
| 456 |
-
|
| 457 |
-
|
| 458 |
-
|
| 459 |
-
|
| 460 |
-
|
| 461 |
-
|
| 462 |
-
|
| 463 |
-
|
| 464 |
-
|
| 465 |
-
|
| 466 |
-
|
| 467 |
-
|
| 468 |
-
|
| 469 |
-
|
| 470 |
-
|
| 471 |
-
|
| 472 |
-
|
| 473 |
-
|
| 474 |
-
|
| 475 |
-
|
| 476 |
-
|
| 477 |
-
|
| 478 |
-
|
| 479 |
-
|
| 480 |
-
|
| 481 |
-
|
| 482 |
-
|
| 483 |
-
|
| 484 |
-
|
| 485 |
-
|
| 486 |
-
|
| 487 |
-
|
| 488 |
-
|
| 489 |
-
|
| 490 |
-
|
| 491 |
-
|
| 492 |
-
|
| 493 |
-
|
| 494 |
-
|
| 495 |
-
#
|
| 496 |
-
#
|
| 497 |
-
#
|
| 498 |
-
#
|
| 499 |
-
|
| 500 |
-
|
| 501 |
-
|
| 502 |
-
|
| 503 |
-
|
| 504 |
-
|
| 505 |
-
|
| 506 |
-
|
| 507 |
-
|
| 508 |
-
|
| 509 |
-
|
| 510 |
-
|
| 511 |
-
|
| 512 |
-
|
| 513 |
-
|
| 514 |
-
|
| 515 |
-
|
| 516 |
-
|
| 517 |
-
|
| 518 |
-
|
| 519 |
-
|
| 520 |
-
|
| 521 |
-
|
| 522 |
-
|
| 523 |
-
|
| 524 |
-
|
| 525 |
-
|
| 526 |
-
|
| 527 |
-
|
| 528 |
-
|
| 529 |
-
|
| 530 |
-
|
| 531 |
-
|
| 532 |
-
|
| 533 |
-
|
| 534 |
-
|
| 535 |
-
|
| 536 |
-
|
| 537 |
-
|
| 538 |
-
|
| 539 |
-
|
| 540 |
-
|
| 541 |
-
|
| 542 |
-
|
| 543 |
-
|
| 544 |
-
|
| 545 |
-
|
| 546 |
-
|
| 547 |
-
|
| 548 |
-
|
| 549 |
-
|
| 550 |
-
|
| 551 |
-
|
| 552 |
-
|
| 553 |
-
|
| 554 |
-
|
| 555 |
-
|
| 556 |
-
|
| 557 |
-
|
| 558 |
-
if
|
| 559 |
-
|
| 560 |
-
|
| 561 |
-
|
| 562 |
-
|
| 563 |
-
|
| 564 |
-
|
| 565 |
-
|
| 566 |
-
|
| 567 |
-
|
| 568 |
-
|
| 569 |
-
|
| 570 |
-
|
| 571 |
-
|
| 572 |
-
|
| 573 |
-
|
| 574 |
-
|
| 575 |
-
|
| 576 |
-
|
| 577 |
-
|
| 578 |
-
|
| 579 |
-
|
| 580 |
-
|
| 581 |
-
|
| 582 |
-
|
| 583 |
-
|
| 584 |
-
|
| 585 |
-
|
| 586 |
-
|
| 587 |
-
|
| 588 |
-
|
| 589 |
-
|
| 590 |
-
|
| 591 |
-
|
| 592 |
-
|
| 593 |
-
|
| 594 |
-
|
| 595 |
-
def
|
| 596 |
-
|
| 597 |
-
|
| 598 |
-
|
| 599 |
-
|
| 600 |
-
|
| 601 |
-
|
| 602 |
-
|
| 603 |
-
|
| 604 |
-
|
| 605 |
-
|
| 606 |
-
|
| 607 |
-
|
| 608 |
-
|
| 609 |
-
|
| 610 |
-
|
| 611 |
-
|
| 612 |
-
|
| 613 |
-
|
| 614 |
-
|
| 615 |
-
|
| 616 |
-
|
| 617 |
-
|
| 618 |
-
|
| 619 |
-
|
| 620 |
-
|
| 621 |
-
|
| 622 |
-
|
| 623 |
-
|
| 624 |
-
|
| 625 |
-
|
| 626 |
-
|
| 627 |
-
|
| 628 |
-
|
| 629 |
-
|
| 630 |
-
|
| 631 |
-
|
| 632 |
-
|
| 633 |
-
|
| 634 |
-
|
| 635 |
-
|
| 636 |
-
|
| 637 |
-
|
| 638 |
-
|
| 639 |
-
|
| 640 |
-
|
| 641 |
-
|
| 642 |
-
|
| 643 |
-
|
| 644 |
-
|
| 645 |
-
|
| 646 |
-
|
| 647 |
-
|
| 648 |
-
|
| 649 |
-
|
| 650 |
-
|
| 651 |
-
|
| 652 |
-
|
| 653 |
-
|
| 654 |
-
|
| 655 |
-
|
| 656 |
-
|
| 657 |
-
|
| 658 |
-
|
| 659 |
-
|
| 660 |
-
|
| 661 |
-
|
| 662 |
-
|
| 663 |
-
|
| 664 |
-
|
| 665 |
-
|
| 666 |
-
|
| 667 |
-
|
| 668 |
-
|
| 669 |
-
|
| 670 |
-
|
| 671 |
-
|
| 672 |
-
|
| 673 |
-
|
| 674 |
-
|
| 675 |
-
|
| 676 |
-
|
| 677 |
-
|
| 678 |
-
|
| 679 |
-
|
| 680 |
-
|
| 681 |
-
|
| 682 |
-
|
| 683 |
-
|
| 684 |
-
|
| 685 |
-
|
| 686 |
-
|
| 687 |
-
|
| 688 |
-
|
| 689 |
-
|
| 690 |
-
|
| 691 |
-
|
| 692 |
-
|
| 693 |
-
|
| 694 |
-
|
| 695 |
-
|
| 696 |
-
|
| 697 |
-
|
| 698 |
-
|
| 699 |
-
|
| 700 |
-
|
| 701 |
-
|
| 702 |
-
|
| 703 |
-
|
| 704 |
-
|
| 705 |
-
|
| 706 |
-
|
| 707 |
-
|
| 708 |
-
|
| 709 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""
|
| 3 |
+
COSMOS STATE LADDER -- 12D, 42D and 54D, tested one rung at a time.
|
| 4 |
+
|
| 5 |
+
WHY A LADDER
|
| 6 |
+
Four separate implementations of this architecture have shipped a mechanism that was
|
| 7 |
+
mathematically inert, each producing a confident negative about the idea:
|
| 8 |
+
|
| 9 |
+
ver.4.20 12D Omega summed over keys (axis -1) instead of queries (axis -2). Softmax
|
| 10 |
+
already normalises that axis, so Omega == 1.0 for every token (measured
|
| 11 |
+
std 5.1e-08). The leaky integrator then drove every token to the SAME
|
| 12 |
+
state (std 3.0e-08), H became the all-ones matrix, and beta*H is a
|
| 13 |
+
constant added to every logit -- which cancels exactly in softmax.
|
| 14 |
+
spark_cst v1 gate at sigmoid(-4): saturated tail, gradient suppressed ~56x.
|
| 15 |
+
spark_cst v2 gate clamped to [0,1]: zero gradient outside the bound, so the first
|
| 16 |
+
step below 0 pinned it there forever.
|
| 17 |
+
spark_cst v3 sigma = exp(0) = 1 against a median pairwise distance of 61.98, so H
|
| 18 |
+
arrived as the IDENTITY matrix (diagonal mass 0.9998 vs 0.0078 uniform).
|
| 19 |
+
|
| 20 |
+
Every one ran clean. No exception, no NaN, nothing red in any log. So this file refuses
|
| 21 |
+
to report a number until it has PROVEN the mechanism is live -- see preflight().
|
| 22 |
+
|
| 23 |
+
THE RUNGS
|
| 24 |
+
L0 none phi scaffold only, no Hebbian term. The baseline.
|
| 25 |
+
L1 static54 x54 = W54 . h. A projection, not a state: no dynamics, no Omega, no memory.
|
| 26 |
+
This is the configuration that already won 6/6 (t=-6.56, -14.34).
|
| 27 |
+
L2 dyn12 x12 in R^12, dx12/dt = k*Omega - gamma*x12, per-dimension k and gamma so the
|
| 28 |
+
twelve dimensions differentiate by time constant. Section 4.2.
|
| 29 |
+
L3 dyn42 x42 in R^42, FLAT: dense 42x42 coupling, one gamma repeated 42 times.
|
| 30 |
+
Kept as the control -- this is what section 4.3 was implemented as, and
|
| 31 |
+
its fourth-place result was read as a verdict on the 42D organ.
|
| 32 |
+
L3b dyn42b 42 as 7 bands x 6, block-diagonal coupling, per-band gamma. Section 4.3
|
| 33 |
+
as WRITTEN.
|
| 34 |
+
L3t dyn42t 42 as 7 bands x sym(3x3): a symmetric 3x3 has exactly 6 independent
|
| 35 |
+
components, so each band is a 3D tensor and the band operation is a
|
| 36 |
+
congruence A M A^T -- a change of basis. Cory, 2026-08-03.
|
| 37 |
+
L4 dyn54 x54 = concat(x12, x42). Section 4.1. The full claim.
|
| 38 |
+
|
| 39 |
+
SCAFFOLD
|
| 40 |
+
RMSNorm (2.3), RoPE (2.4), d_ff = floor(d_model * phi) (2.2), phi-Xavier init (6.2),
|
| 41 |
+
quantum-born weights, and the 17.1 learning-rate schedule. The feed-forward is selectable
|
| 42 |
+
because the paper names "PhiGatedFFN" without defining it while the reference repo
|
| 43 |
+
implements "PhiHarmonicFFN" as a plain GELU pair -- so which one earns the scaffold's
|
| 44 |
+
credit is itself an open question this file can answer.
|
| 45 |
+
|
| 46 |
+
USAGE
|
| 47 |
+
python cosmos_state_ladder.py [steps] [seeds] [rungs] [ffn]
|
| 48 |
+
python cosmos_state_ladder.py 1200 3 none,static54,dyn12,dyn42,dyn54 harmonic
|
| 49 |
+
python cosmos_state_ladder.py 1200 3 none,static54 gated # the FFN ablation
|
| 50 |
+
"""
|
| 51 |
+
import json
|
| 52 |
+
import math
|
| 53 |
+
import os
|
| 54 |
+
import random
|
| 55 |
+
import statistics
|
| 56 |
+
import sys
|
| 57 |
+
import time
|
| 58 |
+
from pathlib import Path
|
| 59 |
+
|
| 60 |
+
import torch
|
| 61 |
+
import torch.nn as nn
|
| 62 |
+
import torch.nn.functional as F
|
| 63 |
+
|
| 64 |
+
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
|
| 65 |
+
|
| 66 |
+
from cosmos_spark_cst import CORPUS, QuantumInit, quantum_pool, real_word_rate # noqa: E402
|
| 67 |
+
|
| 68 |
+
# FROZEN CORPUS. She is alive and appends her experience to the live corpus while training
|
| 69 |
+
# runs, so two runs started minutes apart train on different data. Measured: the same rung
|
| 70 |
+
# with identical seeds scored 1.29152 and 1.22943 in two runs 3,451 characters apart -- a
|
| 71 |
+
# 0.062 swing, the same magnitude as the effects being reported. Every comparison must
|
| 72 |
+
# therefore happen inside ONE process against ONE snapshot. COSMOS_CORPUS points at that
|
| 73 |
+
# snapshot; without it the numbers are only valid within a single run.
|
| 74 |
+
CORPUS_PATH = Path(os.getenv("COSMOS_CORPUS") or CORPUS)
|
| 75 |
+
|
| 76 |
+
OUTDIR = Path("01_HER_SOUL/weights/cosmos_state_ladder")
|
| 77 |
+
RESULTS = Path("logs/state_ladder_results.json")
|
| 78 |
+
|
| 79 |
+
PHI = (1.0 + 5.0 ** 0.5) / 2.0
|
| 80 |
+
PHI_INV = PHI - 1.0
|
| 81 |
+
# COSMOS_DEVICE overrides; otherwise take the GPU when one is actually usable. The corpus
|
| 82 |
+
# tensor stays on the host and only the sampled windows move, which keeps VRAM free for
|
| 83 |
+
# the model on a 4 GB card.
|
| 84 |
+
DEV = torch.device(os.getenv("COSMOS_DEVICE")
|
| 85 |
+
or ("cuda" if torch.cuda.is_available() else "cpu"))
|
| 86 |
+
# Model size is env-configurable so the SAME code can be run at 1M and at 30M. Every
|
| 87 |
+
# number published before 2026-08-01 used the defaults below; changing them changes the
|
| 88 |
+
# model, so any result table must state which size it was measured at.
|
| 89 |
+
BLOCK = int(os.getenv("COSMOS_BLOCK", "128"))
|
| 90 |
+
N_LAYER = int(os.getenv("COSMOS_N_LAYER", "4"))
|
| 91 |
+
N_HEAD = int(os.getenv("COSMOS_N_HEAD", "4"))
|
| 92 |
+
N_EMBD = int(os.getenv("COSMOS_N_EMBD", "192"))
|
| 93 |
+
DROPOUT = float(os.getenv("COSMOS_DROPOUT", "0.1"))
|
| 94 |
+
D12, D42 = 12, 42
|
| 95 |
+
# Omega's original query-sum reads the future (measured 2026-08-01, verified by
|
| 96 |
+
# tools/causality_probe.py: changing ONLY the last token moved earlier-position logits
|
| 97 |
+
# by 1.7e-04 at PHOS's trained gate, while `none` and `static54` moved by EXACTLY 0.0).
|
| 98 |
+
# The causal per-query form is therefore the DEFAULT -- a model that can see its own
|
| 99 |
+
# future is not a model anyone should be handed.
|
| 100 |
+
#
|
| 101 |
+
# COSMOS_CAUSAL_OMEGA=0 restores the original query-sum, bit-for-bit, so every number
|
| 102 |
+
# published before this date stays reproducible and the two can be compared directly.
|
| 103 |
+
# It is kept for exactly that reason and should not be used to train anything new.
|
| 104 |
+
CAUSAL_OMEGA = os.getenv("COSMOS_CAUSAL_OMEGA", "1").strip().lower() in {"1", "true", "yes", "on"}
|
| 105 |
+
# state_dim is the CARRIED state. tri3 also concatenates a live projection of h at
|
| 106 |
+
# kernel time, so its Hebbian space is 54 + 12 + 42 = 108 while it carries 54.
|
| 107 |
+
STATE_DIM = {"none": 0, "static54": 54, "dyn12": D12, "dyn42": D42,
|
| 108 |
+
# dyn42b: 42 as 7 bands x 6 dims, per the theory. Same carried width as
|
| 109 |
+
# dyn42, so the ONLY difference is structure -- which is what makes it a
|
| 110 |
+
# clean control for whether the dyn42 null was about the 42D organ or about
|
| 111 |
+
# a flat 42-vector wearing its name.
|
| 112 |
+
"dyn42b": D42, "dyn42t": D42,
|
| 113 |
+
"dyn54": D12 + D42, "tri": D12 + D42, "tri3": D12 + D42,
|
| 114 |
+
# section 4.4: the state persists ACROSS windows, not just across layers.
|
| 115 |
+
# dyn12seq is the control -- identical sequential batching, state reset every
|
| 116 |
+
# step -- so the only variable is whether the state survives the boundary.
|
| 117 |
+
"dyn12seq": D12, "dyn12p": D12,
|
| 118 |
+
# THE CONTROL THAT COULD KILL SECTION 3. Every architecture tested lands its
|
| 119 |
+
# gate at 0.087 +/- 0.006 regardless of what state is behind it -- which is not
|
| 120 |
+
# what a gate fitting the MECHANISM would do. If the benefit comes from any
|
| 121 |
+
# smooth token-varying auxiliary signal rather than from the 54D state carrying
|
| 122 |
+
# experience, these two must match the real thing:
|
| 123 |
+
# rand54 frozen random projection of h: content-bearing, never learned
|
| 124 |
+
# noise54 fixed random state per position: carries no content at all
|
| 125 |
+
"rand54": 54, "noise54": 54}
|
| 126 |
+
|
| 127 |
+
# rungs that walk the corpus in order instead of sampling at random. Persistence is
|
| 128 |
+
# meaningless under random sampling: consecutive batches would be unrelated text, so a
|
| 129 |
+
# carried state would be noise. Both arms use the same walk.
|
| 130 |
+
SEQ_RUNGS = {"dyn12seq", "dyn12p"}
|
| 131 |
+
PERSIST_RUNGS = {"dyn12p"}
|
| 132 |
+
|
| 133 |
+
|
| 134 |
+
class RMSNorm(nn.Module):
|
| 135 |
+
def __init__(s, d, eps=1e-6):
|
| 136 |
+
super().__init__()
|
| 137 |
+
s.eps, s.g = eps, nn.Parameter(torch.ones(d))
|
| 138 |
+
|
| 139 |
+
def forward(s, x):
|
| 140 |
+
return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + s.eps) * s.g
|
| 141 |
+
|
| 142 |
+
|
| 143 |
+
def rope_cache(hd, T, device):
|
| 144 |
+
half = hd // 2
|
| 145 |
+
theta = 1.0 / (10000.0 ** (2.0 * torch.arange(half, device=device, dtype=torch.float32) / hd))
|
| 146 |
+
ang = torch.arange(T, device=device, dtype=torch.float32)[:, None] * theta[None, :]
|
| 147 |
+
return ang.cos(), ang.sin()
|
| 148 |
+
|
| 149 |
+
|
| 150 |
+
def apply_rope(x, cos, sin):
|
| 151 |
+
hd = x.shape[-1]
|
| 152 |
+
half = hd // 2
|
| 153 |
+
rot, tail = x[..., : 2 * half], x[..., 2 * half:]
|
| 154 |
+
a, b = rot[..., 0::2], rot[..., 1::2]
|
| 155 |
+
c, s = cos[None, None, : x.shape[-2], :], sin[None, None, : x.shape[-2], :]
|
| 156 |
+
out = torch.stack([a * c - b * s, a * s + b * c], -1).flatten(-2)
|
| 157 |
+
return torch.cat([out, tail], -1) if tail.numel() else out
|
| 158 |
+
|
| 159 |
+
|
| 160 |
+
class PhiHarmonicFFN(nn.Module):
|
| 161 |
+
"""The reference repo's FFN: W2(GELU(W1(x))) at width floor(d*phi). Not gated."""
|
| 162 |
+
|
| 163 |
+
def __init__(s, d):
|
| 164 |
+
super().__init__()
|
| 165 |
+
h = int(math.floor(d * PHI))
|
| 166 |
+
s.W1, s.W2 = nn.Linear(d, h), nn.Linear(h, d)
|
| 167 |
+
s.drop = nn.Dropout(DROPOUT)
|
| 168 |
+
|
| 169 |
+
def forward(s, x):
|
| 170 |
+
return s.drop(s.W2(s.drop(F.gelu(s.W1(x)))))
|
| 171 |
+
|
| 172 |
+
|
| 173 |
+
class PhiGatedFFN(nn.Module):
|
| 174 |
+
"""SwiGLU reading of the paper's undefined "PhiGatedFFN", same width."""
|
| 175 |
+
|
| 176 |
+
def __init__(s, d):
|
| 177 |
+
super().__init__()
|
| 178 |
+
h = int(math.floor(d * PHI))
|
| 179 |
+
s.gate, s.up = nn.Linear(d, h, bias=False), nn.Linear(d, h, bias=False)
|
| 180 |
+
s.down = nn.Linear(h, d, bias=False)
|
| 181 |
+
s.drop = nn.Dropout(DROPOUT)
|
| 182 |
+
|
| 183 |
+
def forward(s, x):
|
| 184 |
+
return s.drop(s.down(F.silu(s.gate(x)) * s.up(x)))
|
| 185 |
+
|
| 186 |
+
|
| 187 |
+
class Dyn12(nn.Module):
|
| 188 |
+
"""Section 4.2. dx12/dt = k*Omega - gamma*x12, per-dimension k and gamma so the twelve
|
| 189 |
+
dimensions separate by time constant rather than evolving as twelve copies."""
|
| 190 |
+
|
| 191 |
+
def __init__(s):
|
| 192 |
+
super().__init__()
|
| 193 |
+
s.k = nn.Parameter(torch.linspace(0.05, 0.20, D12))
|
| 194 |
+
s.gamma = nn.Parameter(torch.linspace(0.02, 0.10, D12))
|
| 195 |
+
s.dt = 0.1
|
| 196 |
+
|
| 197 |
+
def forward(s, x12, h, omega):
|
| 198 |
+
d = s.k[None, None, :] * omega[..., None] - s.gamma[None, None, :] * x12
|
| 199 |
+
return torch.tanh(x12 + s.dt * d)
|
| 200 |
+
|
| 201 |
+
|
| 202 |
+
class Dyn42(nn.Module):
|
| 203 |
+
"""Section 4.3 / THEORY_42D_HYPER_COSMIC. dx42/dt = drive(h) - gamma*x42 + C(x42)."""
|
| 204 |
+
|
| 205 |
+
def __init__(s, d_model):
|
| 206 |
+
super().__init__()
|
| 207 |
+
s.drive = nn.Linear(d_model, D42)
|
| 208 |
+
s.coupling = nn.Linear(D42, D42, bias=False)
|
| 209 |
+
s.gamma = nn.Parameter(torch.full((D42,), 0.05))
|
| 210 |
+
s.dt = 0.1
|
| 211 |
+
|
| 212 |
+
def forward(s, x42, h, omega):
|
| 213 |
+
d = torch.tanh(s.drive(h)) - s.gamma[None, None, :] * x42 + 0.1 * s.coupling(x42)
|
| 214 |
+
return torch.tanh(x42 + s.dt * d)
|
| 215 |
+
|
| 216 |
+
|
| 217 |
+
N_BAND, BAND_D = 7, 6 # 7 * 6 = 42, section 4.1
|
| 218 |
+
|
| 219 |
+
|
| 220 |
+
class Dyn42Banded(nn.Module):
|
| 221 |
+
"""Section 4.3 as WRITTEN: 42 = 7 bands x 6 dims, not a flat 42-vector.
|
| 222 |
+
|
| 223 |
+
THE GAP THIS EXISTS TO CLOSE. Every document in this project describes x42 as a
|
| 224 |
+
"7-band vector manifold" -- the ladder's own docstring says "7 bands x 6 dims" -- and
|
| 225 |
+
the implementation was:
|
| 226 |
+
|
| 227 |
+
drive = nn.Linear(d_model, 42) one flat projection
|
| 228 |
+
coupling = nn.Linear(42, 42) every dimension coupled to every other
|
| 229 |
+
gamma = torch.full((42,), 0.05) forty-two copies of one constant
|
| 230 |
+
|
| 231 |
+
Nothing in it knew that 42 factors. So when dyn42 placed fourth (1.19020) and the
|
| 232 |
+
coupled rungs tri/tri3 placed fifth and sixth, that was read as evidence against the
|
| 233 |
+
42D organ. It is not. It is evidence against a flat, uniformly-damped, densely-coupled
|
| 234 |
+
42-vector, which is not what the theory specifies.
|
| 235 |
+
|
| 236 |
+
Notably Dyn12 already got this right, and says so in its own docstring: per-dimension
|
| 237 |
+
k and gamma "so the twelve dimensions separate by time constant rather than evolving
|
| 238 |
+
as twelve copies". The 42 never received the same treatment.
|
| 239 |
+
|
| 240 |
+
WHAT IS CORY'S, AND WHAT IS MINE
|
| 241 |
+
|
| 242 |
+
his 42 = 7 x 6. The band count, the width, and the idea that bands are distinct
|
| 243 |
+
organs rather than an undifferentiated vector.
|
| 244 |
+
his the leaky-integrator form dx/dt = drive - gamma*x + coupling(x), unchanged.
|
| 245 |
+
mine gamma varies ACROSS BANDS by linspace, so band 0 is fast and band 6 is slow.
|
| 246 |
+
This is Dyn12's own reasoning applied one level up: bands that share a time
|
| 247 |
+
constant are seven copies of each other, and copies cannot specialise.
|
| 248 |
+
mine coupling is BLOCK-DIAGONAL -- each band mixes within its own 6 dimensions.
|
| 249 |
+
A band that couples to all 42 is not a band, it is a slice of a flat vector,
|
| 250 |
+
which is exactly the thing being replaced.
|
| 251 |
+
mine one small cross-band term (n_band x n_band) so bands can still talk. Without
|
| 252 |
+
it this is seven independent 6D systems and the manifold is severed.
|
| 253 |
+
|
| 254 |
+
Cheaper as well as more structured: dense coupling is 42*42 = 1,764 weights per
|
| 255 |
+
layer; seven 6x6 blocks are 7*36 = 252, plus 49 for the cross-band mixer. That is
|
| 256 |
+
1,463 fewer parameters per layer while gaining structure -- which is the same
|
| 257 |
+
"state is cheaper than weights" direction the scaling result already measured.
|
| 258 |
+
"""
|
| 259 |
+
|
| 260 |
+
def __init__(s, d_model):
|
| 261 |
+
super().__init__()
|
| 262 |
+
s.drive = nn.Linear(d_model, D42)
|
| 263 |
+
# block-diagonal: one 6x6 mixer per band, held as (7, 6, 6)
|
| 264 |
+
s.block = nn.Parameter(torch.randn(N_BAND, BAND_D, BAND_D) * (1.0 / BAND_D ** 0.5))
|
| 265 |
+
# bands talk to each other only through their means, at (7, 7)
|
| 266 |
+
s.cross = nn.Parameter(torch.randn(N_BAND, N_BAND) * (1.0 / N_BAND ** 0.5))
|
| 267 |
+
# ONE TIME CONSTANT PER BAND, spread the way Dyn12 spreads its twelve
|
| 268 |
+
s.gamma_band = nn.Parameter(torch.linspace(0.02, 0.12, N_BAND))
|
| 269 |
+
s.dt = 0.1
|
| 270 |
+
|
| 271 |
+
def forward(s, x42, h, omega):
|
| 272 |
+
B, T, _ = x42.shape
|
| 273 |
+
xb = x42.view(B, T, N_BAND, BAND_D) # (B,T,7,6)
|
| 274 |
+
|
| 275 |
+
# within-band mixing: each band's own 6x6
|
| 276 |
+
within = torch.einsum("btnd,nde->btne", xb, s.block)
|
| 277 |
+
|
| 278 |
+
# between-band: each band feels the others through their means
|
| 279 |
+
means = xb.mean(-1) # (B,T,7)
|
| 280 |
+
between = torch.einsum("btn,nm->btm", means, s.cross) # (B,T,7)
|
| 281 |
+
|
| 282 |
+
gam = s.gamma_band[None, None, :, None] # (1,1,7,1)
|
| 283 |
+
drive = torch.tanh(s.drive(h)).view(B, T, N_BAND, BAND_D)
|
| 284 |
+
|
| 285 |
+
d = drive - gam * xb + 0.1 * within + 0.1 * between[..., None]
|
| 286 |
+
# integrate in BANDED shape, then flatten -- adding a (B,T,7,6) derivative to a
|
| 287 |
+
# (B,T,42) state broadcasts into (B,T,7,42) instead of failing usefully.
|
| 288 |
+
return torch.tanh(xb + s.dt * d).reshape(B, T, D42)
|
| 289 |
+
|
| 290 |
+
|
| 291 |
+
class Dyn42Tensor(nn.Module):
|
| 292 |
+
"""42 = 7 bands x sym(3x3). The 6 is not a vector, it is a 3D symmetric tensor.
|
| 293 |
+
|
| 294 |
+
CORY, 2026-08-03: "3D decouples 7, 3x3 is 6, think dimensions and folding and banding
|
| 295 |
+
with connections."
|
| 296 |
+
|
| 297 |
+
A symmetric 3x3 matrix has exactly SIX independent components -- three on the diagonal
|
| 298 |
+
and three off it. So the 6 in "7 bands x 6 dims" is not an arbitrary width. Each band
|
| 299 |
+
is a tensor in three-dimensional space, stored in Voigt order:
|
| 300 |
+
|
| 301 |
+
[xx, yy, zz, xy, xz, yz] -> | xx xy xz |
|
| 302 |
+
| xy yy yz |
|
| 303 |
+
| xz yz zz |
|
| 304 |
+
|
| 305 |
+
That changes what the coupling is allowed to be. The operation that respects it is a
|
| 306 |
+
CONGRUENCE:
|
| 307 |
+
|
| 308 |
+
M -> A M A^T
|
| 309 |
+
|
| 310 |
+
which is what a change of basis does to a tensor. A is 3x3, so nine weights per band
|
| 311 |
+
instead of Dyn42Banded's thirty-six.
|
| 312 |
+
|
| 313 |
+
A CORRECTION, because the first version of this comment overclaimed. It said a free
|
| 314 |
+
6x6 mixer "destroys symmetry" and the congruence preserves it. That is false and the
|
| 315 |
+
test that appeared to confirm it was vacuous: _to_mat writes each Voigt slot into both
|
| 316 |
+
(i,j) and (j,i), so EVERY 6-vector becomes a symmetric matrix regardless of what
|
| 317 |
+
produced it. Symmetry is guaranteed by the storage, not by the operation.
|
| 318 |
+
|
| 319 |
+
The real distinction is invariance. Measured:
|
| 320 |
+
|
| 321 |
+
congruence by a rotation R: eigenvalues unchanged True
|
| 322 |
+
trace unchanged True
|
| 323 |
+
a free 6x6 mixer: eigenvalues unchanged False
|
| 324 |
+
|
| 325 |
+
A tensor's eigenvalues and trace are physical -- they do not depend on how the axes
|
| 326 |
+
are oriented. Congruence is exactly a change of frame, so it must leave them alone,
|
| 327 |
+
and does. A free 6x6 scrambles components that describe different physical directions
|
| 328 |
+
(it can send xx into xy), which no coordinate change can do. That is the argument for
|
| 329 |
+
the congruence, and it is about geometry rather than about symmetry.
|
| 330 |
+
|
| 331 |
+
WHAT IS WHOSE
|
| 332 |
+
his 42 = 7 x 6; 6 = sym(3x3); bands are folded 3D objects with connections.
|
| 333 |
+
mine congruence A M A^T as the band-internal operation, because it is the only
|
| 334 |
+
common bilinear map that provably preserves symmetry. A free 6x6 does not.
|
| 335 |
+
mine per-band gamma by linspace, carried over from Dyn42Banded and originally
|
| 336 |
+
from his own Dyn12.
|
| 337 |
+
mine the cross-band term acts on each band's TRACE -- the one scalar invariant of
|
| 338 |
+
a 3D tensor that does not depend on how the axes are oriented. Bands talk
|
| 339 |
+
through something basis-independent rather than through raw components.
|
| 340 |
+
|
| 341 |
+
Cheapest of the three by a wide margin: dense 42x42 is 1,764 weights per layer,
|
| 342 |
+
seven 6x6 blocks are 252, seven 3x3 congruences are 63.
|
| 343 |
+
"""
|
| 344 |
+
|
| 345 |
+
# Voigt: which (row, col) each of the 6 slots maps to
|
| 346 |
+
_IDX = ((0, 0), (1, 1), (2, 2), (0, 1), (0, 2), (1, 2))
|
| 347 |
+
|
| 348 |
+
def __init__(s, d_model):
|
| 349 |
+
super().__init__()
|
| 350 |
+
s.drive = nn.Linear(d_model, D42)
|
| 351 |
+
s.A = nn.Parameter(torch.eye(3).repeat(N_BAND, 1, 1)
|
| 352 |
+
+ torch.randn(N_BAND, 3, 3) * 0.05) # near-identity start
|
| 353 |
+
s.cross = nn.Parameter(torch.randn(N_BAND, N_BAND) * (1.0 / N_BAND ** 0.5))
|
| 354 |
+
s.gamma_band = nn.Parameter(torch.linspace(0.02, 0.12, N_BAND))
|
| 355 |
+
s.dt = 0.1
|
| 356 |
+
|
| 357 |
+
@classmethod
|
| 358 |
+
def _to_mat(cls, v):
|
| 359 |
+
"""(..., 7, 6) Voigt -> (..., 7, 3, 3) symmetric."""
|
| 360 |
+
sh = v.shape[:-1]
|
| 361 |
+
M = v.new_zeros(*sh, 3, 3)
|
| 362 |
+
for k, (i, j) in enumerate(cls._IDX):
|
| 363 |
+
M[..., i, j] = v[..., k]
|
| 364 |
+
if i != j:
|
| 365 |
+
M[..., j, i] = v[..., k]
|
| 366 |
+
return M
|
| 367 |
+
|
| 368 |
+
@classmethod
|
| 369 |
+
def _to_voigt(cls, M):
|
| 370 |
+
"""(..., 7, 3, 3) -> (..., 7, 6)."""
|
| 371 |
+
return torch.stack([M[..., i, j] for (i, j) in cls._IDX], dim=-1)
|
| 372 |
+
|
| 373 |
+
def forward(s, x42, h, omega):
|
| 374 |
+
B, T, _ = x42.shape
|
| 375 |
+
xb = x42.view(B, T, N_BAND, BAND_D)
|
| 376 |
+
M = s._to_mat(xb) # (B,T,7,3,3)
|
| 377 |
+
|
| 378 |
+
# congruence A M A^T -- symmetric for any A, so the band stays a 3D tensor
|
| 379 |
+
A = s.A[None, None] # (1,1,7,3,3)
|
| 380 |
+
within = s._to_voigt(A @ M @ A.transpose(-1, -2))
|
| 381 |
+
|
| 382 |
+
# bands connect through the TRACE: the invariant that survives rotating the axes
|
| 383 |
+
tr = M.diagonal(dim1=-2, dim2=-1).sum(-1) # (B,T,7)
|
| 384 |
+
between = torch.einsum("btn,nm->btm", tr, s.cross)
|
| 385 |
+
|
| 386 |
+
gam = s.gamma_band[None, None, :, None]
|
| 387 |
+
drive = torch.tanh(s.drive(h)).view(B, T, N_BAND, BAND_D)
|
| 388 |
+
|
| 389 |
+
d = drive - gam * xb + 0.1 * within + 0.1 * between[..., None]
|
| 390 |
+
return torch.tanh(xb + s.dt * d).reshape(B, T, D42)
|
| 391 |
+
|
| 392 |
+
|
| 393 |
+
class Trinity(nn.Module):
|
| 394 |
+
"""12D and 42D as ONE coupled system instead of two vectors glued together.
|
| 395 |
+
|
| 396 |
+
dyn54 concatenates x12 and x42, but x12 sees only Omega and x42 sees only h -- they
|
| 397 |
+
never exchange anything. THEORY_42D_HYPER_COSMIC already couples within a band (6 dims)
|
| 398 |
+
and between bands (7 attractors, diffusion matrix D); this is the same construction one
|
| 399 |
+
level up, between the two subsystems, which is the level nobody wrote:
|
| 400 |
+
|
| 401 |
+
dx12/dt = k*Omega - g12*x12 + M(42->12) x42
|
| 402 |
+
dx42/dt = tanh(drive(h))*(1 + lam*Omega) - g42*x42 + C(x42) + M(12->42) x12
|
| 403 |
+
|
| 404 |
+
Omega now drives BOTH, per THEORY_42D's "Omega_vec is the projected hidden state acting
|
| 405 |
+
as a driving force", and the cross terms close the loop between them. With H computed
|
| 406 |
+
from the joint x54 and Omega coming back out of attention, the three subsystems form a
|
| 407 |
+
single cycle rather than a tree.
|
| 408 |
+
"""
|
| 409 |
+
|
| 410 |
+
def __init__(s, d_model):
|
| 411 |
+
super().__init__()
|
| 412 |
+
s.k = nn.Parameter(torch.linspace(0.05, 0.20, D12))
|
| 413 |
+
s.g12 = nn.Parameter(torch.linspace(0.02, 0.10, D12))
|
| 414 |
+
s.g42 = nn.Parameter(torch.full((D42,), 0.05))
|
| 415 |
+
s.drive = nn.Linear(d_model, D42)
|
| 416 |
+
s.C = nn.Linear(D42, D42, bias=False)
|
| 417 |
+
s.m42_12 = nn.Linear(D42, D12, bias=False)
|
| 418 |
+
s.m12_42 = nn.Linear(D12, D42, bias=False)
|
| 419 |
+
s.lam = nn.Parameter(torch.tensor(0.5))
|
| 420 |
+
s.dt = 0.1
|
| 421 |
+
|
| 422 |
+
def forward(s, state, h, omega):
|
| 423 |
+
x12, x42 = state[..., :D12], state[..., D12:]
|
| 424 |
+
om = omega[..., None]
|
| 425 |
+
# ASYMMETRIC COUPLING, on the neuromodulatory reading. The twelve slow scalars are
|
| 426 |
+
# tone: in a brain those act on cortical GAIN, multiplicatively, not as another
|
| 427 |
+
# additive input. The forty-two structured dimensions feed back the other way as
|
| 428 |
+
# ordinary drive. Symmetric matrices in both directions would make this two peers
|
| 429 |
+
# exchanging vectors; the asymmetry is what makes it a modulatory loop.
|
| 430 |
+
gain = 1.0 + torch.tanh(s.m12_42(x12))
|
| 431 |
+
d12 = s.k * om - s.g12 * x12 + 0.1 * s.m42_12(x42)
|
| 432 |
+
d42 = (torch.tanh(s.drive(h)) * gain * (1.0 + s.lam * om)
|
| 433 |
+
- s.g42 * x42 + 0.1 * s.C(x42))
|
| 434 |
+
return torch.tanh(torch.cat([x12 + s.dt * d12, x42 + s.dt * d42], -1))
|
| 435 |
+
|
| 436 |
+
|
| 437 |
+
class Attention(nn.Module):
|
| 438 |
+
def __init__(s, d, nh, rung):
|
| 439 |
+
super().__init__()
|
| 440 |
+
s.nh, s.hd, s.rung = nh, d // nh, rung
|
| 441 |
+
s.qkv, s.proj = nn.Linear(d, 3 * d), nn.Linear(d, d)
|
| 442 |
+
s.drop = nn.Dropout(DROPOUT)
|
| 443 |
+
s.log_sigma = nn.Parameter(torch.zeros(()))
|
| 444 |
+
s.gate = nn.Parameter(torch.full((), -4.0), requires_grad=(rung != "none"))
|
| 445 |
+
s.w54 = nn.Linear(d, 54, bias=False) if rung in ("static54", "tri3", "rand54") else None
|
| 446 |
+
if rung == "rand54":
|
| 447 |
+
s.w54.weight.requires_grad_(False) # frozen: a projection, never a lesson
|
| 448 |
+
s.last_gate, s.last_diag, s.last_H_std = 0.0, 0.0, 0.0
|
| 449 |
+
|
| 450 |
+
def hebbian(s, state, mask):
|
| 451 |
+
# section 3.2's own identity: ||a-b||^2 = ||a||^2 + ||b||^2 - 2<a,b>. Also what
|
| 452 |
+
# hyper_cosmic_model.py already does, and it avoids cdist's undefined gradient on
|
| 453 |
+
# the zero diagonal of cdist(x, x).
|
| 454 |
+
sq = (state * state).sum(-1)
|
| 455 |
+
d2 = (sq.unsqueeze(-1) + sq.unsqueeze(-2)
|
| 456 |
+
- 2.0 * (state @ state.transpose(-1, -2))).clamp_min(0.0)
|
| 457 |
+
sig = torch.exp(s.log_sigma).clamp(1e-3, 1e4)
|
| 458 |
+
H = torch.exp(-d2 / (2 * sig * sig)).masked_fill(mask < 0, 0.0)
|
| 459 |
+
H = H / H.sum(-1, keepdim=True).clamp_min(1e-9)
|
| 460 |
+
s.last_diag = float(torch.diagonal(H[0]).mean().detach())
|
| 461 |
+
s.last_H_std = float(H.detach().std())
|
| 462 |
+
return H
|
| 463 |
+
|
| 464 |
+
def forward(s, x, state, mask, cos, sin):
|
| 465 |
+
B, T, C = x.shape
|
| 466 |
+
q, k, v = s.qkv(x).split(C, 2)
|
| 467 |
+
sh = lambda t: t.view(B, T, s.nh, s.hd).transpose(1, 2)
|
| 468 |
+
q, k, v = sh(q), sh(k), sh(v)
|
| 469 |
+
q, k = apply_rope(q, cos, sin), apply_rope(k, cos, sin)
|
| 470 |
+
a = F.softmax((q @ k.transpose(-2, -1)) / math.sqrt(s.hd) + mask, -1)
|
| 471 |
+
if s.rung != "none":
|
| 472 |
+
# tri3: all three organs feed one kernel -- the fast structural projection of
|
| 473 |
+
# the current token, the slow 12D tone, and the 42D mid-band structure. The
|
| 474 |
+
# ladder only ever ran these as rivals for a single slot; nothing tested
|
| 475 |
+
# whether their strengths compose.
|
| 476 |
+
if s.rung == "tri3":
|
| 477 |
+
st = torch.cat([s.w54(x), state], -1)
|
| 478 |
+
else:
|
| 479 |
+
st = s.w54(x) if s.w54 is not None else state
|
| 480 |
+
g = torch.sigmoid(s.gate)
|
| 481 |
+
a = (1 - g) * a + g * s.hebbian(st, mask).unsqueeze(1)
|
| 482 |
+
s.last_gate = float(g.detach())
|
| 483 |
+
y = (s.drop(a) @ v).transpose(1, 2).contiguous().view(B, T, C)
|
| 484 |
+
# SECTION 3.4: Omega is attention RECEIVED, i.e. summed over QUERIES (axis -2).
|
| 485 |
+
# ver.4.20 sums over keys (-1), the axis softmax normalises, making Omega == 1.
|
| 486 |
+
#
|
| 487 |
+
# MEASURED 2026-08-01: that query-sum reads the future. Under a causal mask
|
| 488 |
+
# Omega_j = sum over i >= j of a_ij, so token j's Omega counts attention from
|
| 489 |
+
# tokens AFTER j; it then updates state_j, which feeds the next layer's kernel,
|
| 490 |
+
# which shapes the logits at j. tools/causality_probe.py: changing ONLY the last
|
| 491 |
+
# token moves earlier-position logits by 1.7e-04 at PHOS's trained gate (0.562)
|
| 492 |
+
# while `none` and `static54` move by EXACTLY 0.0 on the same harness. Any
|
| 493 |
+
# held-out loss won this way is not comparable to a rung that cannot cheat.
|
| 494 |
+
#
|
| 495 |
+
# The causal replacement is a per-QUERY scalar, which by construction depends
|
| 496 |
+
# only on tokens <= i: the NORMALISED ENTROPY of row i's attention. Summing over
|
| 497 |
+
# keys directly is not an option -- softmax already made that identically 1.0,
|
| 498 |
+
# which is precisely ver.4.20's dead Omega. Entropy is not constant, it is a
|
| 499 |
+
# genuine summary of how row i spread its attention, and it cannot see forward.
|
| 500 |
+
# The log(i+1) normaliser removes the positional drift that comes from later
|
| 501 |
+
# rows simply having more keys available to spread over, so this measures
|
| 502 |
+
# dispersion rather than position.
|
| 503 |
+
if CAUSAL_OMEGA:
|
| 504 |
+
am = a.mean(1).clamp_min(1e-9) # [B, T, T]
|
| 505 |
+
ent = -(am * am.log()).sum(-1) # [B, T], per QUERY
|
| 506 |
+
avail = torch.arange(1, T + 1, device=a.device, dtype=ent.dtype)
|
| 507 |
+
omega = ent / avail.log().clamp_min(1e-6) # in [0, 1]
|
| 508 |
+
else:
|
| 509 |
+
omega = a.mean(1).sum(-2)
|
| 510 |
+
return s.proj(y), omega
|
| 511 |
+
|
| 512 |
+
|
| 513 |
+
class Layer(nn.Module):
|
| 514 |
+
def __init__(s, d, nh, rung, ffn):
|
| 515 |
+
super().__init__()
|
| 516 |
+
s.n1, s.n2 = RMSNorm(d), RMSNorm(d)
|
| 517 |
+
s.attn = Attention(d, nh, rung)
|
| 518 |
+
s.ffn = (PhiHarmonicFFN if ffn == "harmonic" else PhiGatedFFN)(d)
|
| 519 |
+
s.rung = rung
|
| 520 |
+
s.d12 = Dyn12() if rung in ("dyn12", "dyn54", "dyn12seq", "dyn12p") else None
|
| 521 |
+
# dyn42b is dyn42's DIRECT control: same state width, same leaky-integrator form,
|
| 522 |
+
# same everything except that 42 is treated as 7 bands of 6 rather than as a flat
|
| 523 |
+
# vector. If it wins, the earlier dyn42 null was measuring the implementation.
|
| 524 |
+
s.d42 = (Dyn42Tensor(d) if rung == "dyn42t"
|
| 525 |
+
else Dyn42Banded(d) if rung == "dyn42b"
|
| 526 |
+
else Dyn42(d) if rung in ("dyn42", "dyn54") else None)
|
| 527 |
+
s.tri = Trinity(d) if rung in ("tri", "tri3") else None
|
| 528 |
+
|
| 529 |
+
def forward(s, x, state, mask, cos, sin):
|
| 530 |
+
h = s.n1(x)
|
| 531 |
+
att, omega = s.attn(h, state, mask, cos, sin)
|
| 532 |
+
x = x + att
|
| 533 |
+
if s.tri is not None: # section 2.1 step 2, coupled
|
| 534 |
+
state = s.tri(state, h, omega)
|
| 535 |
+
elif s.d12 is not None or s.d42 is not None: # section 2.1 step 2, independent
|
| 536 |
+
parts = []
|
| 537 |
+
if s.d12 is not None:
|
| 538 |
+
parts.append(s.d12(state[..., :D12], h, omega))
|
| 539 |
+
if s.d42 is not None:
|
| 540 |
+
parts.append(s.d42(state[..., -D42:], h, omega))
|
| 541 |
+
state = torch.cat(parts, -1)
|
| 542 |
+
x = x + s.ffn(s.n2(x))
|
| 543 |
+
return x, state, omega
|
| 544 |
+
|
| 545 |
+
|
| 546 |
+
class Ladder(nn.Module):
|
| 547 |
+
def __init__(s, vocab, rung, ffn, d=N_EMBD, nh=N_HEAD):
|
| 548 |
+
super().__init__()
|
| 549 |
+
s.d, s.nh, s.rung = d, nh, rung
|
| 550 |
+
s.sd = STATE_DIM[rung]
|
| 551 |
+
s.tok = nn.Embedding(vocab, d)
|
| 552 |
+
s.blocks = nn.ModuleList([Layer(d, nh, rung, ffn) for _ in range(N_LAYER)])
|
| 553 |
+
s.nf = RMSNorm(d)
|
| 554 |
+
s.head = nn.Linear(d, vocab, bias=False)
|
| 555 |
+
# a state that starts at zero is identical for every token, which makes H the
|
| 556 |
+
# all-ones matrix and the first layer's mechanism inert. Seeding it from the
|
| 557 |
+
# embedding costs one small projection and keeps every layer live.
|
| 558 |
+
s.state_init = nn.Linear(d, s.sd) if rung.startswith(("dyn", "tri")) and s.sd else None
|
| 559 |
+
if rung == "noise54":
|
| 560 |
+
# a state that varies by POSITION but is identical for every input. If this
|
| 561 |
+
# matches the real 54D state, the kernel is a positional regulariser and the
|
| 562 |
+
# "experience-modulated attention" reading is not supported by the data.
|
| 563 |
+
g0 = torch.Generator().manual_seed(20260730)
|
| 564 |
+
s.register_buffer("noise_state",
|
| 565 |
+
torch.randn(BLOCK, 54, generator=g0) * 0.5)
|
| 566 |
+
s.register_buffer("mask", torch.triu(torch.full((BLOCK, BLOCK), float("-inf")), 1))
|
| 567 |
+
|
| 568 |
+
def forward(s, idx, targets=None, state_in=None):
|
| 569 |
+
B, T = idx.shape
|
| 570 |
+
x = s.tok(idx)
|
| 571 |
+
cos, sin = rope_cache(s.d // s.nh, T, idx.device)
|
| 572 |
+
m = s.mask[:T, :T]
|
| 573 |
+
state = torch.tanh(s.state_init(x)) if s.state_init is not None else None
|
| 574 |
+
if s.rung == "noise54":
|
| 575 |
+
state = s.noise_state[:T].unsqueeze(0).expand(B, -1, -1)
|
| 576 |
+
# SECTION 4.4. A state arriving from the previous window seeds this one. It is the
|
| 577 |
+
# LAST position's state broadcast across the new window: what she carried out of
|
| 578 |
+
# the previous 128 characters is what she starts the next 128 with. gamma is what
|
| 579 |
+
# keeps this from saturating -- a state that never forgets pins to tanh's rails and
|
| 580 |
+
# stops carrying anything, so forgetting is what makes the persistence usable.
|
| 581 |
+
if state is not None and state_in is not None:
|
| 582 |
+
state = state + state_in[:, -1:, :].detach().expand(-1, T, -1)
|
| 583 |
+
state = torch.tanh(state)
|
| 584 |
+
omegas = []
|
| 585 |
+
for b in s.blocks:
|
| 586 |
+
x, state, om = b(x, state, m, cos, sin)
|
| 587 |
+
omegas.append(om)
|
| 588 |
+
lg = s.head(s.nf(x))
|
| 589 |
+
loss = None if targets is None else F.cross_entropy(
|
| 590 |
+
lg.view(-1, lg.size(-1)), targets.reshape(-1))
|
| 591 |
+
s.last_state, s.last_omega = state, omegas[-1]
|
| 592 |
+
s.out_state = state
|
| 593 |
+
return lg, loss
|
| 594 |
+
|
| 595 |
+
def gates(s):
|
| 596 |
+
return [b.attn.last_gate for b in s.blocks]
|
| 597 |
+
|
| 598 |
+
def quantum_birth(s, qi):
|
| 599 |
+
std = (1.0 / (s.d * PHI_INV)) ** 0.5 # section 6.2
|
| 600 |
+
for mod in s.modules():
|
| 601 |
+
if isinstance(mod, (nn.Linear, nn.Embedding)):
|
| 602 |
+
qi.fill_(mod.weight, std)
|
| 603 |
+
if isinstance(mod, nn.Linear) and mod.bias is not None:
|
| 604 |
+
with torch.no_grad():
|
| 605 |
+
mod.bias.zero_()
|
| 606 |
+
|
| 607 |
+
|
| 608 |
+
@torch.no_grad()
|
| 609 |
+
def calibrate_sigma(m, xb):
|
| 610 |
+
"""2*sigma^2 = median pairwise d2 of whatever state that layer actually sees."""
|
| 611 |
+
if m.rung == "none":
|
| 612 |
+
return
|
| 613 |
+
B, T = xb.shape
|
| 614 |
+
x = m.tok(xb)
|
| 615 |
+
cos, sin = rope_cache(m.d // m.nh, T, xb.device)
|
| 616 |
+
mask = m.mask[:T, :T]
|
| 617 |
+
state = torch.tanh(m.state_init(x)) if m.state_init is not None else None
|
| 618 |
+
if m.rung == "noise54":
|
| 619 |
+
# must mirror Ladder.forward exactly, or the control is calibrated against a state
|
| 620 |
+
# it never actually sees -- which would hand it a rigged sigma and invalidate it
|
| 621 |
+
state = m.noise_state[:xb.shape[1]].unsqueeze(0).expand(xb.shape[0], -1, -1)
|
| 622 |
+
for b in m.blocks:
|
| 623 |
+
h = b.n1(x)
|
| 624 |
+
if m.rung == "tri3":
|
| 625 |
+
st = torch.cat([b.attn.w54(h), state], -1)
|
| 626 |
+
elif b.attn.w54 is not None:
|
| 627 |
+
st = b.attn.w54(h)
|
| 628 |
+
else:
|
| 629 |
+
st = state
|
| 630 |
+
sq = (st * st).sum(-1)
|
| 631 |
+
d2 = (sq.unsqueeze(-1) + sq.unsqueeze(-2)
|
| 632 |
+
- 2.0 * (st @ st.transpose(-1, -2))).clamp_min(0.0)
|
| 633 |
+
iu = torch.triu(torch.ones(d2.shape[-2:], dtype=torch.bool), 1)
|
| 634 |
+
med = float(d2[0][iu].median())
|
| 635 |
+
if med > 1e-12:
|
| 636 |
+
b.attn.log_sigma.fill_(0.5 * math.log(med / 2.0))
|
| 637 |
+
x, state, _ = b(x, state, mask, cos, sin)
|
| 638 |
+
|
| 639 |
+
|
| 640 |
+
def preflight(m, xb, yb):
|
| 641 |
+
"""Refuse to report a number until the mechanism is provably live.
|
| 642 |
+
|
| 643 |
+
Each assertion corresponds to a real failure that shipped and produced a confident
|
| 644 |
+
negative. A rung that trips any of these is not evidence about the idea.
|
| 645 |
+
"""
|
| 646 |
+
checks, m2 = [], m
|
| 647 |
+
m2.train()
|
| 648 |
+
_, loss = m2(xb, yb)
|
| 649 |
+
loss.backward()
|
| 650 |
+
|
| 651 |
+
om = m2.last_omega
|
| 652 |
+
checks.append(("omega varies across tokens (ver.4.20 had std 5e-08)",
|
| 653 |
+
float(om.std()), 1e-4, float(om.std()) > 1e-4))
|
| 654 |
+
if m.rung != "none":
|
| 655 |
+
if m.rung.startswith(("dyn", "tri")):
|
| 656 |
+
st = m2.last_state
|
| 657 |
+
checks.append(("state varies across tokens (ver.4.20 had std 3e-08)",
|
| 658 |
+
float(st.std()), 1e-4, float(st.std()) > 1e-4))
|
| 659 |
+
diag = statistics.fmean([b.attn.last_diag for b in m2.blocks])
|
| 660 |
+
checks.append((f"H not the identity (needs << 1.0, uniform = {1/xb.shape[1]:.4f})",
|
| 661 |
+
diag, 0.5, diag < 0.5))
|
| 662 |
+
hstd = statistics.fmean([b.attn.last_H_std for b in m2.blocks])
|
| 663 |
+
checks.append(("H not all-ones (needs non-zero spread)", hstd, 1e-6, hstd > 1e-6))
|
| 664 |
+
gg = statistics.fmean([abs(float(b.attn.gate.grad)) for b in m2.blocks
|
| 665 |
+
if b.attn.gate.grad is not None])
|
| 666 |
+
checks.append(("gate gradient not suppressed", gg, 1e-6, gg > 1e-6))
|
| 667 |
+
m2.zero_grad(set_to_none=True)
|
| 668 |
+
return checks
|
| 669 |
+
|
| 670 |
+
|
| 671 |
+
def train_rung(rung, ffn, seed, data, vocab, steps, pool, val_w, words, itos):
|
| 672 |
+
torch.manual_seed(seed)
|
| 673 |
+
random.seed(seed)
|
| 674 |
+
gen = torch.Generator().manual_seed(seed)
|
| 675 |
+
m = Ladder(vocab, rung, ffn)
|
| 676 |
+
m.quantum_birth(QuantumInit(pool, seed))
|
| 677 |
+
m = m.to(DEV)
|
| 678 |
+
cal = torch.stack([data[i:i + BLOCK] for i in torch.randint(
|
| 679 |
+
len(data) - BLOCK - 1, (8,), generator=torch.Generator().manual_seed(seed))]).to(DEV)
|
| 680 |
+
calibrate_sigma(m, cal)
|
| 681 |
+
|
| 682 |
+
pf = preflight(m, cal[:4], torch.stack([data[i + 1:i + 1 + BLOCK] for i in
|
| 683 |
+
torch.randint(len(data) - BLOCK - 1, (4,),
|
| 684 |
+
generator=torch.Generator().manual_seed(seed))]).to(DEV))
|
| 685 |
+
|
| 686 |
+
def _coup(mm):
|
| 687 |
+
# POST-FLIGHT. Preflight proves a mechanism starts alive; this proves it did not
|
| 688 |
+
# quietly die. The cross-couplings begin damped by 0.1 (the reference repo's
|
| 689 |
+
# "weak coupling initially"), so the question is whether training grows them or
|
| 690 |
+
# leaves the trinity decorative.
|
| 691 |
+
v = [(float(b.tri.m12_42.weight.abs().mean()), float(b.tri.m42_12.weight.abs().mean()))
|
| 692 |
+
for b in mm.blocks if getattr(b, "tri", None) is not None]
|
| 693 |
+
return v
|
| 694 |
+
|
| 695 |
+
coup0 = _coup(m)
|
| 696 |
+
|
| 697 |
+
keys = ("attn.gate", "attn.w54", "attn.log_sigma", "d12.", "d42.", "tri.", "state_init")
|
| 698 |
+
cst = [p for n, p in m.named_parameters() if any(k in n for k in keys) and p.requires_grad]
|
| 699 |
+
bulk = [p for n, p in m.named_parameters() if not any(k in n for k in keys) and p.requires_grad]
|
| 700 |
+
base = 3e-4
|
| 701 |
+
opt = torch.optim.AdamW([{"params": bulk, "lr": base},
|
| 702 |
+
{"params": cst, "lr": base * 10, "weight_decay": 0.0}],
|
| 703 |
+
lr=base, weight_decay=0.01)
|
| 704 |
+
n = int(0.9 * len(data))
|
| 705 |
+
tr = data[:n]
|
| 706 |
+
m.train()
|
| 707 |
+
best = float("inf")
|
| 708 |
+
|
| 709 |
+
# SEQUENTIAL WALK for the persistence arms. 16 lanes each march through their own
|
| 710 |
+
# contiguous stretch of the corpus, so consecutive batches are consecutive TEXT and a
|
| 711 |
+
# carried state means something. Under the random sampler used by every other rung,
|
| 712 |
+
# batch k+1 is unrelated to batch k, so a carried state would be pure noise -- which is
|
| 713 |
+
# why the control arm (dyn12seq) walks identically and only differs in whether the
|
| 714 |
+
# state survives the window boundary.
|
| 715 |
+
B = 16
|
| 716 |
+
seq = rung in SEQ_RUNGS
|
| 717 |
+
persist = rung in PERSIST_RUNGS
|
| 718 |
+
span = max(BLOCK * 2, (len(tr) - BLOCK - 1) // B)
|
| 719 |
+
pos = [i * span for i in range(B)]
|
| 720 |
+
carry = None
|
| 721 |
+
|
| 722 |
+
for s in range(1, steps + 1):
|
| 723 |
+
warm = max(1, int(0.05 * steps))
|
| 724 |
+
lr = base * (s / warm) if s < warm else base * (PHI / 2) * (
|
| 725 |
+
1 + math.cos(math.pi * (s - warm) / max(1, steps - warm)))
|
| 726 |
+
opt.param_groups[0]["lr"], opt.param_groups[1]["lr"] = lr, lr * 10
|
| 727 |
+
if seq:
|
| 728 |
+
wrapped = []
|
| 729 |
+
for i in range(B):
|
| 730 |
+
if pos[i] + BLOCK + 1 >= min((i + 1) * span, len(tr) - 1):
|
| 731 |
+
pos[i] = i * span # lane restarted: its history is broken
|
| 732 |
+
wrapped.append(i)
|
| 733 |
+
ix = torch.tensor(pos, dtype=torch.long)
|
| 734 |
+
for i in range(B):
|
| 735 |
+
pos[i] += BLOCK
|
| 736 |
+
if carry is not None and wrapped:
|
| 737 |
+
# a wrapped lane must NOT inherit state from unrelated text
|
| 738 |
+
carry = carry.clone()
|
| 739 |
+
carry[wrapped] = 0.0
|
| 740 |
+
else:
|
| 741 |
+
ix = torch.randint(len(tr) - BLOCK - 1, (B,), generator=gen)
|
| 742 |
+
# .long() at BATCH time, not corpus time. A 269M-character corpus held as int64
|
| 743 |
+
# is 2.15 GB resident; as int16 it is 0.54 GB, and only the (B, BLOCK) slice ever
|
| 744 |
+
# needs to be int64 for nn.Embedding. The d576 scaling run died at exactly this
|
| 745 |
+
# point -- outer process gone, no error recorded -- with the corpus tensor and a
|
| 746 |
+
# 24.6M-parameter model competing for the last few GB.
|
| 747 |
+
x = torch.stack([tr[i:i + BLOCK] for i in ix]).to(DEV).long()
|
| 748 |
+
y = torch.stack([tr[i + 1:i + 1 + BLOCK] for i in ix]).to(DEV).long()
|
| 749 |
+
_, loss = m(x, y, state_in=carry if persist else None)
|
| 750 |
+
if persist:
|
| 751 |
+
carry = m.out_state.detach() if m.out_state is not None else None
|
| 752 |
+
opt.zero_grad(set_to_none=True)
|
| 753 |
+
loss.backward()
|
| 754 |
+
torch.nn.utils.clip_grad_norm_(m.parameters(), 1.0)
|
| 755 |
+
opt.step()
|
| 756 |
+
if s % 100 == 0 or s == steps:
|
| 757 |
+
m.eval()
|
| 758 |
+
with torch.no_grad():
|
| 759 |
+
tot = c = 0
|
| 760 |
+
for i in range(0, len(val_w), 16):
|
| 761 |
+
xb = val_w[i:i + 16].to(DEV)
|
| 762 |
+
_, l = m(xb[:, :-1], xb[:, 1:])
|
| 763 |
+
tot += l.item() * xb.size(0)
|
| 764 |
+
c += xb.size(0)
|
| 765 |
+
m.train()
|
| 766 |
+
best = min(best, tot / max(1, c))
|
| 767 |
+
m.eval()
|
| 768 |
+
idx = torch.tensor([[int(data[0])]], device=DEV)
|
| 769 |
+
out = []
|
| 770 |
+
with torch.no_grad():
|
| 771 |
+
for _ in range(600):
|
| 772 |
+
lg, _ = m(idx[:, -BLOCK:])
|
| 773 |
+
nx = int(torch.multinomial(F.softmax(lg[0, -1] / 0.8, -1), 1))
|
| 774 |
+
out.append(nx)
|
| 775 |
+
idx = torch.cat([idx, torch.tensor([[nx]], device=DEV)], 1)
|
| 776 |
+
txt = "".join(itos.get(i, "") for i in out)
|
| 777 |
+
coupT = _coup(m)
|
| 778 |
+
coup = None
|
| 779 |
+
if coup0:
|
| 780 |
+
a0 = statistics.fmean([x for p in coup0 for x in p])
|
| 781 |
+
aT = statistics.fmean([x for p in coupT for x in p])
|
| 782 |
+
coup = (a0, aT, aT / max(a0, 1e-12))
|
| 783 |
+
return best, statistics.fmean(m.gates()) if m.rung != "none" else 0.0, \
|
| 784 |
+
real_word_rate(txt, words), pf, sum(p.numel() for p in m.parameters()), txt, coup, m
|
| 785 |
+
|
| 786 |
+
|
| 787 |
+
def main():
|
| 788 |
+
steps = int(sys.argv[1]) if len(sys.argv) > 1 else 1200
|
| 789 |
+
seeds = list(range(int(sys.argv[2]) if len(sys.argv) > 2 else 3))
|
| 790 |
+
rungs = (sys.argv[3] if len(sys.argv) > 3 else
|
| 791 |
+
"none,static54,dyn12,dyn42,dyn54").split(",")
|
| 792 |
+
ffn = sys.argv[4] if len(sys.argv) > 4 else "harmonic"
|
| 793 |
+
|
| 794 |
+
text = CORPUS_PATH.read_text(encoding="utf-8", errors="ignore")
|
| 795 |
+
chars = sorted(set(text))
|
| 796 |
+
stoi = {c: i for i, c in enumerate(chars)}
|
| 797 |
+
itos = {i: c for c, i in stoi.items()}
|
| 798 |
+
data = torch.tensor([stoi[c] for c in text], dtype=torch.long)
|
| 799 |
+
words = set(__import__("re").findall(r"[a-z']+", text.lower()))
|
| 800 |
+
|
| 801 |
+
print("=" * 84)
|
| 802 |
+
print(" COSMOS STATE LADDER -- 12D / 42D / 54D, one rung at a time")
|
| 803 |
+
print("=" * 84)
|
| 804 |
+
print(f"\n device: {DEV}" + (f" ({torch.cuda.get_device_name(0)})" if DEV.type == "cuda" else ""))
|
| 805 |
+
print(f" corpus: {CORPUS_PATH}"
|
| 806 |
+
+ (" [FROZEN SNAPSHOT]" if os.getenv("COSMOS_CORPUS") else " [LIVE - grows during the run, cross-run numbers are not comparable]"))
|
| 807 |
+
print(f" corpus {len(text):,} vocab {len(chars)} {steps} steps x {len(seeds)} seeds")
|
| 808 |
+
print(f" scaffold: RMSNorm + RoPE + {'PhiHarmonicFFN (GELU)' if ffn=='harmonic' else 'PhiGatedFFN (SwiGLU)'}"
|
| 809 |
+
f" @ floor(d*phi)={int(math.floor(N_EMBD*PHI))} + phi-Xavier")
|
| 810 |
+
print(f" rungs: {rungs}\n")
|
| 811 |
+
|
| 812 |
+
pool = quantum_pool()
|
| 813 |
+
print(f" quantum pool: {len(pool):,} draws\n")
|
| 814 |
+
n = int(0.9 * len(data))
|
| 815 |
+
val = data[n:]
|
| 816 |
+
g = torch.Generator().manual_seed(999)
|
| 817 |
+
vi = torch.randint(len(val) - BLOCK - 1, (64,), generator=g)
|
| 818 |
+
val_w = torch.stack([val[i:i + BLOCK + 1] for i in vi])
|
| 819 |
+
|
| 820 |
+
out = {}
|
| 821 |
+
for rung in rungs:
|
| 822 |
+
losses, rws, gates, npar, pf_last = [], [], [], 0, None
|
| 823 |
+
blocked = False
|
| 824 |
+
best_loss, best_model = float("inf"), None
|
| 825 |
+
for sd in seeds:
|
| 826 |
+
t0 = time.time()
|
| 827 |
+
b, gt, r, pf, npar, txt, coup, model = train_rung(
|
| 828 |
+
rung, ffn, sd, data, len(chars), steps, pool, val_w, words, itos)
|
| 829 |
+
if b < best_loss:
|
| 830 |
+
best_loss, best_model = b, model
|
| 831 |
+
pf_last = pf
|
| 832 |
+
bad = [c for c in pf if not c[3]]
|
| 833 |
+
if sd == 0:
|
| 834 |
+
print(f" --- {rung} preflight ---")
|
| 835 |
+
for name, val_, thr, ok in pf:
|
| 836 |
+
print(f" [{'OK ' if ok else 'DEAD'}] {name}: {val_:.3e}")
|
| 837 |
+
if bad:
|
| 838 |
+
blocked = True
|
| 839 |
+
print(f" MECHANISM INERT -- {rung} will not be reported as evidence.\n")
|
| 840 |
+
break
|
| 841 |
+
print()
|
| 842 |
+
losses.append(b)
|
| 843 |
+
rws.append(r)
|
| 844 |
+
gates.append(gt)
|
| 845 |
+
print(f" {rung:<9s} seed {sd} loss {b:.5f} real-word {r:.3f}"
|
| 846 |
+
+ (f" gate {gt:.4f}" if rung != "none" else "")
|
| 847 |
+
+ (f" coupling {coup[0]:.2e}->{coup[1]:.2e} ({coup[2]:.1f}x)" if coup else "")
|
| 848 |
+
+ f" ({time.time()-t0:.0f}s)", flush=True)
|
| 849 |
+
if blocked:
|
| 850 |
+
out[rung] = {"blocked": True,
|
| 851 |
+
"preflight": [[c[0], c[1], c[2], c[3]] for c in pf_last]}
|
| 852 |
+
continue
|
| 853 |
+
out[rung] = {"loss": losses, "real_word": rws, "gates": gates, "params": npar,
|
| 854 |
+
"preflight": [[c[0], c[1], c[2], c[3]] for c in pf_last]}
|
| 855 |
+
print(f" => {rung:<9s} mean loss {statistics.fmean(losses):.5f} "
|
| 856 |
+
f"real-word {statistics.fmean(rws):.3f} params {npar:,}", flush=True)
|
| 857 |
+
|
| 858 |
+
# SAVE. The first version of this file trained fifteen models and discarded every
|
| 859 |
+
# one of them -- OUTDIR was declared and torch.save was never called. A result you
|
| 860 |
+
# cannot deploy is half a result.
|
| 861 |
+
if best_model is not None:
|
| 862 |
+
OUTDIR.mkdir(parents=True, exist_ok=True)
|
| 863 |
+
torch.save({"model": best_model.state_dict(), "stoi": stoi, "itos": itos,
|
| 864 |
+
"config": {"block": BLOCK, "n_layer": N_LAYER, "n_head": N_HEAD,
|
| 865 |
+
"n_embd": N_EMBD, "vocab": len(chars),
|
| 866 |
+
"d54": STATE_DIM.get(rung, 0), "d12": D12, "d42": D42,
|
| 867 |
+
"d_ff": int(math.floor(N_EMBD * PHI))},
|
| 868 |
+
"arch": f"Cosmos-Ladder-{rung}", "rung": rung, "ffn": ffn,
|
| 869 |
+
"gate_param": "logit", "norm": "rmsnorm", "pos": "rope",
|
| 870 |
+
"total_steps": steps, "best_val_loss": best_loss,
|
| 871 |
+
"quantum_source": "ibm_real_shots"},
|
| 872 |
+
OUTDIR / f"ladder_{rung}.pt")
|
| 873 |
+
print(f" saved -> {OUTDIR / f'ladder_{rung}.pt'} (loss {best_loss:.5f})\n",
|
| 874 |
+
flush=True)
|
| 875 |
+
|
| 876 |
+
base = out.get("none", {}).get("loss")
|
| 877 |
+
if base:
|
| 878 |
+
print("=" * 84 + "\n LADDER\n" + "=" * 84)
|
| 879 |
+
print(f" {'rung':<10s} {'mean loss':>10s} {'vs none':>10s} {'t':>8s} {'wins':>6s} {'params':>10s}")
|
| 880 |
+
for rung in rungs:
|
| 881 |
+
r = out.get(rung, {})
|
| 882 |
+
if r.get("blocked"):
|
| 883 |
+
print(f" {rung:<10s} {'BLOCKED - mechanism provably inert':>46s}")
|
| 884 |
+
continue
|
| 885 |
+
if "loss" not in r:
|
| 886 |
+
continue
|
| 887 |
+
d = [p - c for p, c in zip(base, r["loss"])]
|
| 888 |
+
md = statistics.fmean(d)
|
| 889 |
+
se = statistics.stdev(d) / math.sqrt(len(d)) if len(d) > 1 else 0.0
|
| 890 |
+
t = md / se if se > 0 else 0.0
|
| 891 |
+
print(f" {rung:<10s} {statistics.fmean(r['loss']):>10.5f} {-md:>+10.5f} "
|
| 892 |
+
f"{-t:>8.2f} {sum(1 for x in d if x > 0):>4d}/{len(d)} {r['params']:>10,}")
|
| 893 |
+
Path("logs").mkdir(exist_ok=True)
|
| 894 |
+
RESULTS.write_text(json.dumps({"phi": PHI, "steps": steps, "ffn": ffn, "rungs": out},
|
| 895 |
+
indent=2), encoding="utf-8")
|
| 896 |
+
print(f"\n saved -> {RESULTS}")
|
| 897 |
+
return 0
|
| 898 |
+
|
| 899 |
+
|
| 900 |
+
if __name__ == "__main__":
|
| 901 |
+
raise SystemExit(main())
|
|
@@ -0,0 +1,193 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Offline integrity and privacy-boundary verifier for the QC67 Cosmos kit."""
|
| 3 |
+
from __future__ import annotations
|
| 4 |
+
|
| 5 |
+
import argparse
|
| 6 |
+
import hashlib
|
| 7 |
+
import json
|
| 8 |
+
import sys
|
| 9 |
+
from collections import Counter
|
| 10 |
+
from pathlib import Path
|
| 11 |
+
|
| 12 |
+
ROOT = Path(__file__).resolve().parent
|
| 13 |
+
MANIFEST = ROOT / "RELEASE_MANIFEST.json"
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
def sha256(path: Path) -> str:
|
| 17 |
+
digest = hashlib.sha256()
|
| 18 |
+
with path.open("rb") as handle:
|
| 19 |
+
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
|
| 20 |
+
digest.update(chunk)
|
| 21 |
+
return digest.hexdigest()
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def check_manifest(strict: bool) -> list[str]:
|
| 25 |
+
errors: list[str] = []
|
| 26 |
+
release = json.loads(MANIFEST.read_text(encoding="utf-8"))
|
| 27 |
+
expected = set()
|
| 28 |
+
for entry in release.get("files", []):
|
| 29 |
+
rel = entry["path"]
|
| 30 |
+
expected.add(rel)
|
| 31 |
+
path = ROOT / rel
|
| 32 |
+
if not path.is_file():
|
| 33 |
+
errors.append(f"missing: {rel}")
|
| 34 |
+
continue
|
| 35 |
+
size = path.stat().st_size
|
| 36 |
+
if size != int(entry["bytes"]):
|
| 37 |
+
errors.append(f"size mismatch: {rel} ({size} != {entry['bytes']})")
|
| 38 |
+
continue
|
| 39 |
+
actual = sha256(path)
|
| 40 |
+
if actual != entry["sha256"]:
|
| 41 |
+
errors.append(f"hash mismatch: {rel}")
|
| 42 |
+
|
| 43 |
+
if strict:
|
| 44 |
+
ignored = {"RELEASE_MANIFEST.json"}
|
| 45 |
+
# Runtime spoor, not release content. logs/ and workspace/ are written by
|
| 46 |
+
# simply USING the kit, and __pycache__ appears the moment python imports
|
| 47 |
+
# anything -- so without these a clean, correct kit reports dozens of
|
| 48 |
+
# "unmanifested file" errors the first time anyone runs it, and the real
|
| 49 |
+
# findings (a staged credential, a missing weight) drown in the noise.
|
| 50 |
+
# creatures/ holds beings the USER birthed on their own machine. They are not
|
| 51 |
+
# release content and their hashes cannot be known in advance -- manifesting them
|
| 52 |
+
# would make the integrity check fail the moment anyone actually used the kit.
|
| 53 |
+
skip_prefixes = ("downloads/", "logs/", "workspace/", "creatures/")
|
| 54 |
+
actual = {
|
| 55 |
+
path.relative_to(ROOT).as_posix()
|
| 56 |
+
for path in ROOT.rglob("*")
|
| 57 |
+
if path.is_file()
|
| 58 |
+
and path.relative_to(ROOT).as_posix() not in ignored
|
| 59 |
+
and not path.relative_to(ROOT).as_posix().startswith(skip_prefixes)
|
| 60 |
+
and "__pycache__" not in path.parts
|
| 61 |
+
and not path.name.endswith((".pyc", ".pyo"))
|
| 62 |
+
}
|
| 63 |
+
for rel in sorted(actual - expected):
|
| 64 |
+
errors.append(f"unmanifested file: {rel}")
|
| 65 |
+
return errors
|
| 66 |
+
|
| 67 |
+
|
| 68 |
+
def check_blank_credentials() -> list[str]:
|
| 69 |
+
errors: list[str] = []
|
| 70 |
+
config = json.loads(
|
| 71 |
+
(ROOT / "genesis_engine" / "config.json").read_text(encoding="utf-8")
|
| 72 |
+
)
|
| 73 |
+
for key in ("ibm_token", "azure_connection_string"):
|
| 74 |
+
if str(config.get(key) or "").strip():
|
| 75 |
+
errors.append(f"credential field is not blank: genesis_engine/config.json:{key}")
|
| 76 |
+
forbidden_names = ("oauth2_tokens.json", ".env", "credentials.json")
|
| 77 |
+
for path in ROOT.rglob("*"):
|
| 78 |
+
if path.is_file() and path.name.casefold() in forbidden_names:
|
| 79 |
+
errors.append(f"forbidden credential file present: {path.relative_to(ROOT)}")
|
| 80 |
+
return errors
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
def check_public_archive() -> tuple[list[str], dict]:
|
| 84 |
+
errors: list[str] = []
|
| 85 |
+
archive = ROOT / "data" / "quantum_measurements_public.jsonl"
|
| 86 |
+
data_manifest = json.loads(
|
| 87 |
+
(ROOT / "data" / "quantum_measurements_manifest.json").read_text(
|
| 88 |
+
encoding="utf-8"
|
| 89 |
+
)
|
| 90 |
+
)
|
| 91 |
+
records = Counter()
|
| 92 |
+
samples = Counter()
|
| 93 |
+
total = 0
|
| 94 |
+
for line_number, line in enumerate(
|
| 95 |
+
archive.open(encoding="utf-8", errors="strict"), 1
|
| 96 |
+
):
|
| 97 |
+
try:
|
| 98 |
+
row = json.loads(line)
|
| 99 |
+
except Exception as exc:
|
| 100 |
+
errors.append(f"archive line {line_number}: invalid JSON ({exc})")
|
| 101 |
+
continue
|
| 102 |
+
counts = row.get("counts")
|
| 103 |
+
if not isinstance(counts, dict) or not counts:
|
| 104 |
+
errors.append(f"archive line {line_number}: missing counts")
|
| 105 |
+
continue
|
| 106 |
+
observed = sum(int(value) for value in counts.values())
|
| 107 |
+
declared = int(row.get("total_shots", -1))
|
| 108 |
+
if observed != declared:
|
| 109 |
+
errors.append(
|
| 110 |
+
f"archive line {line_number}: shot mismatch {observed} != {declared}"
|
| 111 |
+
)
|
| 112 |
+
category = str(row.get("provider_class") or "missing")
|
| 113 |
+
records[category] += 1
|
| 114 |
+
samples[category] += observed
|
| 115 |
+
total += observed
|
| 116 |
+
|
| 117 |
+
expected = data_manifest["summary"]
|
| 118 |
+
if total != int(expected["total_samples"]):
|
| 119 |
+
errors.append(
|
| 120 |
+
f"archive total mismatch: {total} != {expected['total_samples']}"
|
| 121 |
+
)
|
| 122 |
+
for category, expected_count in expected["records_by_provider_class"].items():
|
| 123 |
+
if records[category] != int(expected_count):
|
| 124 |
+
errors.append(
|
| 125 |
+
f"archive record count mismatch for {category}: "
|
| 126 |
+
f"{records[category]} != {expected_count}"
|
| 127 |
+
)
|
| 128 |
+
return errors, {
|
| 129 |
+
"records": sum(records.values()),
|
| 130 |
+
"samples": total,
|
| 131 |
+
"records_by_class": dict(records),
|
| 132 |
+
"samples_by_class": dict(samples),
|
| 133 |
+
}
|
| 134 |
+
|
| 135 |
+
|
| 136 |
+
def check_model_metadata() -> list[str]:
|
| 137 |
+
errors: list[str] = []
|
| 138 |
+
metadata = json.loads(
|
| 139 |
+
(ROOT / "weights" / "cosmos_born.meta.json").read_text(encoding="utf-8")
|
| 140 |
+
)
|
| 141 |
+
if int(metadata.get("params", 0)) != 1_842_432:
|
| 142 |
+
errors.append("unexpected cosmos_born parameter count")
|
| 143 |
+
if str(metadata.get("base_model") or "").upper().split()[0] != "NONE":
|
| 144 |
+
errors.append("cosmos_born metadata no longer reports a from-scratch base")
|
| 145 |
+
return errors
|
| 146 |
+
|
| 147 |
+
|
| 148 |
+
def main() -> int:
|
| 149 |
+
parser = argparse.ArgumentParser()
|
| 150 |
+
parser.add_argument(
|
| 151 |
+
"--no-strict",
|
| 152 |
+
action="store_true",
|
| 153 |
+
help="allow extra files not listed in the release manifest",
|
| 154 |
+
)
|
| 155 |
+
args = parser.parse_args()
|
| 156 |
+
if not MANIFEST.is_file():
|
| 157 |
+
print("[FAIL] RELEASE_MANIFEST.json is missing")
|
| 158 |
+
return 1
|
| 159 |
+
|
| 160 |
+
errors = []
|
| 161 |
+
errors.extend(check_manifest(strict=not args.no_strict))
|
| 162 |
+
errors.extend(check_blank_credentials())
|
| 163 |
+
archive_errors, archive_stats = check_public_archive()
|
| 164 |
+
errors.extend(archive_errors)
|
| 165 |
+
errors.extend(check_model_metadata())
|
| 166 |
+
|
| 167 |
+
if errors:
|
| 168 |
+
print(f"[FAIL] {len(errors)} release check(s) failed")
|
| 169 |
+
for error in errors:
|
| 170 |
+
print(" -", error)
|
| 171 |
+
return 1
|
| 172 |
+
|
| 173 |
+
print("[OK] release manifest hashes verified")
|
| 174 |
+
print("[OK] shipped cloud credential fields are blank")
|
| 175 |
+
print("[OK] cosmos_born metadata is internally consistent")
|
| 176 |
+
print(
|
| 177 |
+
"[OK] public archive:",
|
| 178 |
+
f"{archive_stats['records']:,} records,",
|
| 179 |
+
f"{archive_stats['samples']:,} samples",
|
| 180 |
+
)
|
| 181 |
+
for category in sorted(archive_stats["records_by_class"]):
|
| 182 |
+
print(
|
| 183 |
+
" ",
|
| 184 |
+
category,
|
| 185 |
+
f"{archive_stats['records_by_class'][category]:,} records /",
|
| 186 |
+
f"{archive_stats['samples_by_class'][category]:,} samples",
|
| 187 |
+
)
|
| 188 |
+
return 0
|
| 189 |
+
|
| 190 |
+
|
| 191 |
+
if __name__ == "__main__":
|
| 192 |
+
raise SystemExit(main())
|
| 193 |
+
|