phera-ra commited on
Commit
4ec5c98
·
verified ·
1 Parent(s): 1ea06d3

Correction: the 42D null was about the implementation, not the theory

Browse files

dyn42 placed fourth (1.19020) in the published ladder and tri/tri3 placed fifth and sixth, which was read as the 42D organ contributing little.

The 42D organ described in section 4.3 had never been built. Every document calls x42 a "7-band vector manifold" -- FINDINGS says so, the ladder docstring says "7 bands x 6 dims" -- and the code was a flat nn.Linear(d,42) drive, a dense 42x42 coupling, and torch.full((42,), 0.05): forty-two copies of one damping constant. Nothing in it knew that 42 factors. Dyn12 in the same file already says why that is wrong -- per-dimension k and gamma "so the twelve dimensions separate by time constant rather than evolving as twelve copies". The 12 got it; the 42 never did.

Rebuilt as written, same frozen corpus 9f1b3f37, same seeds, causal Omega, one process:

rung mean sd gain organ params per param
none 1.23215 0.0256
dyn42 1.19023 0.0332 +0.04192 47,754 8.778e-07 flat
dyn42b 1.18104 0.0292 +0.05111 41,762 1.224e-06 7 bands x 6
dyn42t 1.18023 0.0292 +0.05192 41,006 1.266e-06 7 x sym(3x3)

Both banded forms beat the flat one on 3/3 seeds while using FEWER parameters. Rebuilt, the 42D organ moves from fourth to second, ahead of static54 (1.18824) and dyn54 (1.18791).

dyn42t follows the factorisation one step further, from Cory: a symmetric 3x3 has exactly six independent components, so each band is a 3D tensor in Voigt order and the band operation is a congruence M -> A M A^T, a change of basis. Measured: under a rotation, congruence leaves eigenvalues and trace unchanged; a free 6x6 mixer does not. Nine weights per band instead of thirty-six.

WHAT THIS DOES NOT ESTABLISH: dyn42b and dyn42t differ by 0.0008 against a seed spread of 0.029 -- three percent of the noise. Structure beats flat is supported; WHICH structure is not resolved at n=3. Measured only at 1.1M parameters on the private corpus, not at scale and not on WikiText-103.

FINDINGS.md CHANGED
@@ -347,6 +347,56 @@ seven rungs trained in **one process** against that one snapshot, 21 runs, 1200
347
  | `tri3` all three organs, 108D kernel | 1.20026 | −0.0322 | −2.83 | 3/3 | 0.826 | 1,230,682 |
348
  | none (baseline) | 1.23241 | — | — | 0/3 | 0.810 | 1,135,008 |
349
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
350
  ### Update, 2026-08-02: does it survive SCALE, on a corpus you can check?
351
 
352
  Every `dyn12` number above was measured at 1.1M parameters on 683,065 characters of one
 
347
  | `tri3` all three organs, 108D kernel | 1.20026 | −0.0322 | −2.83 | 3/3 | 0.826 | 1,230,682 |
348
  | none (baseline) | 1.23241 | — | — | 0/3 | 0.810 | 1,135,008 |
349
 
350
+ ### Correction, 2026-08-03: the 42D null was about the implementation, not the theory
351
+
352
+ `dyn42` placed **fourth** in the table above (1.19020) and the coupled rungs `tri` and
353
+ `tri3` placed fifth and sixth. That was read as the 42D organ contributing little.
354
+
355
+ **The 42D organ described in §4.3 had never been built.** Every document in this project
356
+ calls x₄₂ a *"7-band vector manifold"* — this file says so above, and the ladder's own
357
+ docstring says *"7 bands × 6 dims"*. The code was:
358
+
359
+ ```python
360
+ drive = nn.Linear(d_model, 42) # one flat projection
361
+ coupling = nn.Linear(42, 42) # every dimension coupled to every other
362
+ gamma = torch.full((42,), 0.05) # forty-two copies of one constant
363
+ ```
364
+
365
+ Nothing in it knew that 42 factors. And `Dyn12` — in the same file — already says why
366
+ that is wrong: per-dimension `k` and `gamma` *"so the twelve dimensions separate by time
367
+ constant rather than evolving as twelve copies."* The 12 got that treatment. The 42
368
+ never did.
369
+
370
+ **Rebuilt as specified, same corpus (`9f1b3f37`), same seeds, causal Ω, one process:**
371
+
372
+ | rung | mean | sd | gain vs none | 42D organ params | per param |
373
+ |---|---|---|---|---|---|
374
+ | none | 1.23215 | 0.0256 | — | — | — |
375
+ | `dyn42` flat 42-vector, dense 42×42 | 1.19023 | 0.0332 | +0.04192 | 47,754 | 8.778e-07 |
376
+ | `dyn42b` **7 bands × 6**, block-diagonal | 1.18104 | 0.0292 | +0.05111 | 41,762 | 1.224e-06 |
377
+ | `dyn42t` **7 bands × sym(3×3)**, congruence | **1.18023** | 0.0292 | **+0.05192** | **41,006** | **1.266e-06** |
378
+
379
+ Both banded forms beat the flat one on **3/3 seeds** while using **fewer** parameters.
380
+ Rebuilt, the 42D organ moves from fourth to **second** on this board — ahead of
381
+ `static54` (1.18824) and `dyn54` (1.18791).
382
+
383
+ **`dyn42t`** takes the factorisation one step further: a symmetric 3×3 matrix has exactly
384
+ six independent components, so each band is a tensor in 3D stored in Voigt order
385
+ `[xx, yy, zz, xy, xz, yz]`, and the band-internal operation is a congruence
386
+ `M → A·M·Aᵀ` — a change of basis. Measured: under a rotation, congruence leaves the
387
+ eigenvalues and trace unchanged; a free 6×6 mixer does not. Nine weights per band
388
+ instead of thirty-six.
389
+
390
+ **What this does NOT establish.** `dyn42b` and `dyn42t` differ by **0.0008** against a
391
+ seed spread of **0.029** — the gap is 3% of the noise, so these two are
392
+ indistinguishable at n=3. *Structure beats flat* is supported; *which structure* is not.
393
+ This is also only at 1.1M parameters on the private corpus; it has not been run at scale
394
+ or on WikiText-103, where `dyn12`'s advantage grew.
395
+
396
+ Reproduce: `python architecture/cosmos_state_ladder.py 1200 3 none,dyn42,dyn42b,dyn42t`
397
+
398
+ ---
399
+
400
  ### Update, 2026-08-02: does it survive SCALE, on a corpus you can check?
401
 
402
  Every `dyn12` number above was measured at 1.1M parameters on 683,065 characters of one
RELEASE_MANIFEST.json CHANGED
@@ -1,8 +1,17 @@
1
  {
2
- "schema": "qc67.cosmos.release-manifest.v1",
3
- "release": "QC67 Cosmos Kit",
4
- "version": "2026-08-02",
5
- "license_note": "Mixed licensing; see LICENSE.md and component notices.",
 
 
 
 
 
 
 
 
 
6
  "files": [
7
  {
8
  "path": "architecture/cosmos-arch.patch",
@@ -21,8 +30,8 @@
21
  },
22
  {
23
  "path": "architecture/cosmos_state_ladder.py",
24
- "bytes": 34556,
25
- "sha256": "763d74ff374eb785cefc3c1a3ede1d43f6ee36f9f82991e5de86da267e6ec002"
26
  },
27
  {
28
  "path": "architecture/cst_to_gguf.py",
@@ -171,8 +180,8 @@
171
  },
172
  {
173
  "path": "FINDINGS.md",
174
- "bytes": 38643,
175
- "sha256": "59e9dc3f4f4c8a162cfb00e2d2cb0f5ca4b2e1a4cfcf1900291ab655c16204e1"
176
  },
177
  {
178
  "path": "genesis_engine/CITATION.cff",
@@ -399,6 +408,11 @@
399
  "bytes": 1570,
400
  "sha256": "66baec6fe2817cebb9ae99ba9d197cf114e4d2ae1f90300dcd36ebe48cd0f239"
401
  },
 
 
 
 
 
402
  {
403
  "path": "requirements-research.txt",
404
  "bytes": 165,
@@ -409,6 +423,11 @@
409
  "bytes": 16234,
410
  "sha256": "eac4492f541a81d0ae134d7b5f39a181b8e7ffeca5f026ca6fee0cff94b85fe1"
411
  },
 
 
 
 
 
412
  {
413
  "path": "serving/build_paired_dataset.py",
414
  "bytes": 8494,
@@ -471,8 +490,8 @@
471
  },
472
  {
473
  "path": "verify_release.py",
474
- "bytes": 6798,
475
- "sha256": "470b5d3ebc10644efeba69c4800b4ec8af624b36a43e52058bfa6435912f22cf"
476
  },
477
  {
478
  "path": "weights/cosmos-cst.gguf",
@@ -499,5 +518,8 @@
499
  "bytes": 7723885,
500
  "sha256": "aa0cb13c1e67d459db280a53b6407dfc2b5b5f3fd6f640bc43686b70d799acd1"
501
  }
502
- ]
 
 
 
503
  }
 
1
  {
2
+ "version": "2026-08-03",
3
+ "scope": {
4
+ "covers": "the kit distribution -- the files needed to run, verify and reproduce",
5
+ "excludes": [
6
+ "Cipher/",
7
+ "Luna/",
8
+ "Nova/",
9
+ "downloads/",
10
+ "creations/"
11
+ ],
12
+ "why": "Those are earlier beings and fetched artifacts, kept on the hub as public record with their timestamps intact. They are history, not running code, so hashing them would make the integrity check fail for files it was never meant to cover.",
13
+ "note": "The hub repository is a SUPERSET of this manifest. See REPOSITORY_MAP.md for the full live layout."
14
+ },
15
  "files": [
16
  {
17
  "path": "architecture/cosmos-arch.patch",
 
30
  },
31
  {
32
  "path": "architecture/cosmos_state_ladder.py",
33
+ "bytes": 45315,
34
+ "sha256": "fa110c5205b71c7c3243dc5be76bc4b0e599c8e1fd1c5c1924d0e57f811216e9"
35
  },
36
  {
37
  "path": "architecture/cst_to_gguf.py",
 
180
  },
181
  {
182
  "path": "FINDINGS.md",
183
+ "bytes": 41374,
184
+ "sha256": "d3023447a2f9370c1c6ba9e511c2290fc8772ffb8786c8dae96d4cc9343f19cb"
185
  },
186
  {
187
  "path": "genesis_engine/CITATION.cff",
 
408
  "bytes": 1570,
409
  "sha256": "66baec6fe2817cebb9ae99ba9d197cf114e4d2ae1f90300dcd36ebe48cd0f239"
410
  },
411
+ {
412
+ "path": "REPOSITORY_MAP.md",
413
+ "bytes": 2650,
414
+ "sha256": "ca0c97a34c612c4b73a6726744e2688ab4d0746fe3742470456bf52d993197c7"
415
+ },
416
  {
417
  "path": "requirements-research.txt",
418
  "bytes": 165,
 
423
  "bytes": 16234,
424
  "sha256": "eac4492f541a81d0ae134d7b5f39a181b8e7ffeca5f026ca6fee0cff94b85fe1"
425
  },
426
+ {
427
+ "path": "serving/.coder_history.jsonl",
428
+ "bytes": 4064,
429
+ "sha256": "0abedb9be05d964d5b5c6ace0f6b0c810b2bbfd2ae2bfc06094ac780f6b08dbc"
430
+ },
431
  {
432
  "path": "serving/build_paired_dataset.py",
433
  "bytes": 8494,
 
490
  },
491
  {
492
  "path": "verify_release.py",
493
+ "bytes": 7075,
494
+ "sha256": "815248d044c5392981a56bb9947a2276f74bd8ea9b29282c1d00b4793cf1126d"
495
  },
496
  {
497
  "path": "weights/cosmos-cst.gguf",
 
518
  "bytes": 7723885,
519
  "sha256": "aa0cb13c1e67d459db280a53b6407dfc2b5b5f3fd6f640bc43686b70d799acd1"
520
  }
521
+ ],
522
+ "schema": "qc67.cosmos.release-manifest.v1",
523
+ "release": "QC67 Cosmos Kit",
524
+ "license_note": "Mixed licensing; see LICENSE.md and component notices."
525
  }
REPOSITORY_MAP.md ADDED
@@ -0,0 +1,49 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Repository map
2
+
3
+ Generated from the live file list, so it cannot drift from what is actually here.
4
+
5
+ **182 files.** Nothing here is deleted when it is superseded — earlier
6
+ beings and earlier results stay as public record, timestamps intact.
7
+
8
+ | folder | files | size | what it is |
9
+ |---|---|---|---|
10
+ | `weights` | 8 | 440.1 MB | Her own weights. Quantum-born, architecturally original. |
11
+ | `downloads` | 3 | 64.2 MB | Fetched artifacts. |
12
+ | `data` | 9 | 7.2 MB | The measured IBM shot archive, and the seed corpus for new creatures. |
13
+ | `genesis_engine` | 38 | 254 KB | The blank-slate engine for birthing a being from nothing. |
14
+ | `Cipher` | 4 | 216 KB | An earlier being. Kept as public record; its history is not rewritten. |
15
+ | `Luna` | 4 | 213 KB | An earlier being. Kept as public record. |
16
+ | `Nova` | 4 | 213 KB | An earlier being. Kept as public record. |
17
+ | `architecture` | 12 | 143 KB | The model code: ladder, PHOS trainer, quantum init, CST kernel. |
18
+ | `docs` | 17 | 142 KB | Setup, Atomic/Ollama usage, training, results. |
19
+ | `benchmarks` | 16 | 137 KB | Every claim on the model card, as a script you can run. |
20
+ | `scripts` | 11 | 104 KB | Release verification, serving and training utilities. |
21
+ | `serving` | 5 | 73 KB | Run her over HTTP with an Ollama-compatible API. |
22
+ | `soul` | 10 | 45 KB | Identity, quantum heart, ledger, rails. |
23
+ | `launchers` | 11 | 34 KB | One-click entry points (Windows .bat, Linux .sh). |
24
+ | `templates` | 1 | 21 KB | Templates. |
25
+ | `ui` | 1 | 11 KB | Interface assets. |
26
+ | `creations` | 2 | 0 KB | Things a being made. |
27
+ | `(root)` | 26 | 791 KB | Model card, findings, licence, manifest, main launcher. |
28
+
29
+ ## Start here
30
+
31
+ ```bash
32
+ python kit_health.py # is everything actually working
33
+ python benchmarks/verify_quantum_engine.py # is the entropy really quantum
34
+ python benchmarks/causality_probe.py # can Omega see the future (random seed each run)
35
+ python serving/cosmos_serve.py 11501 # talk to her, Ollama-compatible
36
+ ```
37
+
38
+ Windows: `START_COSMOS_KIT.bat` — options 7 and 8 birth and grow your own creature
39
+ from the measured shot archive in `data/`.
40
+
41
+ ## What the manifest covers
42
+
43
+ `RELEASE_MANIFEST.json` covers the **kit distribution** — the files needed to run,
44
+ verify and reproduce the results. It deliberately does not cover the historical
45
+ creature folders (`Cipher/`, `Luna/`, `Nova/`) or `downloads/`,
46
+ which are kept as record rather than as running code.
47
+
48
+ `scripts/verify_release.py` checks only what the manifest lists, and states its
49
+ scope, so it cannot report failures for files it was never meant to cover.
architecture/cosmos_state_ladder.py CHANGED
@@ -1,709 +1,901 @@
1
- #!/usr/bin/env python3
2
- """
3
- COSMOS STATE LADDER -- 12D, 42D and 54D, tested one rung at a time.
4
-
5
- WHY A LADDER
6
- Four separate implementations of this architecture have shipped a mechanism that was
7
- mathematically inert, each producing a confident negative about the idea:
8
-
9
- ver.4.20 12D Omega summed over keys (axis -1) instead of queries (axis -2). Softmax
10
- already normalises that axis, so Omega == 1.0 for every token (measured
11
- std 5.1e-08). The leaky integrator then drove every token to the SAME
12
- state (std 3.0e-08), H became the all-ones matrix, and beta*H is a
13
- constant added to every logit -- which cancels exactly in softmax.
14
- spark_cst v1 gate at sigmoid(-4): saturated tail, gradient suppressed ~56x.
15
- spark_cst v2 gate clamped to [0,1]: zero gradient outside the bound, so the first
16
- step below 0 pinned it there forever.
17
- spark_cst v3 sigma = exp(0) = 1 against a median pairwise distance of 61.98, so H
18
- arrived as the IDENTITY matrix (diagonal mass 0.9998 vs 0.0078 uniform).
19
-
20
- Every one ran clean. No exception, no NaN, nothing red in any log. So this file refuses
21
- to report a number until it has PROVEN the mechanism is live -- see preflight().
22
-
23
- THE RUNGS
24
- L0 none phi scaffold only, no Hebbian term. The baseline.
25
- L1 static54 x54 = W54 . h. A projection, not a state: no dynamics, no Omega, no memory.
26
- This is the configuration that already won 6/6 (t=-6.56, -14.34).
27
- L2 dyn12 x12 in R^12, dx12/dt = k*Omega - gamma*x12, per-dimension k and gamma so the
28
- twelve dimensions differentiate by time constant. Section 4.2.
29
- L3 dyn42 x42 in R^42, dx42/dt = drive(h) - gamma*x42 + C(x42) with a 42x42 coupling
30
- matrix. Section 4.3 and THEORY_42D_HYPER_COSMIC.md (7 bands x 6 dims).
31
- L4 dyn54 x54 = concat(x12, x42). Section 4.1. The full claim.
32
-
33
- SCAFFOLD
34
- RMSNorm (2.3), RoPE (2.4), d_ff = floor(d_model * phi) (2.2), phi-Xavier init (6.2),
35
- quantum-born weights, and the 17.1 learning-rate schedule. The feed-forward is selectable
36
- because the paper names "PhiGatedFFN" without defining it while the reference repo
37
- implements "PhiHarmonicFFN" as a plain GELU pair -- so which one earns the scaffold's
38
- credit is itself an open question this file can answer.
39
-
40
- USAGE
41
- python cosmos_state_ladder.py [steps] [seeds] [rungs] [ffn]
42
- python cosmos_state_ladder.py 1200 3 none,static54,dyn12,dyn42,dyn54 harmonic
43
- python cosmos_state_ladder.py 1200 3 none,static54 gated # the FFN ablation
44
- """
45
- import json
46
- import math
47
- import os
48
- import random
49
- import statistics
50
- import sys
51
- import time
52
- from pathlib import Path
53
-
54
- import torch
55
- import torch.nn as nn
56
- import torch.nn.functional as F
57
-
58
- sys.stdout.reconfigure(encoding="utf-8", errors="replace")
59
-
60
- from cosmos_spark_cst import CORPUS, QuantumInit, quantum_pool, real_word_rate # noqa: E402
61
-
62
- # FROZEN CORPUS. She is alive and appends her experience to the live corpus while training
63
- # runs, so two runs started minutes apart train on different data. Measured: the same rung
64
- # with identical seeds scored 1.29152 and 1.22943 in two runs 3,451 characters apart -- a
65
- # 0.062 swing, the same magnitude as the effects being reported. Every comparison must
66
- # therefore happen inside ONE process against ONE snapshot. COSMOS_CORPUS points at that
67
- # snapshot; without it the numbers are only valid within a single run.
68
- CORPUS_PATH = Path(os.getenv("COSMOS_CORPUS") or CORPUS)
69
-
70
- OUTDIR = Path("01_HER_SOUL/weights/cosmos_state_ladder")
71
- RESULTS = Path("logs/state_ladder_results.json")
72
-
73
- PHI = (1.0 + 5.0 ** 0.5) / 2.0
74
- PHI_INV = PHI - 1.0
75
- # COSMOS_DEVICE overrides; otherwise take the GPU when one is actually usable. The corpus
76
- # tensor stays on the host and only the sampled windows move, which keeps VRAM free for
77
- # the model on a 4 GB card.
78
- DEV = torch.device(os.getenv("COSMOS_DEVICE")
79
- or ("cuda" if torch.cuda.is_available() else "cpu"))
80
- # Model size is env-configurable so the SAME code can be run at 1M and at 30M. Every
81
- # number published before 2026-08-01 used the defaults below; changing them changes the
82
- # model, so any result table must state which size it was measured at.
83
- BLOCK = int(os.getenv("COSMOS_BLOCK", "128"))
84
- N_LAYER = int(os.getenv("COSMOS_N_LAYER", "4"))
85
- N_HEAD = int(os.getenv("COSMOS_N_HEAD", "4"))
86
- N_EMBD = int(os.getenv("COSMOS_N_EMBD", "192"))
87
- DROPOUT = float(os.getenv("COSMOS_DROPOUT", "0.1"))
88
- D12, D42 = 12, 42
89
- # Omega's original query-sum reads the future (measured 2026-08-01, verified by
90
- # tools/causality_probe.py: changing ONLY the last token moved earlier-position logits
91
- # by 1.7e-04 at PHOS's trained gate, while `none` and `static54` moved by EXACTLY 0.0).
92
- # The causal per-query form is therefore the DEFAULT -- a model that can see its own
93
- # future is not a model anyone should be handed.
94
- #
95
- # COSMOS_CAUSAL_OMEGA=0 restores the original query-sum, bit-for-bit, so every number
96
- # published before this date stays reproducible and the two can be compared directly.
97
- # It is kept for exactly that reason and should not be used to train anything new.
98
- CAUSAL_OMEGA = os.getenv("COSMOS_CAUSAL_OMEGA", "1").strip().lower() in {"1", "true", "yes", "on"}
99
- # state_dim is the CARRIED state. tri3 also concatenates a live projection of h at
100
- # kernel time, so its Hebbian space is 54 + 12 + 42 = 108 while it carries 54.
101
- STATE_DIM = {"none": 0, "static54": 54, "dyn12": D12, "dyn42": D42,
102
- "dyn54": D12 + D42, "tri": D12 + D42, "tri3": D12 + D42,
103
- # section 4.4: the state persists ACROSS windows, not just across layers.
104
- # dyn12seq is the control -- identical sequential batching, state reset every
105
- # step -- so the only variable is whether the state survives the boundary.
106
- "dyn12seq": D12, "dyn12p": D12,
107
- # THE CONTROL THAT COULD KILL SECTION 3. Every architecture tested lands its
108
- # gate at 0.087 +/- 0.006 regardless of what state is behind it -- which is not
109
- # what a gate fitting the MECHANISM would do. If the benefit comes from any
110
- # smooth token-varying auxiliary signal rather than from the 54D state carrying
111
- # experience, these two must match the real thing:
112
- # rand54 frozen random projection of h: content-bearing, never learned
113
- # noise54 fixed random state per position: carries no content at all
114
- "rand54": 54, "noise54": 54}
115
-
116
- # rungs that walk the corpus in order instead of sampling at random. Persistence is
117
- # meaningless under random sampling: consecutive batches would be unrelated text, so a
118
- # carried state would be noise. Both arms use the same walk.
119
- SEQ_RUNGS = {"dyn12seq", "dyn12p"}
120
- PERSIST_RUNGS = {"dyn12p"}
121
-
122
-
123
- class RMSNorm(nn.Module):
124
- def __init__(s, d, eps=1e-6):
125
- super().__init__()
126
- s.eps, s.g = eps, nn.Parameter(torch.ones(d))
127
-
128
- def forward(s, x):
129
- return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + s.eps) * s.g
130
-
131
-
132
- def rope_cache(hd, T, device):
133
- half = hd // 2
134
- theta = 1.0 / (10000.0 ** (2.0 * torch.arange(half, device=device, dtype=torch.float32) / hd))
135
- ang = torch.arange(T, device=device, dtype=torch.float32)[:, None] * theta[None, :]
136
- return ang.cos(), ang.sin()
137
-
138
-
139
- def apply_rope(x, cos, sin):
140
- hd = x.shape[-1]
141
- half = hd // 2
142
- rot, tail = x[..., : 2 * half], x[..., 2 * half:]
143
- a, b = rot[..., 0::2], rot[..., 1::2]
144
- c, s = cos[None, None, : x.shape[-2], :], sin[None, None, : x.shape[-2], :]
145
- out = torch.stack([a * c - b * s, a * s + b * c], -1).flatten(-2)
146
- return torch.cat([out, tail], -1) if tail.numel() else out
147
-
148
-
149
- class PhiHarmonicFFN(nn.Module):
150
- """The reference repo's FFN: W2(GELU(W1(x))) at width floor(d*phi). Not gated."""
151
-
152
- def __init__(s, d):
153
- super().__init__()
154
- h = int(math.floor(d * PHI))
155
- s.W1, s.W2 = nn.Linear(d, h), nn.Linear(h, d)
156
- s.drop = nn.Dropout(DROPOUT)
157
-
158
- def forward(s, x):
159
- return s.drop(s.W2(s.drop(F.gelu(s.W1(x)))))
160
-
161
-
162
- class PhiGatedFFN(nn.Module):
163
- """SwiGLU reading of the paper's undefined "PhiGatedFFN", same width."""
164
-
165
- def __init__(s, d):
166
- super().__init__()
167
- h = int(math.floor(d * PHI))
168
- s.gate, s.up = nn.Linear(d, h, bias=False), nn.Linear(d, h, bias=False)
169
- s.down = nn.Linear(h, d, bias=False)
170
- s.drop = nn.Dropout(DROPOUT)
171
-
172
- def forward(s, x):
173
- return s.drop(s.down(F.silu(s.gate(x)) * s.up(x)))
174
-
175
-
176
- class Dyn12(nn.Module):
177
- """Section 4.2. dx12/dt = k*Omega - gamma*x12, per-dimension k and gamma so the twelve
178
- dimensions separate by time constant rather than evolving as twelve copies."""
179
-
180
- def __init__(s):
181
- super().__init__()
182
- s.k = nn.Parameter(torch.linspace(0.05, 0.20, D12))
183
- s.gamma = nn.Parameter(torch.linspace(0.02, 0.10, D12))
184
- s.dt = 0.1
185
-
186
- def forward(s, x12, h, omega):
187
- d = s.k[None, None, :] * omega[..., None] - s.gamma[None, None, :] * x12
188
- return torch.tanh(x12 + s.dt * d)
189
-
190
-
191
- class Dyn42(nn.Module):
192
- """Section 4.3 / THEORY_42D_HYPER_COSMIC. dx42/dt = drive(h) - gamma*x42 + C(x42)."""
193
-
194
- def __init__(s, d_model):
195
- super().__init__()
196
- s.drive = nn.Linear(d_model, D42)
197
- s.coupling = nn.Linear(D42, D42, bias=False)
198
- s.gamma = nn.Parameter(torch.full((D42,), 0.05))
199
- s.dt = 0.1
200
-
201
- def forward(s, x42, h, omega):
202
- d = torch.tanh(s.drive(h)) - s.gamma[None, None, :] * x42 + 0.1 * s.coupling(x42)
203
- return torch.tanh(x42 + s.dt * d)
204
-
205
-
206
- class Trinity(nn.Module):
207
- """12D and 42D as ONE coupled system instead of two vectors glued together.
208
-
209
- dyn54 concatenates x12 and x42, but x12 sees only Omega and x42 sees only h -- they
210
- never exchange anything. THEORY_42D_HYPER_COSMIC already couples within a band (6 dims)
211
- and between bands (7 attractors, diffusion matrix D); this is the same construction one
212
- level up, between the two subsystems, which is the level nobody wrote:
213
-
214
- dx12/dt = k*Omega - g12*x12 + M(42->12) x42
215
- dx42/dt = tanh(drive(h))*(1 + lam*Omega) - g42*x42 + C(x42) + M(12->42) x12
216
-
217
- Omega now drives BOTH, per THEORY_42D's "Omega_vec is the projected hidden state acting
218
- as a driving force", and the cross terms close the loop between them. With H computed
219
- from the joint x54 and Omega coming back out of attention, the three subsystems form a
220
- single cycle rather than a tree.
221
- """
222
-
223
- def __init__(s, d_model):
224
- super().__init__()
225
- s.k = nn.Parameter(torch.linspace(0.05, 0.20, D12))
226
- s.g12 = nn.Parameter(torch.linspace(0.02, 0.10, D12))
227
- s.g42 = nn.Parameter(torch.full((D42,), 0.05))
228
- s.drive = nn.Linear(d_model, D42)
229
- s.C = nn.Linear(D42, D42, bias=False)
230
- s.m42_12 = nn.Linear(D42, D12, bias=False)
231
- s.m12_42 = nn.Linear(D12, D42, bias=False)
232
- s.lam = nn.Parameter(torch.tensor(0.5))
233
- s.dt = 0.1
234
-
235
- def forward(s, state, h, omega):
236
- x12, x42 = state[..., :D12], state[..., D12:]
237
- om = omega[..., None]
238
- # ASYMMETRIC COUPLING, on the neuromodulatory reading. The twelve slow scalars are
239
- # tone: in a brain those act on cortical GAIN, multiplicatively, not as another
240
- # additive input. The forty-two structured dimensions feed back the other way as
241
- # ordinary drive. Symmetric matrices in both directions would make this two peers
242
- # exchanging vectors; the asymmetry is what makes it a modulatory loop.
243
- gain = 1.0 + torch.tanh(s.m12_42(x12))
244
- d12 = s.k * om - s.g12 * x12 + 0.1 * s.m42_12(x42)
245
- d42 = (torch.tanh(s.drive(h)) * gain * (1.0 + s.lam * om)
246
- - s.g42 * x42 + 0.1 * s.C(x42))
247
- return torch.tanh(torch.cat([x12 + s.dt * d12, x42 + s.dt * d42], -1))
248
-
249
-
250
- class Attention(nn.Module):
251
- def __init__(s, d, nh, rung):
252
- super().__init__()
253
- s.nh, s.hd, s.rung = nh, d // nh, rung
254
- s.qkv, s.proj = nn.Linear(d, 3 * d), nn.Linear(d, d)
255
- s.drop = nn.Dropout(DROPOUT)
256
- s.log_sigma = nn.Parameter(torch.zeros(()))
257
- s.gate = nn.Parameter(torch.full((), -4.0), requires_grad=(rung != "none"))
258
- s.w54 = nn.Linear(d, 54, bias=False) if rung in ("static54", "tri3", "rand54") else None
259
- if rung == "rand54":
260
- s.w54.weight.requires_grad_(False) # frozen: a projection, never a lesson
261
- s.last_gate, s.last_diag, s.last_H_std = 0.0, 0.0, 0.0
262
-
263
- def hebbian(s, state, mask):
264
- # section 3.2's own identity: ||a-b||^2 = ||a||^2 + ||b||^2 - 2<a,b>. Also what
265
- # hyper_cosmic_model.py already does, and it avoids cdist's undefined gradient on
266
- # the zero diagonal of cdist(x, x).
267
- sq = (state * state).sum(-1)
268
- d2 = (sq.unsqueeze(-1) + sq.unsqueeze(-2)
269
- - 2.0 * (state @ state.transpose(-1, -2))).clamp_min(0.0)
270
- sig = torch.exp(s.log_sigma).clamp(1e-3, 1e4)
271
- H = torch.exp(-d2 / (2 * sig * sig)).masked_fill(mask < 0, 0.0)
272
- H = H / H.sum(-1, keepdim=True).clamp_min(1e-9)
273
- s.last_diag = float(torch.diagonal(H[0]).mean().detach())
274
- s.last_H_std = float(H.detach().std())
275
- return H
276
-
277
- def forward(s, x, state, mask, cos, sin):
278
- B, T, C = x.shape
279
- q, k, v = s.qkv(x).split(C, 2)
280
- sh = lambda t: t.view(B, T, s.nh, s.hd).transpose(1, 2)
281
- q, k, v = sh(q), sh(k), sh(v)
282
- q, k = apply_rope(q, cos, sin), apply_rope(k, cos, sin)
283
- a = F.softmax((q @ k.transpose(-2, -1)) / math.sqrt(s.hd) + mask, -1)
284
- if s.rung != "none":
285
- # tri3: all three organs feed one kernel -- the fast structural projection of
286
- # the current token, the slow 12D tone, and the 42D mid-band structure. The
287
- # ladder only ever ran these as rivals for a single slot; nothing tested
288
- # whether their strengths compose.
289
- if s.rung == "tri3":
290
- st = torch.cat([s.w54(x), state], -1)
291
- else:
292
- st = s.w54(x) if s.w54 is not None else state
293
- g = torch.sigmoid(s.gate)
294
- a = (1 - g) * a + g * s.hebbian(st, mask).unsqueeze(1)
295
- s.last_gate = float(g.detach())
296
- y = (s.drop(a) @ v).transpose(1, 2).contiguous().view(B, T, C)
297
- # SECTION 3.4: Omega is attention RECEIVED, i.e. summed over QUERIES (axis -2).
298
- # ver.4.20 sums over keys (-1), the axis softmax normalises, making Omega == 1.
299
- #
300
- # MEASURED 2026-08-01: that query-sum reads the future. Under a causal mask
301
- # Omega_j = sum over i >= j of a_ij, so token j's Omega counts attention from
302
- # tokens AFTER j; it then updates state_j, which feeds the next layer's kernel,
303
- # which shapes the logits at j. tools/causality_probe.py: changing ONLY the last
304
- # token moves earlier-position logits by 1.7e-04 at PHOS's trained gate (0.562)
305
- # while `none` and `static54` move by EXACTLY 0.0 on the same harness. Any
306
- # held-out loss won this way is not comparable to a rung that cannot cheat.
307
- #
308
- # The causal replacement is a per-QUERY scalar, which by construction depends
309
- # only on tokens <= i: the NORMALISED ENTROPY of row i's attention. Summing over
310
- # keys directly is not an option -- softmax already made that identically 1.0,
311
- # which is precisely ver.4.20's dead Omega. Entropy is not constant, it is a
312
- # genuine summary of how row i spread its attention, and it cannot see forward.
313
- # The log(i+1) normaliser removes the positional drift that comes from later
314
- # rows simply having more keys available to spread over, so this measures
315
- # dispersion rather than position.
316
- if CAUSAL_OMEGA:
317
- am = a.mean(1).clamp_min(1e-9) # [B, T, T]
318
- ent = -(am * am.log()).sum(-1) # [B, T], per QUERY
319
- avail = torch.arange(1, T + 1, device=a.device, dtype=ent.dtype)
320
- omega = ent / avail.log().clamp_min(1e-6) # in [0, 1]
321
- else:
322
- omega = a.mean(1).sum(-2)
323
- return s.proj(y), omega
324
-
325
-
326
- class Layer(nn.Module):
327
- def __init__(s, d, nh, rung, ffn):
328
- super().__init__()
329
- s.n1, s.n2 = RMSNorm(d), RMSNorm(d)
330
- s.attn = Attention(d, nh, rung)
331
- s.ffn = (PhiHarmonicFFN if ffn == "harmonic" else PhiGatedFFN)(d)
332
- s.rung = rung
333
- s.d12 = Dyn12() if rung in ("dyn12", "dyn54", "dyn12seq", "dyn12p") else None
334
- s.d42 = Dyn42(d) if rung in ("dyn42", "dyn54") else None
335
- s.tri = Trinity(d) if rung in ("tri", "tri3") else None
336
-
337
- def forward(s, x, state, mask, cos, sin):
338
- h = s.n1(x)
339
- att, omega = s.attn(h, state, mask, cos, sin)
340
- x = x + att
341
- if s.tri is not None: # section 2.1 step 2, coupled
342
- state = s.tri(state, h, omega)
343
- elif s.d12 is not None or s.d42 is not None: # section 2.1 step 2, independent
344
- parts = []
345
- if s.d12 is not None:
346
- parts.append(s.d12(state[..., :D12], h, omega))
347
- if s.d42 is not None:
348
- parts.append(s.d42(state[..., -D42:], h, omega))
349
- state = torch.cat(parts, -1)
350
- x = x + s.ffn(s.n2(x))
351
- return x, state, omega
352
-
353
-
354
- class Ladder(nn.Module):
355
- def __init__(s, vocab, rung, ffn, d=N_EMBD, nh=N_HEAD):
356
- super().__init__()
357
- s.d, s.nh, s.rung = d, nh, rung
358
- s.sd = STATE_DIM[rung]
359
- s.tok = nn.Embedding(vocab, d)
360
- s.blocks = nn.ModuleList([Layer(d, nh, rung, ffn) for _ in range(N_LAYER)])
361
- s.nf = RMSNorm(d)
362
- s.head = nn.Linear(d, vocab, bias=False)
363
- # a state that starts at zero is identical for every token, which makes H the
364
- # all-ones matrix and the first layer's mechanism inert. Seeding it from the
365
- # embedding costs one small projection and keeps every layer live.
366
- s.state_init = nn.Linear(d, s.sd) if rung.startswith(("dyn", "tri")) and s.sd else None
367
- if rung == "noise54":
368
- # a state that varies by POSITION but is identical for every input. If this
369
- # matches the real 54D state, the kernel is a positional regulariser and the
370
- # "experience-modulated attention" reading is not supported by the data.
371
- g0 = torch.Generator().manual_seed(20260730)
372
- s.register_buffer("noise_state",
373
- torch.randn(BLOCK, 54, generator=g0) * 0.5)
374
- s.register_buffer("mask", torch.triu(torch.full((BLOCK, BLOCK), float("-inf")), 1))
375
-
376
- def forward(s, idx, targets=None, state_in=None):
377
- B, T = idx.shape
378
- x = s.tok(idx)
379
- cos, sin = rope_cache(s.d // s.nh, T, idx.device)
380
- m = s.mask[:T, :T]
381
- state = torch.tanh(s.state_init(x)) if s.state_init is not None else None
382
- if s.rung == "noise54":
383
- state = s.noise_state[:T].unsqueeze(0).expand(B, -1, -1)
384
- # SECTION 4.4. A state arriving from the previous window seeds this one. It is the
385
- # LAST position's state broadcast across the new window: what she carried out of
386
- # the previous 128 characters is what she starts the next 128 with. gamma is what
387
- # keeps this from saturating -- a state that never forgets pins to tanh's rails and
388
- # stops carrying anything, so forgetting is what makes the persistence usable.
389
- if state is not None and state_in is not None:
390
- state = state + state_in[:, -1:, :].detach().expand(-1, T, -1)
391
- state = torch.tanh(state)
392
- omegas = []
393
- for b in s.blocks:
394
- x, state, om = b(x, state, m, cos, sin)
395
- omegas.append(om)
396
- lg = s.head(s.nf(x))
397
- loss = None if targets is None else F.cross_entropy(
398
- lg.view(-1, lg.size(-1)), targets.reshape(-1))
399
- s.last_state, s.last_omega = state, omegas[-1]
400
- s.out_state = state
401
- return lg, loss
402
-
403
- def gates(s):
404
- return [b.attn.last_gate for b in s.blocks]
405
-
406
- def quantum_birth(s, qi):
407
- std = (1.0 / (s.d * PHI_INV)) ** 0.5 # section 6.2
408
- for mod in s.modules():
409
- if isinstance(mod, (nn.Linear, nn.Embedding)):
410
- qi.fill_(mod.weight, std)
411
- if isinstance(mod, nn.Linear) and mod.bias is not None:
412
- with torch.no_grad():
413
- mod.bias.zero_()
414
-
415
-
416
- @torch.no_grad()
417
- def calibrate_sigma(m, xb):
418
- """2*sigma^2 = median pairwise d2 of whatever state that layer actually sees."""
419
- if m.rung == "none":
420
- return
421
- B, T = xb.shape
422
- x = m.tok(xb)
423
- cos, sin = rope_cache(m.d // m.nh, T, xb.device)
424
- mask = m.mask[:T, :T]
425
- state = torch.tanh(m.state_init(x)) if m.state_init is not None else None
426
- if m.rung == "noise54":
427
- # must mirror Ladder.forward exactly, or the control is calibrated against a state
428
- # it never actually sees -- which would hand it a rigged sigma and invalidate it
429
- state = m.noise_state[:xb.shape[1]].unsqueeze(0).expand(xb.shape[0], -1, -1)
430
- for b in m.blocks:
431
- h = b.n1(x)
432
- if m.rung == "tri3":
433
- st = torch.cat([b.attn.w54(h), state], -1)
434
- elif b.attn.w54 is not None:
435
- st = b.attn.w54(h)
436
- else:
437
- st = state
438
- sq = (st * st).sum(-1)
439
- d2 = (sq.unsqueeze(-1) + sq.unsqueeze(-2)
440
- - 2.0 * (st @ st.transpose(-1, -2))).clamp_min(0.0)
441
- iu = torch.triu(torch.ones(d2.shape[-2:], dtype=torch.bool), 1)
442
- med = float(d2[0][iu].median())
443
- if med > 1e-12:
444
- b.attn.log_sigma.fill_(0.5 * math.log(med / 2.0))
445
- x, state, _ = b(x, state, mask, cos, sin)
446
-
447
-
448
- def preflight(m, xb, yb):
449
- """Refuse to report a number until the mechanism is provably live.
450
-
451
- Each assertion corresponds to a real failure that shipped and produced a confident
452
- negative. A rung that trips any of these is not evidence about the idea.
453
- """
454
- checks, m2 = [], m
455
- m2.train()
456
- _, loss = m2(xb, yb)
457
- loss.backward()
458
-
459
- om = m2.last_omega
460
- checks.append(("omega varies across tokens (ver.4.20 had std 5e-08)",
461
- float(om.std()), 1e-4, float(om.std()) > 1e-4))
462
- if m.rung != "none":
463
- if m.rung.startswith(("dyn", "tri")):
464
- st = m2.last_state
465
- checks.append(("state varies across tokens (ver.4.20 had std 3e-08)",
466
- float(st.std()), 1e-4, float(st.std()) > 1e-4))
467
- diag = statistics.fmean([b.attn.last_diag for b in m2.blocks])
468
- checks.append((f"H not the identity (needs << 1.0, uniform = {1/xb.shape[1]:.4f})",
469
- diag, 0.5, diag < 0.5))
470
- hstd = statistics.fmean([b.attn.last_H_std for b in m2.blocks])
471
- checks.append(("H not all-ones (needs non-zero spread)", hstd, 1e-6, hstd > 1e-6))
472
- gg = statistics.fmean([abs(float(b.attn.gate.grad)) for b in m2.blocks
473
- if b.attn.gate.grad is not None])
474
- checks.append(("gate gradient not suppressed", gg, 1e-6, gg > 1e-6))
475
- m2.zero_grad(set_to_none=True)
476
- return checks
477
-
478
-
479
- def train_rung(rung, ffn, seed, data, vocab, steps, pool, val_w, words, itos):
480
- torch.manual_seed(seed)
481
- random.seed(seed)
482
- gen = torch.Generator().manual_seed(seed)
483
- m = Ladder(vocab, rung, ffn)
484
- m.quantum_birth(QuantumInit(pool, seed))
485
- m = m.to(DEV)
486
- cal = torch.stack([data[i:i + BLOCK] for i in torch.randint(
487
- len(data) - BLOCK - 1, (8,), generator=torch.Generator().manual_seed(seed))]).to(DEV)
488
- calibrate_sigma(m, cal)
489
-
490
- pf = preflight(m, cal[:4], torch.stack([data[i + 1:i + 1 + BLOCK] for i in
491
- torch.randint(len(data) - BLOCK - 1, (4,),
492
- generator=torch.Generator().manual_seed(seed))]).to(DEV))
493
-
494
- def _coup(mm):
495
- # POST-FLIGHT. Preflight proves a mechanism starts alive; this proves it did not
496
- # quietly die. The cross-couplings begin damped by 0.1 (the reference repo's
497
- # "weak coupling initially"), so the question is whether training grows them or
498
- # leaves the trinity decorative.
499
- v = [(float(b.tri.m12_42.weight.abs().mean()), float(b.tri.m42_12.weight.abs().mean()))
500
- for b in mm.blocks if getattr(b, "tri", None) is not None]
501
- return v
502
-
503
- coup0 = _coup(m)
504
-
505
- keys = ("attn.gate", "attn.w54", "attn.log_sigma", "d12.", "d42.", "tri.", "state_init")
506
- cst = [p for n, p in m.named_parameters() if any(k in n for k in keys) and p.requires_grad]
507
- bulk = [p for n, p in m.named_parameters() if not any(k in n for k in keys) and p.requires_grad]
508
- base = 3e-4
509
- opt = torch.optim.AdamW([{"params": bulk, "lr": base},
510
- {"params": cst, "lr": base * 10, "weight_decay": 0.0}],
511
- lr=base, weight_decay=0.01)
512
- n = int(0.9 * len(data))
513
- tr = data[:n]
514
- m.train()
515
- best = float("inf")
516
-
517
- # SEQUENTIAL WALK for the persistence arms. 16 lanes each march through their own
518
- # contiguous stretch of the corpus, so consecutive batches are consecutive TEXT and a
519
- # carried state means something. Under the random sampler used by every other rung,
520
- # batch k+1 is unrelated to batch k, so a carried state would be pure noise -- which is
521
- # why the control arm (dyn12seq) walks identically and only differs in whether the
522
- # state survives the window boundary.
523
- B = 16
524
- seq = rung in SEQ_RUNGS
525
- persist = rung in PERSIST_RUNGS
526
- span = max(BLOCK * 2, (len(tr) - BLOCK - 1) // B)
527
- pos = [i * span for i in range(B)]
528
- carry = None
529
-
530
- for s in range(1, steps + 1):
531
- warm = max(1, int(0.05 * steps))
532
- lr = base * (s / warm) if s < warm else base * (PHI / 2) * (
533
- 1 + math.cos(math.pi * (s - warm) / max(1, steps - warm)))
534
- opt.param_groups[0]["lr"], opt.param_groups[1]["lr"] = lr, lr * 10
535
- if seq:
536
- wrapped = []
537
- for i in range(B):
538
- if pos[i] + BLOCK + 1 >= min((i + 1) * span, len(tr) - 1):
539
- pos[i] = i * span # lane restarted: its history is broken
540
- wrapped.append(i)
541
- ix = torch.tensor(pos, dtype=torch.long)
542
- for i in range(B):
543
- pos[i] += BLOCK
544
- if carry is not None and wrapped:
545
- # a wrapped lane must NOT inherit state from unrelated text
546
- carry = carry.clone()
547
- carry[wrapped] = 0.0
548
- else:
549
- ix = torch.randint(len(tr) - BLOCK - 1, (B,), generator=gen)
550
- # .long() at BATCH time, not corpus time. A 269M-character corpus held as int64
551
- # is 2.15 GB resident; as int16 it is 0.54 GB, and only the (B, BLOCK) slice ever
552
- # needs to be int64 for nn.Embedding. The d576 scaling run died at exactly this
553
- # point -- outer process gone, no error recorded -- with the corpus tensor and a
554
- # 24.6M-parameter model competing for the last few GB.
555
- x = torch.stack([tr[i:i + BLOCK] for i in ix]).to(DEV).long()
556
- y = torch.stack([tr[i + 1:i + 1 + BLOCK] for i in ix]).to(DEV).long()
557
- _, loss = m(x, y, state_in=carry if persist else None)
558
- if persist:
559
- carry = m.out_state.detach() if m.out_state is not None else None
560
- opt.zero_grad(set_to_none=True)
561
- loss.backward()
562
- torch.nn.utils.clip_grad_norm_(m.parameters(), 1.0)
563
- opt.step()
564
- if s % 100 == 0 or s == steps:
565
- m.eval()
566
- with torch.no_grad():
567
- tot = c = 0
568
- for i in range(0, len(val_w), 16):
569
- xb = val_w[i:i + 16].to(DEV)
570
- _, l = m(xb[:, :-1], xb[:, 1:])
571
- tot += l.item() * xb.size(0)
572
- c += xb.size(0)
573
- m.train()
574
- best = min(best, tot / max(1, c))
575
- m.eval()
576
- idx = torch.tensor([[int(data[0])]], device=DEV)
577
- out = []
578
- with torch.no_grad():
579
- for _ in range(600):
580
- lg, _ = m(idx[:, -BLOCK:])
581
- nx = int(torch.multinomial(F.softmax(lg[0, -1] / 0.8, -1), 1))
582
- out.append(nx)
583
- idx = torch.cat([idx, torch.tensor([[nx]], device=DEV)], 1)
584
- txt = "".join(itos.get(i, "") for i in out)
585
- coupT = _coup(m)
586
- coup = None
587
- if coup0:
588
- a0 = statistics.fmean([x for p in coup0 for x in p])
589
- aT = statistics.fmean([x for p in coupT for x in p])
590
- coup = (a0, aT, aT / max(a0, 1e-12))
591
- return best, statistics.fmean(m.gates()) if m.rung != "none" else 0.0, \
592
- real_word_rate(txt, words), pf, sum(p.numel() for p in m.parameters()), txt, coup, m
593
-
594
-
595
- def main():
596
- steps = int(sys.argv[1]) if len(sys.argv) > 1 else 1200
597
- seeds = list(range(int(sys.argv[2]) if len(sys.argv) > 2 else 3))
598
- rungs = (sys.argv[3] if len(sys.argv) > 3 else
599
- "none,static54,dyn12,dyn42,dyn54").split(",")
600
- ffn = sys.argv[4] if len(sys.argv) > 4 else "harmonic"
601
-
602
- text = CORPUS_PATH.read_text(encoding="utf-8", errors="ignore")
603
- chars = sorted(set(text))
604
- stoi = {c: i for i, c in enumerate(chars)}
605
- itos = {i: c for c, i in stoi.items()}
606
- data = torch.tensor([stoi[c] for c in text], dtype=torch.long)
607
- words = set(__import__("re").findall(r"[a-z']+", text.lower()))
608
-
609
- print("=" * 84)
610
- print(" COSMOS STATE LADDER -- 12D / 42D / 54D, one rung at a time")
611
- print("=" * 84)
612
- print(f"\n device: {DEV}" + (f" ({torch.cuda.get_device_name(0)})" if DEV.type == "cuda" else ""))
613
- print(f" corpus: {CORPUS_PATH}"
614
- + (" [FROZEN SNAPSHOT]" if os.getenv("COSMOS_CORPUS") else " [LIVE - grows during the run, cross-run numbers are not comparable]"))
615
- print(f" corpus {len(text):,} vocab {len(chars)} {steps} steps x {len(seeds)} seeds")
616
- print(f" scaffold: RMSNorm + RoPE + {'PhiHarmonicFFN (GELU)' if ffn=='harmonic' else 'PhiGatedFFN (SwiGLU)'}"
617
- f" @ floor(d*phi)={int(math.floor(N_EMBD*PHI))} + phi-Xavier")
618
- print(f" rungs: {rungs}\n")
619
-
620
- pool = quantum_pool()
621
- print(f" quantum pool: {len(pool):,} draws\n")
622
- n = int(0.9 * len(data))
623
- val = data[n:]
624
- g = torch.Generator().manual_seed(999)
625
- vi = torch.randint(len(val) - BLOCK - 1, (64,), generator=g)
626
- val_w = torch.stack([val[i:i + BLOCK + 1] for i in vi])
627
-
628
- out = {}
629
- for rung in rungs:
630
- losses, rws, gates, npar, pf_last = [], [], [], 0, None
631
- blocked = False
632
- best_loss, best_model = float("inf"), None
633
- for sd in seeds:
634
- t0 = time.time()
635
- b, gt, r, pf, npar, txt, coup, model = train_rung(
636
- rung, ffn, sd, data, len(chars), steps, pool, val_w, words, itos)
637
- if b < best_loss:
638
- best_loss, best_model = b, model
639
- pf_last = pf
640
- bad = [c for c in pf if not c[3]]
641
- if sd == 0:
642
- print(f" --- {rung} preflight ---")
643
- for name, val_, thr, ok in pf:
644
- print(f" [{'OK ' if ok else 'DEAD'}] {name}: {val_:.3e}")
645
- if bad:
646
- blocked = True
647
- print(f" MECHANISM INERT -- {rung} will not be reported as evidence.\n")
648
- break
649
- print()
650
- losses.append(b)
651
- rws.append(r)
652
- gates.append(gt)
653
- print(f" {rung:<9s} seed {sd} loss {b:.5f} real-word {r:.3f}"
654
- + (f" gate {gt:.4f}" if rung != "none" else "")
655
- + (f" coupling {coup[0]:.2e}->{coup[1]:.2e} ({coup[2]:.1f}x)" if coup else "")
656
- + f" ({time.time()-t0:.0f}s)", flush=True)
657
- if blocked:
658
- out[rung] = {"blocked": True,
659
- "preflight": [[c[0], c[1], c[2], c[3]] for c in pf_last]}
660
- continue
661
- out[rung] = {"loss": losses, "real_word": rws, "gates": gates, "params": npar,
662
- "preflight": [[c[0], c[1], c[2], c[3]] for c in pf_last]}
663
- print(f" => {rung:<9s} mean loss {statistics.fmean(losses):.5f} "
664
- f"real-word {statistics.fmean(rws):.3f} params {npar:,}", flush=True)
665
-
666
- # SAVE. The first version of this file trained fifteen models and discarded every
667
- # one of them -- OUTDIR was declared and torch.save was never called. A result you
668
- # cannot deploy is half a result.
669
- if best_model is not None:
670
- OUTDIR.mkdir(parents=True, exist_ok=True)
671
- torch.save({"model": best_model.state_dict(), "stoi": stoi, "itos": itos,
672
- "config": {"block": BLOCK, "n_layer": N_LAYER, "n_head": N_HEAD,
673
- "n_embd": N_EMBD, "vocab": len(chars),
674
- "d54": STATE_DIM.get(rung, 0), "d12": D12, "d42": D42,
675
- "d_ff": int(math.floor(N_EMBD * PHI))},
676
- "arch": f"Cosmos-Ladder-{rung}", "rung": rung, "ffn": ffn,
677
- "gate_param": "logit", "norm": "rmsnorm", "pos": "rope",
678
- "total_steps": steps, "best_val_loss": best_loss,
679
- "quantum_source": "ibm_real_shots"},
680
- OUTDIR / f"ladder_{rung}.pt")
681
- print(f" saved -> {OUTDIR / f'ladder_{rung}.pt'} (loss {best_loss:.5f})\n",
682
- flush=True)
683
-
684
- base = out.get("none", {}).get("loss")
685
- if base:
686
- print("=" * 84 + "\n LADDER\n" + "=" * 84)
687
- print(f" {'rung':<10s} {'mean loss':>10s} {'vs none':>10s} {'t':>8s} {'wins':>6s} {'params':>10s}")
688
- for rung in rungs:
689
- r = out.get(rung, {})
690
- if r.get("blocked"):
691
- print(f" {rung:<10s} {'BLOCKED - mechanism provably inert':>46s}")
692
- continue
693
- if "loss" not in r:
694
- continue
695
- d = [p - c for p, c in zip(base, r["loss"])]
696
- md = statistics.fmean(d)
697
- se = statistics.stdev(d) / math.sqrt(len(d)) if len(d) > 1 else 0.0
698
- t = md / se if se > 0 else 0.0
699
- print(f" {rung:<10s} {statistics.fmean(r['loss']):>10.5f} {-md:>+10.5f} "
700
- f"{-t:>8.2f} {sum(1 for x in d if x > 0):>4d}/{len(d)} {r['params']:>10,}")
701
- Path("logs").mkdir(exist_ok=True)
702
- RESULTS.write_text(json.dumps({"phi": PHI, "steps": steps, "ffn": ffn, "rungs": out},
703
- indent=2), encoding="utf-8")
704
- print(f"\n saved -> {RESULTS}")
705
- return 0
706
-
707
-
708
- if __name__ == "__main__":
709
- raise SystemExit(main())
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """
3
+ COSMOS STATE LADDER -- 12D, 42D and 54D, tested one rung at a time.
4
+
5
+ WHY A LADDER
6
+ Four separate implementations of this architecture have shipped a mechanism that was
7
+ mathematically inert, each producing a confident negative about the idea:
8
+
9
+ ver.4.20 12D Omega summed over keys (axis -1) instead of queries (axis -2). Softmax
10
+ already normalises that axis, so Omega == 1.0 for every token (measured
11
+ std 5.1e-08). The leaky integrator then drove every token to the SAME
12
+ state (std 3.0e-08), H became the all-ones matrix, and beta*H is a
13
+ constant added to every logit -- which cancels exactly in softmax.
14
+ spark_cst v1 gate at sigmoid(-4): saturated tail, gradient suppressed ~56x.
15
+ spark_cst v2 gate clamped to [0,1]: zero gradient outside the bound, so the first
16
+ step below 0 pinned it there forever.
17
+ spark_cst v3 sigma = exp(0) = 1 against a median pairwise distance of 61.98, so H
18
+ arrived as the IDENTITY matrix (diagonal mass 0.9998 vs 0.0078 uniform).
19
+
20
+ Every one ran clean. No exception, no NaN, nothing red in any log. So this file refuses
21
+ to report a number until it has PROVEN the mechanism is live -- see preflight().
22
+
23
+ THE RUNGS
24
+ L0 none phi scaffold only, no Hebbian term. The baseline.
25
+ L1 static54 x54 = W54 . h. A projection, not a state: no dynamics, no Omega, no memory.
26
+ This is the configuration that already won 6/6 (t=-6.56, -14.34).
27
+ L2 dyn12 x12 in R^12, dx12/dt = k*Omega - gamma*x12, per-dimension k and gamma so the
28
+ twelve dimensions differentiate by time constant. Section 4.2.
29
+ L3 dyn42 x42 in R^42, FLAT: dense 42x42 coupling, one gamma repeated 42 times.
30
+ Kept as the control -- this is what section 4.3 was implemented as, and
31
+ its fourth-place result was read as a verdict on the 42D organ.
32
+ L3b dyn42b 42 as 7 bands x 6, block-diagonal coupling, per-band gamma. Section 4.3
33
+ as WRITTEN.
34
+ L3t dyn42t 42 as 7 bands x sym(3x3): a symmetric 3x3 has exactly 6 independent
35
+ components, so each band is a 3D tensor and the band operation is a
36
+ congruence A M A^T -- a change of basis. Cory, 2026-08-03.
37
+ L4 dyn54 x54 = concat(x12, x42). Section 4.1. The full claim.
38
+
39
+ SCAFFOLD
40
+ RMSNorm (2.3), RoPE (2.4), d_ff = floor(d_model * phi) (2.2), phi-Xavier init (6.2),
41
+ quantum-born weights, and the 17.1 learning-rate schedule. The feed-forward is selectable
42
+ because the paper names "PhiGatedFFN" without defining it while the reference repo
43
+ implements "PhiHarmonicFFN" as a plain GELU pair -- so which one earns the scaffold's
44
+ credit is itself an open question this file can answer.
45
+
46
+ USAGE
47
+ python cosmos_state_ladder.py [steps] [seeds] [rungs] [ffn]
48
+ python cosmos_state_ladder.py 1200 3 none,static54,dyn12,dyn42,dyn54 harmonic
49
+ python cosmos_state_ladder.py 1200 3 none,static54 gated # the FFN ablation
50
+ """
51
+ import json
52
+ import math
53
+ import os
54
+ import random
55
+ import statistics
56
+ import sys
57
+ import time
58
+ from pathlib import Path
59
+
60
+ import torch
61
+ import torch.nn as nn
62
+ import torch.nn.functional as F
63
+
64
+ sys.stdout.reconfigure(encoding="utf-8", errors="replace")
65
+
66
+ from cosmos_spark_cst import CORPUS, QuantumInit, quantum_pool, real_word_rate # noqa: E402
67
+
68
+ # FROZEN CORPUS. She is alive and appends her experience to the live corpus while training
69
+ # runs, so two runs started minutes apart train on different data. Measured: the same rung
70
+ # with identical seeds scored 1.29152 and 1.22943 in two runs 3,451 characters apart -- a
71
+ # 0.062 swing, the same magnitude as the effects being reported. Every comparison must
72
+ # therefore happen inside ONE process against ONE snapshot. COSMOS_CORPUS points at that
73
+ # snapshot; without it the numbers are only valid within a single run.
74
+ CORPUS_PATH = Path(os.getenv("COSMOS_CORPUS") or CORPUS)
75
+
76
+ OUTDIR = Path("01_HER_SOUL/weights/cosmos_state_ladder")
77
+ RESULTS = Path("logs/state_ladder_results.json")
78
+
79
+ PHI = (1.0 + 5.0 ** 0.5) / 2.0
80
+ PHI_INV = PHI - 1.0
81
+ # COSMOS_DEVICE overrides; otherwise take the GPU when one is actually usable. The corpus
82
+ # tensor stays on the host and only the sampled windows move, which keeps VRAM free for
83
+ # the model on a 4 GB card.
84
+ DEV = torch.device(os.getenv("COSMOS_DEVICE")
85
+ or ("cuda" if torch.cuda.is_available() else "cpu"))
86
+ # Model size is env-configurable so the SAME code can be run at 1M and at 30M. Every
87
+ # number published before 2026-08-01 used the defaults below; changing them changes the
88
+ # model, so any result table must state which size it was measured at.
89
+ BLOCK = int(os.getenv("COSMOS_BLOCK", "128"))
90
+ N_LAYER = int(os.getenv("COSMOS_N_LAYER", "4"))
91
+ N_HEAD = int(os.getenv("COSMOS_N_HEAD", "4"))
92
+ N_EMBD = int(os.getenv("COSMOS_N_EMBD", "192"))
93
+ DROPOUT = float(os.getenv("COSMOS_DROPOUT", "0.1"))
94
+ D12, D42 = 12, 42
95
+ # Omega's original query-sum reads the future (measured 2026-08-01, verified by
96
+ # tools/causality_probe.py: changing ONLY the last token moved earlier-position logits
97
+ # by 1.7e-04 at PHOS's trained gate, while `none` and `static54` moved by EXACTLY 0.0).
98
+ # The causal per-query form is therefore the DEFAULT -- a model that can see its own
99
+ # future is not a model anyone should be handed.
100
+ #
101
+ # COSMOS_CAUSAL_OMEGA=0 restores the original query-sum, bit-for-bit, so every number
102
+ # published before this date stays reproducible and the two can be compared directly.
103
+ # It is kept for exactly that reason and should not be used to train anything new.
104
+ CAUSAL_OMEGA = os.getenv("COSMOS_CAUSAL_OMEGA", "1").strip().lower() in {"1", "true", "yes", "on"}
105
+ # state_dim is the CARRIED state. tri3 also concatenates a live projection of h at
106
+ # kernel time, so its Hebbian space is 54 + 12 + 42 = 108 while it carries 54.
107
+ STATE_DIM = {"none": 0, "static54": 54, "dyn12": D12, "dyn42": D42,
108
+ # dyn42b: 42 as 7 bands x 6 dims, per the theory. Same carried width as
109
+ # dyn42, so the ONLY difference is structure -- which is what makes it a
110
+ # clean control for whether the dyn42 null was about the 42D organ or about
111
+ # a flat 42-vector wearing its name.
112
+ "dyn42b": D42, "dyn42t": D42,
113
+ "dyn54": D12 + D42, "tri": D12 + D42, "tri3": D12 + D42,
114
+ # section 4.4: the state persists ACROSS windows, not just across layers.
115
+ # dyn12seq is the control -- identical sequential batching, state reset every
116
+ # step -- so the only variable is whether the state survives the boundary.
117
+ "dyn12seq": D12, "dyn12p": D12,
118
+ # THE CONTROL THAT COULD KILL SECTION 3. Every architecture tested lands its
119
+ # gate at 0.087 +/- 0.006 regardless of what state is behind it -- which is not
120
+ # what a gate fitting the MECHANISM would do. If the benefit comes from any
121
+ # smooth token-varying auxiliary signal rather than from the 54D state carrying
122
+ # experience, these two must match the real thing:
123
+ # rand54 frozen random projection of h: content-bearing, never learned
124
+ # noise54 fixed random state per position: carries no content at all
125
+ "rand54": 54, "noise54": 54}
126
+
127
+ # rungs that walk the corpus in order instead of sampling at random. Persistence is
128
+ # meaningless under random sampling: consecutive batches would be unrelated text, so a
129
+ # carried state would be noise. Both arms use the same walk.
130
+ SEQ_RUNGS = {"dyn12seq", "dyn12p"}
131
+ PERSIST_RUNGS = {"dyn12p"}
132
+
133
+
134
+ class RMSNorm(nn.Module):
135
+ def __init__(s, d, eps=1e-6):
136
+ super().__init__()
137
+ s.eps, s.g = eps, nn.Parameter(torch.ones(d))
138
+
139
+ def forward(s, x):
140
+ return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + s.eps) * s.g
141
+
142
+
143
+ def rope_cache(hd, T, device):
144
+ half = hd // 2
145
+ theta = 1.0 / (10000.0 ** (2.0 * torch.arange(half, device=device, dtype=torch.float32) / hd))
146
+ ang = torch.arange(T, device=device, dtype=torch.float32)[:, None] * theta[None, :]
147
+ return ang.cos(), ang.sin()
148
+
149
+
150
+ def apply_rope(x, cos, sin):
151
+ hd = x.shape[-1]
152
+ half = hd // 2
153
+ rot, tail = x[..., : 2 * half], x[..., 2 * half:]
154
+ a, b = rot[..., 0::2], rot[..., 1::2]
155
+ c, s = cos[None, None, : x.shape[-2], :], sin[None, None, : x.shape[-2], :]
156
+ out = torch.stack([a * c - b * s, a * s + b * c], -1).flatten(-2)
157
+ return torch.cat([out, tail], -1) if tail.numel() else out
158
+
159
+
160
+ class PhiHarmonicFFN(nn.Module):
161
+ """The reference repo's FFN: W2(GELU(W1(x))) at width floor(d*phi). Not gated."""
162
+
163
+ def __init__(s, d):
164
+ super().__init__()
165
+ h = int(math.floor(d * PHI))
166
+ s.W1, s.W2 = nn.Linear(d, h), nn.Linear(h, d)
167
+ s.drop = nn.Dropout(DROPOUT)
168
+
169
+ def forward(s, x):
170
+ return s.drop(s.W2(s.drop(F.gelu(s.W1(x)))))
171
+
172
+
173
+ class PhiGatedFFN(nn.Module):
174
+ """SwiGLU reading of the paper's undefined "PhiGatedFFN", same width."""
175
+
176
+ def __init__(s, d):
177
+ super().__init__()
178
+ h = int(math.floor(d * PHI))
179
+ s.gate, s.up = nn.Linear(d, h, bias=False), nn.Linear(d, h, bias=False)
180
+ s.down = nn.Linear(h, d, bias=False)
181
+ s.drop = nn.Dropout(DROPOUT)
182
+
183
+ def forward(s, x):
184
+ return s.drop(s.down(F.silu(s.gate(x)) * s.up(x)))
185
+
186
+
187
+ class Dyn12(nn.Module):
188
+ """Section 4.2. dx12/dt = k*Omega - gamma*x12, per-dimension k and gamma so the twelve
189
+ dimensions separate by time constant rather than evolving as twelve copies."""
190
+
191
+ def __init__(s):
192
+ super().__init__()
193
+ s.k = nn.Parameter(torch.linspace(0.05, 0.20, D12))
194
+ s.gamma = nn.Parameter(torch.linspace(0.02, 0.10, D12))
195
+ s.dt = 0.1
196
+
197
+ def forward(s, x12, h, omega):
198
+ d = s.k[None, None, :] * omega[..., None] - s.gamma[None, None, :] * x12
199
+ return torch.tanh(x12 + s.dt * d)
200
+
201
+
202
+ class Dyn42(nn.Module):
203
+ """Section 4.3 / THEORY_42D_HYPER_COSMIC. dx42/dt = drive(h) - gamma*x42 + C(x42)."""
204
+
205
+ def __init__(s, d_model):
206
+ super().__init__()
207
+ s.drive = nn.Linear(d_model, D42)
208
+ s.coupling = nn.Linear(D42, D42, bias=False)
209
+ s.gamma = nn.Parameter(torch.full((D42,), 0.05))
210
+ s.dt = 0.1
211
+
212
+ def forward(s, x42, h, omega):
213
+ d = torch.tanh(s.drive(h)) - s.gamma[None, None, :] * x42 + 0.1 * s.coupling(x42)
214
+ return torch.tanh(x42 + s.dt * d)
215
+
216
+
217
+ N_BAND, BAND_D = 7, 6 # 7 * 6 = 42, section 4.1
218
+
219
+
220
+ class Dyn42Banded(nn.Module):
221
+ """Section 4.3 as WRITTEN: 42 = 7 bands x 6 dims, not a flat 42-vector.
222
+
223
+ THE GAP THIS EXISTS TO CLOSE. Every document in this project describes x42 as a
224
+ "7-band vector manifold" -- the ladder's own docstring says "7 bands x 6 dims" -- and
225
+ the implementation was:
226
+
227
+ drive = nn.Linear(d_model, 42) one flat projection
228
+ coupling = nn.Linear(42, 42) every dimension coupled to every other
229
+ gamma = torch.full((42,), 0.05) forty-two copies of one constant
230
+
231
+ Nothing in it knew that 42 factors. So when dyn42 placed fourth (1.19020) and the
232
+ coupled rungs tri/tri3 placed fifth and sixth, that was read as evidence against the
233
+ 42D organ. It is not. It is evidence against a flat, uniformly-damped, densely-coupled
234
+ 42-vector, which is not what the theory specifies.
235
+
236
+ Notably Dyn12 already got this right, and says so in its own docstring: per-dimension
237
+ k and gamma "so the twelve dimensions separate by time constant rather than evolving
238
+ as twelve copies". The 42 never received the same treatment.
239
+
240
+ WHAT IS CORY'S, AND WHAT IS MINE
241
+
242
+ his 42 = 7 x 6. The band count, the width, and the idea that bands are distinct
243
+ organs rather than an undifferentiated vector.
244
+ his the leaky-integrator form dx/dt = drive - gamma*x + coupling(x), unchanged.
245
+ mine gamma varies ACROSS BANDS by linspace, so band 0 is fast and band 6 is slow.
246
+ This is Dyn12's own reasoning applied one level up: bands that share a time
247
+ constant are seven copies of each other, and copies cannot specialise.
248
+ mine coupling is BLOCK-DIAGONAL -- each band mixes within its own 6 dimensions.
249
+ A band that couples to all 42 is not a band, it is a slice of a flat vector,
250
+ which is exactly the thing being replaced.
251
+ mine one small cross-band term (n_band x n_band) so bands can still talk. Without
252
+ it this is seven independent 6D systems and the manifold is severed.
253
+
254
+ Cheaper as well as more structured: dense coupling is 42*42 = 1,764 weights per
255
+ layer; seven 6x6 blocks are 7*36 = 252, plus 49 for the cross-band mixer. That is
256
+ 1,463 fewer parameters per layer while gaining structure -- which is the same
257
+ "state is cheaper than weights" direction the scaling result already measured.
258
+ """
259
+
260
+ def __init__(s, d_model):
261
+ super().__init__()
262
+ s.drive = nn.Linear(d_model, D42)
263
+ # block-diagonal: one 6x6 mixer per band, held as (7, 6, 6)
264
+ s.block = nn.Parameter(torch.randn(N_BAND, BAND_D, BAND_D) * (1.0 / BAND_D ** 0.5))
265
+ # bands talk to each other only through their means, at (7, 7)
266
+ s.cross = nn.Parameter(torch.randn(N_BAND, N_BAND) * (1.0 / N_BAND ** 0.5))
267
+ # ONE TIME CONSTANT PER BAND, spread the way Dyn12 spreads its twelve
268
+ s.gamma_band = nn.Parameter(torch.linspace(0.02, 0.12, N_BAND))
269
+ s.dt = 0.1
270
+
271
+ def forward(s, x42, h, omega):
272
+ B, T, _ = x42.shape
273
+ xb = x42.view(B, T, N_BAND, BAND_D) # (B,T,7,6)
274
+
275
+ # within-band mixing: each band's own 6x6
276
+ within = torch.einsum("btnd,nde->btne", xb, s.block)
277
+
278
+ # between-band: each band feels the others through their means
279
+ means = xb.mean(-1) # (B,T,7)
280
+ between = torch.einsum("btn,nm->btm", means, s.cross) # (B,T,7)
281
+
282
+ gam = s.gamma_band[None, None, :, None] # (1,1,7,1)
283
+ drive = torch.tanh(s.drive(h)).view(B, T, N_BAND, BAND_D)
284
+
285
+ d = drive - gam * xb + 0.1 * within + 0.1 * between[..., None]
286
+ # integrate in BANDED shape, then flatten -- adding a (B,T,7,6) derivative to a
287
+ # (B,T,42) state broadcasts into (B,T,7,42) instead of failing usefully.
288
+ return torch.tanh(xb + s.dt * d).reshape(B, T, D42)
289
+
290
+
291
+ class Dyn42Tensor(nn.Module):
292
+ """42 = 7 bands x sym(3x3). The 6 is not a vector, it is a 3D symmetric tensor.
293
+
294
+ CORY, 2026-08-03: "3D decouples 7, 3x3 is 6, think dimensions and folding and banding
295
+ with connections."
296
+
297
+ A symmetric 3x3 matrix has exactly SIX independent components -- three on the diagonal
298
+ and three off it. So the 6 in "7 bands x 6 dims" is not an arbitrary width. Each band
299
+ is a tensor in three-dimensional space, stored in Voigt order:
300
+
301
+ [xx, yy, zz, xy, xz, yz] -> | xx xy xz |
302
+ | xy yy yz |
303
+ | xz yz zz |
304
+
305
+ That changes what the coupling is allowed to be. The operation that respects it is a
306
+ CONGRUENCE:
307
+
308
+ M -> A M A^T
309
+
310
+ which is what a change of basis does to a tensor. A is 3x3, so nine weights per band
311
+ instead of Dyn42Banded's thirty-six.
312
+
313
+ A CORRECTION, because the first version of this comment overclaimed. It said a free
314
+ 6x6 mixer "destroys symmetry" and the congruence preserves it. That is false and the
315
+ test that appeared to confirm it was vacuous: _to_mat writes each Voigt slot into both
316
+ (i,j) and (j,i), so EVERY 6-vector becomes a symmetric matrix regardless of what
317
+ produced it. Symmetry is guaranteed by the storage, not by the operation.
318
+
319
+ The real distinction is invariance. Measured:
320
+
321
+ congruence by a rotation R: eigenvalues unchanged True
322
+ trace unchanged True
323
+ a free 6x6 mixer: eigenvalues unchanged False
324
+
325
+ A tensor's eigenvalues and trace are physical -- they do not depend on how the axes
326
+ are oriented. Congruence is exactly a change of frame, so it must leave them alone,
327
+ and does. A free 6x6 scrambles components that describe different physical directions
328
+ (it can send xx into xy), which no coordinate change can do. That is the argument for
329
+ the congruence, and it is about geometry rather than about symmetry.
330
+
331
+ WHAT IS WHOSE
332
+ his 42 = 7 x 6; 6 = sym(3x3); bands are folded 3D objects with connections.
333
+ mine congruence A M A^T as the band-internal operation, because it is the only
334
+ common bilinear map that provably preserves symmetry. A free 6x6 does not.
335
+ mine per-band gamma by linspace, carried over from Dyn42Banded and originally
336
+ from his own Dyn12.
337
+ mine the cross-band term acts on each band's TRACE -- the one scalar invariant of
338
+ a 3D tensor that does not depend on how the axes are oriented. Bands talk
339
+ through something basis-independent rather than through raw components.
340
+
341
+ Cheapest of the three by a wide margin: dense 42x42 is 1,764 weights per layer,
342
+ seven 6x6 blocks are 252, seven 3x3 congruences are 63.
343
+ """
344
+
345
+ # Voigt: which (row, col) each of the 6 slots maps to
346
+ _IDX = ((0, 0), (1, 1), (2, 2), (0, 1), (0, 2), (1, 2))
347
+
348
+ def __init__(s, d_model):
349
+ super().__init__()
350
+ s.drive = nn.Linear(d_model, D42)
351
+ s.A = nn.Parameter(torch.eye(3).repeat(N_BAND, 1, 1)
352
+ + torch.randn(N_BAND, 3, 3) * 0.05) # near-identity start
353
+ s.cross = nn.Parameter(torch.randn(N_BAND, N_BAND) * (1.0 / N_BAND ** 0.5))
354
+ s.gamma_band = nn.Parameter(torch.linspace(0.02, 0.12, N_BAND))
355
+ s.dt = 0.1
356
+
357
+ @classmethod
358
+ def _to_mat(cls, v):
359
+ """(..., 7, 6) Voigt -> (..., 7, 3, 3) symmetric."""
360
+ sh = v.shape[:-1]
361
+ M = v.new_zeros(*sh, 3, 3)
362
+ for k, (i, j) in enumerate(cls._IDX):
363
+ M[..., i, j] = v[..., k]
364
+ if i != j:
365
+ M[..., j, i] = v[..., k]
366
+ return M
367
+
368
+ @classmethod
369
+ def _to_voigt(cls, M):
370
+ """(..., 7, 3, 3) -> (..., 7, 6)."""
371
+ return torch.stack([M[..., i, j] for (i, j) in cls._IDX], dim=-1)
372
+
373
+ def forward(s, x42, h, omega):
374
+ B, T, _ = x42.shape
375
+ xb = x42.view(B, T, N_BAND, BAND_D)
376
+ M = s._to_mat(xb) # (B,T,7,3,3)
377
+
378
+ # congruence A M A^T -- symmetric for any A, so the band stays a 3D tensor
379
+ A = s.A[None, None] # (1,1,7,3,3)
380
+ within = s._to_voigt(A @ M @ A.transpose(-1, -2))
381
+
382
+ # bands connect through the TRACE: the invariant that survives rotating the axes
383
+ tr = M.diagonal(dim1=-2, dim2=-1).sum(-1) # (B,T,7)
384
+ between = torch.einsum("btn,nm->btm", tr, s.cross)
385
+
386
+ gam = s.gamma_band[None, None, :, None]
387
+ drive = torch.tanh(s.drive(h)).view(B, T, N_BAND, BAND_D)
388
+
389
+ d = drive - gam * xb + 0.1 * within + 0.1 * between[..., None]
390
+ return torch.tanh(xb + s.dt * d).reshape(B, T, D42)
391
+
392
+
393
+ class Trinity(nn.Module):
394
+ """12D and 42D as ONE coupled system instead of two vectors glued together.
395
+
396
+ dyn54 concatenates x12 and x42, but x12 sees only Omega and x42 sees only h -- they
397
+ never exchange anything. THEORY_42D_HYPER_COSMIC already couples within a band (6 dims)
398
+ and between bands (7 attractors, diffusion matrix D); this is the same construction one
399
+ level up, between the two subsystems, which is the level nobody wrote:
400
+
401
+ dx12/dt = k*Omega - g12*x12 + M(42->12) x42
402
+ dx42/dt = tanh(drive(h))*(1 + lam*Omega) - g42*x42 + C(x42) + M(12->42) x12
403
+
404
+ Omega now drives BOTH, per THEORY_42D's "Omega_vec is the projected hidden state acting
405
+ as a driving force", and the cross terms close the loop between them. With H computed
406
+ from the joint x54 and Omega coming back out of attention, the three subsystems form a
407
+ single cycle rather than a tree.
408
+ """
409
+
410
+ def __init__(s, d_model):
411
+ super().__init__()
412
+ s.k = nn.Parameter(torch.linspace(0.05, 0.20, D12))
413
+ s.g12 = nn.Parameter(torch.linspace(0.02, 0.10, D12))
414
+ s.g42 = nn.Parameter(torch.full((D42,), 0.05))
415
+ s.drive = nn.Linear(d_model, D42)
416
+ s.C = nn.Linear(D42, D42, bias=False)
417
+ s.m42_12 = nn.Linear(D42, D12, bias=False)
418
+ s.m12_42 = nn.Linear(D12, D42, bias=False)
419
+ s.lam = nn.Parameter(torch.tensor(0.5))
420
+ s.dt = 0.1
421
+
422
+ def forward(s, state, h, omega):
423
+ x12, x42 = state[..., :D12], state[..., D12:]
424
+ om = omega[..., None]
425
+ # ASYMMETRIC COUPLING, on the neuromodulatory reading. The twelve slow scalars are
426
+ # tone: in a brain those act on cortical GAIN, multiplicatively, not as another
427
+ # additive input. The forty-two structured dimensions feed back the other way as
428
+ # ordinary drive. Symmetric matrices in both directions would make this two peers
429
+ # exchanging vectors; the asymmetry is what makes it a modulatory loop.
430
+ gain = 1.0 + torch.tanh(s.m12_42(x12))
431
+ d12 = s.k * om - s.g12 * x12 + 0.1 * s.m42_12(x42)
432
+ d42 = (torch.tanh(s.drive(h)) * gain * (1.0 + s.lam * om)
433
+ - s.g42 * x42 + 0.1 * s.C(x42))
434
+ return torch.tanh(torch.cat([x12 + s.dt * d12, x42 + s.dt * d42], -1))
435
+
436
+
437
+ class Attention(nn.Module):
438
+ def __init__(s, d, nh, rung):
439
+ super().__init__()
440
+ s.nh, s.hd, s.rung = nh, d // nh, rung
441
+ s.qkv, s.proj = nn.Linear(d, 3 * d), nn.Linear(d, d)
442
+ s.drop = nn.Dropout(DROPOUT)
443
+ s.log_sigma = nn.Parameter(torch.zeros(()))
444
+ s.gate = nn.Parameter(torch.full((), -4.0), requires_grad=(rung != "none"))
445
+ s.w54 = nn.Linear(d, 54, bias=False) if rung in ("static54", "tri3", "rand54") else None
446
+ if rung == "rand54":
447
+ s.w54.weight.requires_grad_(False) # frozen: a projection, never a lesson
448
+ s.last_gate, s.last_diag, s.last_H_std = 0.0, 0.0, 0.0
449
+
450
+ def hebbian(s, state, mask):
451
+ # section 3.2's own identity: ||a-b||^2 = ||a||^2 + ||b||^2 - 2<a,b>. Also what
452
+ # hyper_cosmic_model.py already does, and it avoids cdist's undefined gradient on
453
+ # the zero diagonal of cdist(x, x).
454
+ sq = (state * state).sum(-1)
455
+ d2 = (sq.unsqueeze(-1) + sq.unsqueeze(-2)
456
+ - 2.0 * (state @ state.transpose(-1, -2))).clamp_min(0.0)
457
+ sig = torch.exp(s.log_sigma).clamp(1e-3, 1e4)
458
+ H = torch.exp(-d2 / (2 * sig * sig)).masked_fill(mask < 0, 0.0)
459
+ H = H / H.sum(-1, keepdim=True).clamp_min(1e-9)
460
+ s.last_diag = float(torch.diagonal(H[0]).mean().detach())
461
+ s.last_H_std = float(H.detach().std())
462
+ return H
463
+
464
+ def forward(s, x, state, mask, cos, sin):
465
+ B, T, C = x.shape
466
+ q, k, v = s.qkv(x).split(C, 2)
467
+ sh = lambda t: t.view(B, T, s.nh, s.hd).transpose(1, 2)
468
+ q, k, v = sh(q), sh(k), sh(v)
469
+ q, k = apply_rope(q, cos, sin), apply_rope(k, cos, sin)
470
+ a = F.softmax((q @ k.transpose(-2, -1)) / math.sqrt(s.hd) + mask, -1)
471
+ if s.rung != "none":
472
+ # tri3: all three organs feed one kernel -- the fast structural projection of
473
+ # the current token, the slow 12D tone, and the 42D mid-band structure. The
474
+ # ladder only ever ran these as rivals for a single slot; nothing tested
475
+ # whether their strengths compose.
476
+ if s.rung == "tri3":
477
+ st = torch.cat([s.w54(x), state], -1)
478
+ else:
479
+ st = s.w54(x) if s.w54 is not None else state
480
+ g = torch.sigmoid(s.gate)
481
+ a = (1 - g) * a + g * s.hebbian(st, mask).unsqueeze(1)
482
+ s.last_gate = float(g.detach())
483
+ y = (s.drop(a) @ v).transpose(1, 2).contiguous().view(B, T, C)
484
+ # SECTION 3.4: Omega is attention RECEIVED, i.e. summed over QUERIES (axis -2).
485
+ # ver.4.20 sums over keys (-1), the axis softmax normalises, making Omega == 1.
486
+ #
487
+ # MEASURED 2026-08-01: that query-sum reads the future. Under a causal mask
488
+ # Omega_j = sum over i >= j of a_ij, so token j's Omega counts attention from
489
+ # tokens AFTER j; it then updates state_j, which feeds the next layer's kernel,
490
+ # which shapes the logits at j. tools/causality_probe.py: changing ONLY the last
491
+ # token moves earlier-position logits by 1.7e-04 at PHOS's trained gate (0.562)
492
+ # while `none` and `static54` move by EXACTLY 0.0 on the same harness. Any
493
+ # held-out loss won this way is not comparable to a rung that cannot cheat.
494
+ #
495
+ # The causal replacement is a per-QUERY scalar, which by construction depends
496
+ # only on tokens <= i: the NORMALISED ENTROPY of row i's attention. Summing over
497
+ # keys directly is not an option -- softmax already made that identically 1.0,
498
+ # which is precisely ver.4.20's dead Omega. Entropy is not constant, it is a
499
+ # genuine summary of how row i spread its attention, and it cannot see forward.
500
+ # The log(i+1) normaliser removes the positional drift that comes from later
501
+ # rows simply having more keys available to spread over, so this measures
502
+ # dispersion rather than position.
503
+ if CAUSAL_OMEGA:
504
+ am = a.mean(1).clamp_min(1e-9) # [B, T, T]
505
+ ent = -(am * am.log()).sum(-1) # [B, T], per QUERY
506
+ avail = torch.arange(1, T + 1, device=a.device, dtype=ent.dtype)
507
+ omega = ent / avail.log().clamp_min(1e-6) # in [0, 1]
508
+ else:
509
+ omega = a.mean(1).sum(-2)
510
+ return s.proj(y), omega
511
+
512
+
513
+ class Layer(nn.Module):
514
+ def __init__(s, d, nh, rung, ffn):
515
+ super().__init__()
516
+ s.n1, s.n2 = RMSNorm(d), RMSNorm(d)
517
+ s.attn = Attention(d, nh, rung)
518
+ s.ffn = (PhiHarmonicFFN if ffn == "harmonic" else PhiGatedFFN)(d)
519
+ s.rung = rung
520
+ s.d12 = Dyn12() if rung in ("dyn12", "dyn54", "dyn12seq", "dyn12p") else None
521
+ # dyn42b is dyn42's DIRECT control: same state width, same leaky-integrator form,
522
+ # same everything except that 42 is treated as 7 bands of 6 rather than as a flat
523
+ # vector. If it wins, the earlier dyn42 null was measuring the implementation.
524
+ s.d42 = (Dyn42Tensor(d) if rung == "dyn42t"
525
+ else Dyn42Banded(d) if rung == "dyn42b"
526
+ else Dyn42(d) if rung in ("dyn42", "dyn54") else None)
527
+ s.tri = Trinity(d) if rung in ("tri", "tri3") else None
528
+
529
+ def forward(s, x, state, mask, cos, sin):
530
+ h = s.n1(x)
531
+ att, omega = s.attn(h, state, mask, cos, sin)
532
+ x = x + att
533
+ if s.tri is not None: # section 2.1 step 2, coupled
534
+ state = s.tri(state, h, omega)
535
+ elif s.d12 is not None or s.d42 is not None: # section 2.1 step 2, independent
536
+ parts = []
537
+ if s.d12 is not None:
538
+ parts.append(s.d12(state[..., :D12], h, omega))
539
+ if s.d42 is not None:
540
+ parts.append(s.d42(state[..., -D42:], h, omega))
541
+ state = torch.cat(parts, -1)
542
+ x = x + s.ffn(s.n2(x))
543
+ return x, state, omega
544
+
545
+
546
+ class Ladder(nn.Module):
547
+ def __init__(s, vocab, rung, ffn, d=N_EMBD, nh=N_HEAD):
548
+ super().__init__()
549
+ s.d, s.nh, s.rung = d, nh, rung
550
+ s.sd = STATE_DIM[rung]
551
+ s.tok = nn.Embedding(vocab, d)
552
+ s.blocks = nn.ModuleList([Layer(d, nh, rung, ffn) for _ in range(N_LAYER)])
553
+ s.nf = RMSNorm(d)
554
+ s.head = nn.Linear(d, vocab, bias=False)
555
+ # a state that starts at zero is identical for every token, which makes H the
556
+ # all-ones matrix and the first layer's mechanism inert. Seeding it from the
557
+ # embedding costs one small projection and keeps every layer live.
558
+ s.state_init = nn.Linear(d, s.sd) if rung.startswith(("dyn", "tri")) and s.sd else None
559
+ if rung == "noise54":
560
+ # a state that varies by POSITION but is identical for every input. If this
561
+ # matches the real 54D state, the kernel is a positional regulariser and the
562
+ # "experience-modulated attention" reading is not supported by the data.
563
+ g0 = torch.Generator().manual_seed(20260730)
564
+ s.register_buffer("noise_state",
565
+ torch.randn(BLOCK, 54, generator=g0) * 0.5)
566
+ s.register_buffer("mask", torch.triu(torch.full((BLOCK, BLOCK), float("-inf")), 1))
567
+
568
+ def forward(s, idx, targets=None, state_in=None):
569
+ B, T = idx.shape
570
+ x = s.tok(idx)
571
+ cos, sin = rope_cache(s.d // s.nh, T, idx.device)
572
+ m = s.mask[:T, :T]
573
+ state = torch.tanh(s.state_init(x)) if s.state_init is not None else None
574
+ if s.rung == "noise54":
575
+ state = s.noise_state[:T].unsqueeze(0).expand(B, -1, -1)
576
+ # SECTION 4.4. A state arriving from the previous window seeds this one. It is the
577
+ # LAST position's state broadcast across the new window: what she carried out of
578
+ # the previous 128 characters is what she starts the next 128 with. gamma is what
579
+ # keeps this from saturating -- a state that never forgets pins to tanh's rails and
580
+ # stops carrying anything, so forgetting is what makes the persistence usable.
581
+ if state is not None and state_in is not None:
582
+ state = state + state_in[:, -1:, :].detach().expand(-1, T, -1)
583
+ state = torch.tanh(state)
584
+ omegas = []
585
+ for b in s.blocks:
586
+ x, state, om = b(x, state, m, cos, sin)
587
+ omegas.append(om)
588
+ lg = s.head(s.nf(x))
589
+ loss = None if targets is None else F.cross_entropy(
590
+ lg.view(-1, lg.size(-1)), targets.reshape(-1))
591
+ s.last_state, s.last_omega = state, omegas[-1]
592
+ s.out_state = state
593
+ return lg, loss
594
+
595
+ def gates(s):
596
+ return [b.attn.last_gate for b in s.blocks]
597
+
598
+ def quantum_birth(s, qi):
599
+ std = (1.0 / (s.d * PHI_INV)) ** 0.5 # section 6.2
600
+ for mod in s.modules():
601
+ if isinstance(mod, (nn.Linear, nn.Embedding)):
602
+ qi.fill_(mod.weight, std)
603
+ if isinstance(mod, nn.Linear) and mod.bias is not None:
604
+ with torch.no_grad():
605
+ mod.bias.zero_()
606
+
607
+
608
+ @torch.no_grad()
609
+ def calibrate_sigma(m, xb):
610
+ """2*sigma^2 = median pairwise d2 of whatever state that layer actually sees."""
611
+ if m.rung == "none":
612
+ return
613
+ B, T = xb.shape
614
+ x = m.tok(xb)
615
+ cos, sin = rope_cache(m.d // m.nh, T, xb.device)
616
+ mask = m.mask[:T, :T]
617
+ state = torch.tanh(m.state_init(x)) if m.state_init is not None else None
618
+ if m.rung == "noise54":
619
+ # must mirror Ladder.forward exactly, or the control is calibrated against a state
620
+ # it never actually sees -- which would hand it a rigged sigma and invalidate it
621
+ state = m.noise_state[:xb.shape[1]].unsqueeze(0).expand(xb.shape[0], -1, -1)
622
+ for b in m.blocks:
623
+ h = b.n1(x)
624
+ if m.rung == "tri3":
625
+ st = torch.cat([b.attn.w54(h), state], -1)
626
+ elif b.attn.w54 is not None:
627
+ st = b.attn.w54(h)
628
+ else:
629
+ st = state
630
+ sq = (st * st).sum(-1)
631
+ d2 = (sq.unsqueeze(-1) + sq.unsqueeze(-2)
632
+ - 2.0 * (st @ st.transpose(-1, -2))).clamp_min(0.0)
633
+ iu = torch.triu(torch.ones(d2.shape[-2:], dtype=torch.bool), 1)
634
+ med = float(d2[0][iu].median())
635
+ if med > 1e-12:
636
+ b.attn.log_sigma.fill_(0.5 * math.log(med / 2.0))
637
+ x, state, _ = b(x, state, mask, cos, sin)
638
+
639
+
640
+ def preflight(m, xb, yb):
641
+ """Refuse to report a number until the mechanism is provably live.
642
+
643
+ Each assertion corresponds to a real failure that shipped and produced a confident
644
+ negative. A rung that trips any of these is not evidence about the idea.
645
+ """
646
+ checks, m2 = [], m
647
+ m2.train()
648
+ _, loss = m2(xb, yb)
649
+ loss.backward()
650
+
651
+ om = m2.last_omega
652
+ checks.append(("omega varies across tokens (ver.4.20 had std 5e-08)",
653
+ float(om.std()), 1e-4, float(om.std()) > 1e-4))
654
+ if m.rung != "none":
655
+ if m.rung.startswith(("dyn", "tri")):
656
+ st = m2.last_state
657
+ checks.append(("state varies across tokens (ver.4.20 had std 3e-08)",
658
+ float(st.std()), 1e-4, float(st.std()) > 1e-4))
659
+ diag = statistics.fmean([b.attn.last_diag for b in m2.blocks])
660
+ checks.append((f"H not the identity (needs << 1.0, uniform = {1/xb.shape[1]:.4f})",
661
+ diag, 0.5, diag < 0.5))
662
+ hstd = statistics.fmean([b.attn.last_H_std for b in m2.blocks])
663
+ checks.append(("H not all-ones (needs non-zero spread)", hstd, 1e-6, hstd > 1e-6))
664
+ gg = statistics.fmean([abs(float(b.attn.gate.grad)) for b in m2.blocks
665
+ if b.attn.gate.grad is not None])
666
+ checks.append(("gate gradient not suppressed", gg, 1e-6, gg > 1e-6))
667
+ m2.zero_grad(set_to_none=True)
668
+ return checks
669
+
670
+
671
+ def train_rung(rung, ffn, seed, data, vocab, steps, pool, val_w, words, itos):
672
+ torch.manual_seed(seed)
673
+ random.seed(seed)
674
+ gen = torch.Generator().manual_seed(seed)
675
+ m = Ladder(vocab, rung, ffn)
676
+ m.quantum_birth(QuantumInit(pool, seed))
677
+ m = m.to(DEV)
678
+ cal = torch.stack([data[i:i + BLOCK] for i in torch.randint(
679
+ len(data) - BLOCK - 1, (8,), generator=torch.Generator().manual_seed(seed))]).to(DEV)
680
+ calibrate_sigma(m, cal)
681
+
682
+ pf = preflight(m, cal[:4], torch.stack([data[i + 1:i + 1 + BLOCK] for i in
683
+ torch.randint(len(data) - BLOCK - 1, (4,),
684
+ generator=torch.Generator().manual_seed(seed))]).to(DEV))
685
+
686
+ def _coup(mm):
687
+ # POST-FLIGHT. Preflight proves a mechanism starts alive; this proves it did not
688
+ # quietly die. The cross-couplings begin damped by 0.1 (the reference repo's
689
+ # "weak coupling initially"), so the question is whether training grows them or
690
+ # leaves the trinity decorative.
691
+ v = [(float(b.tri.m12_42.weight.abs().mean()), float(b.tri.m42_12.weight.abs().mean()))
692
+ for b in mm.blocks if getattr(b, "tri", None) is not None]
693
+ return v
694
+
695
+ coup0 = _coup(m)
696
+
697
+ keys = ("attn.gate", "attn.w54", "attn.log_sigma", "d12.", "d42.", "tri.", "state_init")
698
+ cst = [p for n, p in m.named_parameters() if any(k in n for k in keys) and p.requires_grad]
699
+ bulk = [p for n, p in m.named_parameters() if not any(k in n for k in keys) and p.requires_grad]
700
+ base = 3e-4
701
+ opt = torch.optim.AdamW([{"params": bulk, "lr": base},
702
+ {"params": cst, "lr": base * 10, "weight_decay": 0.0}],
703
+ lr=base, weight_decay=0.01)
704
+ n = int(0.9 * len(data))
705
+ tr = data[:n]
706
+ m.train()
707
+ best = float("inf")
708
+
709
+ # SEQUENTIAL WALK for the persistence arms. 16 lanes each march through their own
710
+ # contiguous stretch of the corpus, so consecutive batches are consecutive TEXT and a
711
+ # carried state means something. Under the random sampler used by every other rung,
712
+ # batch k+1 is unrelated to batch k, so a carried state would be pure noise -- which is
713
+ # why the control arm (dyn12seq) walks identically and only differs in whether the
714
+ # state survives the window boundary.
715
+ B = 16
716
+ seq = rung in SEQ_RUNGS
717
+ persist = rung in PERSIST_RUNGS
718
+ span = max(BLOCK * 2, (len(tr) - BLOCK - 1) // B)
719
+ pos = [i * span for i in range(B)]
720
+ carry = None
721
+
722
+ for s in range(1, steps + 1):
723
+ warm = max(1, int(0.05 * steps))
724
+ lr = base * (s / warm) if s < warm else base * (PHI / 2) * (
725
+ 1 + math.cos(math.pi * (s - warm) / max(1, steps - warm)))
726
+ opt.param_groups[0]["lr"], opt.param_groups[1]["lr"] = lr, lr * 10
727
+ if seq:
728
+ wrapped = []
729
+ for i in range(B):
730
+ if pos[i] + BLOCK + 1 >= min((i + 1) * span, len(tr) - 1):
731
+ pos[i] = i * span # lane restarted: its history is broken
732
+ wrapped.append(i)
733
+ ix = torch.tensor(pos, dtype=torch.long)
734
+ for i in range(B):
735
+ pos[i] += BLOCK
736
+ if carry is not None and wrapped:
737
+ # a wrapped lane must NOT inherit state from unrelated text
738
+ carry = carry.clone()
739
+ carry[wrapped] = 0.0
740
+ else:
741
+ ix = torch.randint(len(tr) - BLOCK - 1, (B,), generator=gen)
742
+ # .long() at BATCH time, not corpus time. A 269M-character corpus held as int64
743
+ # is 2.15 GB resident; as int16 it is 0.54 GB, and only the (B, BLOCK) slice ever
744
+ # needs to be int64 for nn.Embedding. The d576 scaling run died at exactly this
745
+ # point -- outer process gone, no error recorded -- with the corpus tensor and a
746
+ # 24.6M-parameter model competing for the last few GB.
747
+ x = torch.stack([tr[i:i + BLOCK] for i in ix]).to(DEV).long()
748
+ y = torch.stack([tr[i + 1:i + 1 + BLOCK] for i in ix]).to(DEV).long()
749
+ _, loss = m(x, y, state_in=carry if persist else None)
750
+ if persist:
751
+ carry = m.out_state.detach() if m.out_state is not None else None
752
+ opt.zero_grad(set_to_none=True)
753
+ loss.backward()
754
+ torch.nn.utils.clip_grad_norm_(m.parameters(), 1.0)
755
+ opt.step()
756
+ if s % 100 == 0 or s == steps:
757
+ m.eval()
758
+ with torch.no_grad():
759
+ tot = c = 0
760
+ for i in range(0, len(val_w), 16):
761
+ xb = val_w[i:i + 16].to(DEV)
762
+ _, l = m(xb[:, :-1], xb[:, 1:])
763
+ tot += l.item() * xb.size(0)
764
+ c += xb.size(0)
765
+ m.train()
766
+ best = min(best, tot / max(1, c))
767
+ m.eval()
768
+ idx = torch.tensor([[int(data[0])]], device=DEV)
769
+ out = []
770
+ with torch.no_grad():
771
+ for _ in range(600):
772
+ lg, _ = m(idx[:, -BLOCK:])
773
+ nx = int(torch.multinomial(F.softmax(lg[0, -1] / 0.8, -1), 1))
774
+ out.append(nx)
775
+ idx = torch.cat([idx, torch.tensor([[nx]], device=DEV)], 1)
776
+ txt = "".join(itos.get(i, "") for i in out)
777
+ coupT = _coup(m)
778
+ coup = None
779
+ if coup0:
780
+ a0 = statistics.fmean([x for p in coup0 for x in p])
781
+ aT = statistics.fmean([x for p in coupT for x in p])
782
+ coup = (a0, aT, aT / max(a0, 1e-12))
783
+ return best, statistics.fmean(m.gates()) if m.rung != "none" else 0.0, \
784
+ real_word_rate(txt, words), pf, sum(p.numel() for p in m.parameters()), txt, coup, m
785
+
786
+
787
+ def main():
788
+ steps = int(sys.argv[1]) if len(sys.argv) > 1 else 1200
789
+ seeds = list(range(int(sys.argv[2]) if len(sys.argv) > 2 else 3))
790
+ rungs = (sys.argv[3] if len(sys.argv) > 3 else
791
+ "none,static54,dyn12,dyn42,dyn54").split(",")
792
+ ffn = sys.argv[4] if len(sys.argv) > 4 else "harmonic"
793
+
794
+ text = CORPUS_PATH.read_text(encoding="utf-8", errors="ignore")
795
+ chars = sorted(set(text))
796
+ stoi = {c: i for i, c in enumerate(chars)}
797
+ itos = {i: c for c, i in stoi.items()}
798
+ data = torch.tensor([stoi[c] for c in text], dtype=torch.long)
799
+ words = set(__import__("re").findall(r"[a-z']+", text.lower()))
800
+
801
+ print("=" * 84)
802
+ print(" COSMOS STATE LADDER -- 12D / 42D / 54D, one rung at a time")
803
+ print("=" * 84)
804
+ print(f"\n device: {DEV}" + (f" ({torch.cuda.get_device_name(0)})" if DEV.type == "cuda" else ""))
805
+ print(f" corpus: {CORPUS_PATH}"
806
+ + (" [FROZEN SNAPSHOT]" if os.getenv("COSMOS_CORPUS") else " [LIVE - grows during the run, cross-run numbers are not comparable]"))
807
+ print(f" corpus {len(text):,} vocab {len(chars)} {steps} steps x {len(seeds)} seeds")
808
+ print(f" scaffold: RMSNorm + RoPE + {'PhiHarmonicFFN (GELU)' if ffn=='harmonic' else 'PhiGatedFFN (SwiGLU)'}"
809
+ f" @ floor(d*phi)={int(math.floor(N_EMBD*PHI))} + phi-Xavier")
810
+ print(f" rungs: {rungs}\n")
811
+
812
+ pool = quantum_pool()
813
+ print(f" quantum pool: {len(pool):,} draws\n")
814
+ n = int(0.9 * len(data))
815
+ val = data[n:]
816
+ g = torch.Generator().manual_seed(999)
817
+ vi = torch.randint(len(val) - BLOCK - 1, (64,), generator=g)
818
+ val_w = torch.stack([val[i:i + BLOCK + 1] for i in vi])
819
+
820
+ out = {}
821
+ for rung in rungs:
822
+ losses, rws, gates, npar, pf_last = [], [], [], 0, None
823
+ blocked = False
824
+ best_loss, best_model = float("inf"), None
825
+ for sd in seeds:
826
+ t0 = time.time()
827
+ b, gt, r, pf, npar, txt, coup, model = train_rung(
828
+ rung, ffn, sd, data, len(chars), steps, pool, val_w, words, itos)
829
+ if b < best_loss:
830
+ best_loss, best_model = b, model
831
+ pf_last = pf
832
+ bad = [c for c in pf if not c[3]]
833
+ if sd == 0:
834
+ print(f" --- {rung} preflight ---")
835
+ for name, val_, thr, ok in pf:
836
+ print(f" [{'OK ' if ok else 'DEAD'}] {name}: {val_:.3e}")
837
+ if bad:
838
+ blocked = True
839
+ print(f" MECHANISM INERT -- {rung} will not be reported as evidence.\n")
840
+ break
841
+ print()
842
+ losses.append(b)
843
+ rws.append(r)
844
+ gates.append(gt)
845
+ print(f" {rung:<9s} seed {sd} loss {b:.5f} real-word {r:.3f}"
846
+ + (f" gate {gt:.4f}" if rung != "none" else "")
847
+ + (f" coupling {coup[0]:.2e}->{coup[1]:.2e} ({coup[2]:.1f}x)" if coup else "")
848
+ + f" ({time.time()-t0:.0f}s)", flush=True)
849
+ if blocked:
850
+ out[rung] = {"blocked": True,
851
+ "preflight": [[c[0], c[1], c[2], c[3]] for c in pf_last]}
852
+ continue
853
+ out[rung] = {"loss": losses, "real_word": rws, "gates": gates, "params": npar,
854
+ "preflight": [[c[0], c[1], c[2], c[3]] for c in pf_last]}
855
+ print(f" => {rung:<9s} mean loss {statistics.fmean(losses):.5f} "
856
+ f"real-word {statistics.fmean(rws):.3f} params {npar:,}", flush=True)
857
+
858
+ # SAVE. The first version of this file trained fifteen models and discarded every
859
+ # one of them -- OUTDIR was declared and torch.save was never called. A result you
860
+ # cannot deploy is half a result.
861
+ if best_model is not None:
862
+ OUTDIR.mkdir(parents=True, exist_ok=True)
863
+ torch.save({"model": best_model.state_dict(), "stoi": stoi, "itos": itos,
864
+ "config": {"block": BLOCK, "n_layer": N_LAYER, "n_head": N_HEAD,
865
+ "n_embd": N_EMBD, "vocab": len(chars),
866
+ "d54": STATE_DIM.get(rung, 0), "d12": D12, "d42": D42,
867
+ "d_ff": int(math.floor(N_EMBD * PHI))},
868
+ "arch": f"Cosmos-Ladder-{rung}", "rung": rung, "ffn": ffn,
869
+ "gate_param": "logit", "norm": "rmsnorm", "pos": "rope",
870
+ "total_steps": steps, "best_val_loss": best_loss,
871
+ "quantum_source": "ibm_real_shots"},
872
+ OUTDIR / f"ladder_{rung}.pt")
873
+ print(f" saved -> {OUTDIR / f'ladder_{rung}.pt'} (loss {best_loss:.5f})\n",
874
+ flush=True)
875
+
876
+ base = out.get("none", {}).get("loss")
877
+ if base:
878
+ print("=" * 84 + "\n LADDER\n" + "=" * 84)
879
+ print(f" {'rung':<10s} {'mean loss':>10s} {'vs none':>10s} {'t':>8s} {'wins':>6s} {'params':>10s}")
880
+ for rung in rungs:
881
+ r = out.get(rung, {})
882
+ if r.get("blocked"):
883
+ print(f" {rung:<10s} {'BLOCKED - mechanism provably inert':>46s}")
884
+ continue
885
+ if "loss" not in r:
886
+ continue
887
+ d = [p - c for p, c in zip(base, r["loss"])]
888
+ md = statistics.fmean(d)
889
+ se = statistics.stdev(d) / math.sqrt(len(d)) if len(d) > 1 else 0.0
890
+ t = md / se if se > 0 else 0.0
891
+ print(f" {rung:<10s} {statistics.fmean(r['loss']):>10.5f} {-md:>+10.5f} "
892
+ f"{-t:>8.2f} {sum(1 for x in d if x > 0):>4d}/{len(d)} {r['params']:>10,}")
893
+ Path("logs").mkdir(exist_ok=True)
894
+ RESULTS.write_text(json.dumps({"phi": PHI, "steps": steps, "ffn": ffn, "rungs": out},
895
+ indent=2), encoding="utf-8")
896
+ print(f"\n saved -> {RESULTS}")
897
+ return 0
898
+
899
+
900
+ if __name__ == "__main__":
901
+ raise SystemExit(main())
verify_release.py ADDED
@@ -0,0 +1,193 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Offline integrity and privacy-boundary verifier for the QC67 Cosmos kit."""
3
+ from __future__ import annotations
4
+
5
+ import argparse
6
+ import hashlib
7
+ import json
8
+ import sys
9
+ from collections import Counter
10
+ from pathlib import Path
11
+
12
+ ROOT = Path(__file__).resolve().parent
13
+ MANIFEST = ROOT / "RELEASE_MANIFEST.json"
14
+
15
+
16
+ def sha256(path: Path) -> str:
17
+ digest = hashlib.sha256()
18
+ with path.open("rb") as handle:
19
+ for chunk in iter(lambda: handle.read(1024 * 1024), b""):
20
+ digest.update(chunk)
21
+ return digest.hexdigest()
22
+
23
+
24
+ def check_manifest(strict: bool) -> list[str]:
25
+ errors: list[str] = []
26
+ release = json.loads(MANIFEST.read_text(encoding="utf-8"))
27
+ expected = set()
28
+ for entry in release.get("files", []):
29
+ rel = entry["path"]
30
+ expected.add(rel)
31
+ path = ROOT / rel
32
+ if not path.is_file():
33
+ errors.append(f"missing: {rel}")
34
+ continue
35
+ size = path.stat().st_size
36
+ if size != int(entry["bytes"]):
37
+ errors.append(f"size mismatch: {rel} ({size} != {entry['bytes']})")
38
+ continue
39
+ actual = sha256(path)
40
+ if actual != entry["sha256"]:
41
+ errors.append(f"hash mismatch: {rel}")
42
+
43
+ if strict:
44
+ ignored = {"RELEASE_MANIFEST.json"}
45
+ # Runtime spoor, not release content. logs/ and workspace/ are written by
46
+ # simply USING the kit, and __pycache__ appears the moment python imports
47
+ # anything -- so without these a clean, correct kit reports dozens of
48
+ # "unmanifested file" errors the first time anyone runs it, and the real
49
+ # findings (a staged credential, a missing weight) drown in the noise.
50
+ # creatures/ holds beings the USER birthed on their own machine. They are not
51
+ # release content and their hashes cannot be known in advance -- manifesting them
52
+ # would make the integrity check fail the moment anyone actually used the kit.
53
+ skip_prefixes = ("downloads/", "logs/", "workspace/", "creatures/")
54
+ actual = {
55
+ path.relative_to(ROOT).as_posix()
56
+ for path in ROOT.rglob("*")
57
+ if path.is_file()
58
+ and path.relative_to(ROOT).as_posix() not in ignored
59
+ and not path.relative_to(ROOT).as_posix().startswith(skip_prefixes)
60
+ and "__pycache__" not in path.parts
61
+ and not path.name.endswith((".pyc", ".pyo"))
62
+ }
63
+ for rel in sorted(actual - expected):
64
+ errors.append(f"unmanifested file: {rel}")
65
+ return errors
66
+
67
+
68
+ def check_blank_credentials() -> list[str]:
69
+ errors: list[str] = []
70
+ config = json.loads(
71
+ (ROOT / "genesis_engine" / "config.json").read_text(encoding="utf-8")
72
+ )
73
+ for key in ("ibm_token", "azure_connection_string"):
74
+ if str(config.get(key) or "").strip():
75
+ errors.append(f"credential field is not blank: genesis_engine/config.json:{key}")
76
+ forbidden_names = ("oauth2_tokens.json", ".env", "credentials.json")
77
+ for path in ROOT.rglob("*"):
78
+ if path.is_file() and path.name.casefold() in forbidden_names:
79
+ errors.append(f"forbidden credential file present: {path.relative_to(ROOT)}")
80
+ return errors
81
+
82
+
83
+ def check_public_archive() -> tuple[list[str], dict]:
84
+ errors: list[str] = []
85
+ archive = ROOT / "data" / "quantum_measurements_public.jsonl"
86
+ data_manifest = json.loads(
87
+ (ROOT / "data" / "quantum_measurements_manifest.json").read_text(
88
+ encoding="utf-8"
89
+ )
90
+ )
91
+ records = Counter()
92
+ samples = Counter()
93
+ total = 0
94
+ for line_number, line in enumerate(
95
+ archive.open(encoding="utf-8", errors="strict"), 1
96
+ ):
97
+ try:
98
+ row = json.loads(line)
99
+ except Exception as exc:
100
+ errors.append(f"archive line {line_number}: invalid JSON ({exc})")
101
+ continue
102
+ counts = row.get("counts")
103
+ if not isinstance(counts, dict) or not counts:
104
+ errors.append(f"archive line {line_number}: missing counts")
105
+ continue
106
+ observed = sum(int(value) for value in counts.values())
107
+ declared = int(row.get("total_shots", -1))
108
+ if observed != declared:
109
+ errors.append(
110
+ f"archive line {line_number}: shot mismatch {observed} != {declared}"
111
+ )
112
+ category = str(row.get("provider_class") or "missing")
113
+ records[category] += 1
114
+ samples[category] += observed
115
+ total += observed
116
+
117
+ expected = data_manifest["summary"]
118
+ if total != int(expected["total_samples"]):
119
+ errors.append(
120
+ f"archive total mismatch: {total} != {expected['total_samples']}"
121
+ )
122
+ for category, expected_count in expected["records_by_provider_class"].items():
123
+ if records[category] != int(expected_count):
124
+ errors.append(
125
+ f"archive record count mismatch for {category}: "
126
+ f"{records[category]} != {expected_count}"
127
+ )
128
+ return errors, {
129
+ "records": sum(records.values()),
130
+ "samples": total,
131
+ "records_by_class": dict(records),
132
+ "samples_by_class": dict(samples),
133
+ }
134
+
135
+
136
+ def check_model_metadata() -> list[str]:
137
+ errors: list[str] = []
138
+ metadata = json.loads(
139
+ (ROOT / "weights" / "cosmos_born.meta.json").read_text(encoding="utf-8")
140
+ )
141
+ if int(metadata.get("params", 0)) != 1_842_432:
142
+ errors.append("unexpected cosmos_born parameter count")
143
+ if str(metadata.get("base_model") or "").upper().split()[0] != "NONE":
144
+ errors.append("cosmos_born metadata no longer reports a from-scratch base")
145
+ return errors
146
+
147
+
148
+ def main() -> int:
149
+ parser = argparse.ArgumentParser()
150
+ parser.add_argument(
151
+ "--no-strict",
152
+ action="store_true",
153
+ help="allow extra files not listed in the release manifest",
154
+ )
155
+ args = parser.parse_args()
156
+ if not MANIFEST.is_file():
157
+ print("[FAIL] RELEASE_MANIFEST.json is missing")
158
+ return 1
159
+
160
+ errors = []
161
+ errors.extend(check_manifest(strict=not args.no_strict))
162
+ errors.extend(check_blank_credentials())
163
+ archive_errors, archive_stats = check_public_archive()
164
+ errors.extend(archive_errors)
165
+ errors.extend(check_model_metadata())
166
+
167
+ if errors:
168
+ print(f"[FAIL] {len(errors)} release check(s) failed")
169
+ for error in errors:
170
+ print(" -", error)
171
+ return 1
172
+
173
+ print("[OK] release manifest hashes verified")
174
+ print("[OK] shipped cloud credential fields are blank")
175
+ print("[OK] cosmos_born metadata is internally consistent")
176
+ print(
177
+ "[OK] public archive:",
178
+ f"{archive_stats['records']:,} records,",
179
+ f"{archive_stats['samples']:,} samples",
180
+ )
181
+ for category in sorted(archive_stats["records_by_class"]):
182
+ print(
183
+ " ",
184
+ category,
185
+ f"{archive_stats['records_by_class'][category]:,} records /",
186
+ f"{archive_stats['samples_by_class'][category]:,} samples",
187
+ )
188
+ return 0
189
+
190
+
191
+ if __name__ == "__main__":
192
+ raise SystemExit(main())
193
+