Release: HF export, model card, charts and film

.gitattributes

@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+comparison_index.png filter=lfs diff=lfs merge=lfs -text
+index_curve.png filter=lfs diff=lfs merge=lfs -text
+loss_curve.png filter=lfs diff=lfs merge=lfs -text
+tinctura-v1.mp4 filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,282 @@
+---
+license: apache-2.0
+language:
+- en
+pipeline_tag: text-generation
+library_name: transformers
+tags:
+- causal-lm
+- language-model
+- base-model
+- pretrained-from-scratch
+- small-language-model
+datasets:
+- HuggingFaceTB/smollm-corpus
+- mlfoundations/dclm-baseline-1.0
+- HuggingFaceTB/finemath
+- nvidia/OpenMathInstruct-2
+- HuggingFaceTB/smoltalk
+---
+
+# tinctura-v1
+
+A 96.2M parameter decoder-only language model pretrained from scratch on 75B tokens. It is the [cagliostro-v3](https://huggingface.co/bench-labs/cagliostro-v3) recipe at two thirds of the size: the same architecture, data, schedule and token count, with 18 layers instead of 30.
+
+On the [Open SLM Leaderboard](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard) Index it scores **20.77**. Among models under 100M parameters that places it second, 0.30 behind Rose-1.5-Medium at 21.07.
+
+The training run is **complete**. 75.00B tokens, 762,939 steps, learning rate decayed to zero.
+
+<video controls src="https://huggingface.co/bench-labs/tinctura-v1/resolve/main/tinctura-v1.mp4" width="100%"></video>
+
+## Results
+
+Zero-shot, measured with lm-evaluation-harness and the leaderboard's own ArithMark-3 script, on the exported float32 weights in this repository.
+
+| Benchmark | Metric | Score |
+|---|---|---:|
+| HellaSwag | acc_norm | 38.01 |
+| ARC-Easy | acc_norm | 48.15 |
+| ARC-Challenge | acc_norm | 26.02 |
+| PIQA | acc_norm | 65.67 |
+| ArithMark-3 | acc_norm | 37.70 |
+| **Open SLM Index** | | **20.77** |
+
+The Index is the leaderboard's own formula, `(N(HellaSwag,25) + N(CombinedARC,25) + N(PIQA,50) + 0.65*N(ArithMark,25)) / 3.65` where `N(v,c) = 100(v-c)/(100-c)` and CombinedARC is the mean of ARC-Easy and ARC-Challenge. The final weights were downloaded back from this repository and evaluated again, and they reproduced the same scores exactly.
+
+The top of the sub-100M field, using the leaderboard's published figures. tinctura-v1 is scored with the same formula but is not yet listed on the board.
+
+| Model | Params | Index |
+|---|---:|---:|
+| Rose-1.5-Medium | 98.2M | 21.07 |
+| **tinctura-v1** | **96.2M** | **20.77** |
+| Surjo-100m | 97.7M | 18.86 |
+| cRia-LM-75M | 75.7M | 18.06 |
+| Rose-Medium | 97.8M | 17.73 |
+| Surjo-50M | 53.8M | 16.40 |
+
+![Index against the sub-100M field](comparison_index.png)
+
+![Per-benchmark comparison with Rose-1.5-Medium](comparison_tasks.png)
+
+Against the leader the result is split. tinctura-v1 is ahead on ARC-Easy and PIQA, level on HellaSwag, and behind on ARC-Challenge and ArithMark-3. ArithMark-3 is most of the gap: 3.0 points there cost about 0.7 Index at its 0.65 weight, more than the whole 0.30 margin. Rose-1.5-Medium was trained on roughly 100B tokens according to its card, against 75B here.
+
+## What a third fewer parameters costs
+
+Because tinctura-v1 follows cagliostro-v3 step for step, the two runs can be compared at identical points in training. Several of these comparisons were run side by side on the same machine with the same evaluation code, using checkpoints from both repositories.
+
+![Index during training](index_curve.png)
+
+Through the stable phase the smaller model trailed by about 1.7 Index over the first 20B tokens and by about 2.9 after that. Single evaluations carry roughly one point of noise, so individual gaps ranged from 0.5 to 3.9. Just before the cooldown, at step 646,000, the paired evaluation read 19.34 against 21.96.
+
+![What the cooldown added](cooldown_gain.png)
+
+The cooldown is where the two models part. cagliostro-v3 gained 4.6 Index over the last 15% of training. tinctura-v1 gained 1.4. The shortfall came in two places. From step 646,000 to 706,000 cagliostro-v3 added 1.95 and tinctura-v1 only 0.21. From 706,000 to 736,000 the two moved together, 1.32 and 1.23. Over the final stretch to the end cagliostro-v3 added another 1.32 while tinctura-v1 held flat. On ARC-Challenge and ArithMark-3 the smaller model finished the cooldown slightly lower than it started it.
+
+![Training loss](loss_curve.png)
+
+After the first few billion tokens the training loss gap to cagliostro-v3 settled between 0.07 and 0.10 and stayed there through the cooldown. The sharp drop at 63.75B is the data mixture changing, not the model improving.
+
+## The cooldown was run twice
+
+The first attempt at the cooldown hit a bug in the data loader. When training switches to the cooldown mixture, each process restarts its data-loading threads. Threads blocked on a full queue could miss the stop signal and keep producing batches from the old mixture next to the new ones, so each process trained on a blend. A standalone test of the loader leaked old-mixture batches in 9 of 10 trials, 20 to 44% of batches after the switch. The loss told on it: at the switch the loss fell by 0.26 where cagliostro-v3's fell by 0.49.
+
+The loader was fixed so that every restart gets its own queue and old threads exit on their own, and the fixed version leaked nothing in the same test. The cooldown was then rerun from the last checkpoint before it, step 644,000. The rerun's loss fell by 0.45 at the switch and tracked cagliostro-v3 as expected. The weights in this repository come from the rerun.
+
+The fix did not change the benchmarks in a measurable way. At step 706,000 the first attempt scored 19.68 and the rerun 19.55, which is within evaluation noise. The data-mixing bug was real, but it was not what held the cooldown gain down.
+
+## Model details
+
+| Field | Value |
+|---|---:|
+| Parameters | 96,200,064 |
+| Non-embedding parameters | 78.2% |
+| Layers | 18 |
+| Hidden size | 640 |
+| Intermediate size | 1,536 |
+| Attention heads | 10 |
+| Key/value heads | 5 |
+| Attention | Grouped query attention with cross-head subspace attenuation |
+| Activation | SwiGLU |
+| Normalization | RMSNorm, eps 1e-6 |
+| Positional encoding | RoPE, theta 100,000 |
+| Context length | 2,048 |
+| Vocabulary | 32,768 BPE |
+| Embeddings | Tied input and output |
+| Logit cap | 15.0 |
+| Weights | float32 safetensors |
+
+The architecture is defined in this repository and is the same code as cagliostro-v3. `trust_remote_code=True` is required because `CagliostroForCausalLM` is not part of transformers.
+
+## Training data
+
+The same mixtures and shards as cagliostro-v3. The first mixture covers the stable phase, and the second takes over when the cooldown begins at 85% of the run.
+
+| Source | Stable phase | Cooldown |
+|---|---:|---:|
+| FineWeb-Edu (deduplicated) | 43.7% | 37.0% |
+| DCLM-Baseline | 28.3% | 5.0% |
+| Cosmopedia v2 | 16.0% | 25.0% |
+| FineMath 3+ | 5.0% | 15.0% |
+| OpenMathInstruct-2 | 3.0% | 13.0% |
+| InfiWebMath 3+ | 2.0% | 0.0% |
+| SmolTalk | 2.0% | 5.0% |
+
+Mathematics rises from 10% to 28% during the cooldown.
+
+## Training setup
+
+| Field | Value |
+|---|---:|
+| Optimizer | AdamW, betas 0.9 and 0.95, weight decay 0.01 |
+| Peak learning rate | 3e-3 |
+| Schedule | Warmup-stable-decay |
+| Warmup | 2,000 steps |
+| Stable phase | steps 2,000 to 648,498 at peak learning rate |
+| Cooldown | 114,441 steps, cosine to zero |
+| Tokens per step | 98,304 |
+| Total steps | 762,939 |
+| Precision | bfloat16 with float32 master weights |
+| Hardware | eight RTX 5090, data parallel, 6 sequences of 2,048 tokens per GPU |
+| Throughput | about 1.05M tokens per second |
+| Wall clock | about 21 hours for the published run, plus 2 hours on the discarded first cooldown |
+
+## Usage
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "bench-labs/tinctura-v1"
+tok = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, dtype=torch.float32)
+
+ids = tok("The capital of France is", return_tensors="pt")
+out = model.generate(**ids, max_new_tokens=32, do_sample=False)
+print(tok.decode(out[0], skip_special_tokens=True))
+```
+
+This is a base model with no instruction tuning and no chat template. It completes text.
+
+## Samples
+
+Twelve fixed prompts, one sample each, unedited. Sampling at temperature 0.7, top-p 0.9, repetition penalty 1.1, 48 new tokens, seed 0 before every prompt. These are the same prompts used for cagliostro-v3. The film shows the first sentence of three of them. The misses are left in.
+
+<details>
+<summary>All 12 samples</summary>
+
+**1.** `The water cycle begins when`
+
+```text
+The water cycle begins when the oceans absorb more carbon dioxide than they release. As a result, they are rapidly cooling and ocean currents move more quickly.
+“This is a pretty good analogy for how climate change might happen,” said Professor Charles G. Hines,
+```
+
+**2.** `Photosynthesis is the process by which`
+
+```text
+Photosynthesis is the process by which plants convert light energy into chemical energy. The reaction produces oxygen and water as a by-product.
+What is photosynthesis?
+Photosynthesis is a type of photosynthetic process that occurs in plants. Photosynthesis takes place in the roots
+```
+
+**3.** `To solve 47 + 68, first`
+
+```text
+To solve 47 + 68, first add the digits together:
+\[ 4 + 7 + 8 = 11 + 8 = 19 \]
+
+2. Now subtract the sum from 5 (the number we want to find) from
+```
+
+**4.** `The Roman Empire fell because`
+
+```text
+The Roman Empire fell because of the threat of the Black Death. The Romans were not interested in fighting off a disease, but rather had an idealism for the destruction of their enemies and the preservation of civilization. This idea was reflected in the policies of Rome, which
+```
+
+**5.** `A prime number is a number that`
+
+```text
+A prime number is a number that has exactly two distinct positive divisors: 1 and itself.
+
+The sum of the digits of an integer is equal to the sum of its digits multiplied by the number of digits in its base-$d$ representation, where
+```
+
+**6.** `The three states of matter are`
+
+```text
+The three states of matter are also called the solids, liquids and gases.
+- Atoms are found in a single layer called an atom.
+- The number of atoms is inversely proportional to the energy.
+- A gas has molecules that have very small amounts
+```
+
+**7.** `Gravity is the force that`
+
+```text
+Gravity is the force that holds everything in place, while gravity is what pulls everything towards the center of the universe.
+
+Now, let's talk about energy. Energy is a form of power, and it can be transformed from one form to another through various processes
+```
+
+**8.** `In 1969, astronauts`
+
+```text
+In 1969, astronauts from the International Space Station (ISS) were to use their space suits for a brief period of time before being sent back to Earth. The astronauts also wore a variety of spacesuits and space suits, which they used to protect them from extreme
+```
+
+**9.** `The heart pumps blood through`
+
+```text
+The heart pumps blood through the body and then the arteries deliver the oxygen to the tissues.
+- Blood is made up of white and red blood cells, which carry oxygen from the lungs to all parts of the body.
+- The red blood cells are also known
+```
+
+**10.** `To find the area of a rectangle you`
+
+```text
+To find the area of a rectangle you need to know two things: the length and width.
+
+The area of a rectangle is given by the formula A = L * W. In this case, we know that L = 10 and W = 12.
+```
+
+**11.** `Volcanoes form when`
+
+```text
+Volcanoes form when the surface of a volcano cools and becomes very hot. The resulting heat causes a lot of steam to rise up through the volcano's vents, which are areas where magma is released. When this steam comes out of the vents
+```
+
+**12.** `The difference between weather and climate is`
+
+```text
+The difference between weather and climate is more important than what the term suggests. The temperature in a given location can be found by measuring how much it gets warmer or colder than the average of that region. If you have a map of your local area, you can look at its
+```
+
+</details>
+
+## Reproducing the evaluation
+
+```
+pip install lm-eval
+python -m lm_eval --model hf \
+  --model_args pretrained=bench-labs/tinctura-v1,dtype=float32,trust_remote_code=True \
+  --tasks hellaswag,arc_easy,arc_challenge,piqa \
+  --num_fewshot 0 --batch_size 8 --device cuda:0
+```
+
+ArithMark-3 uses the script linked from the leaderboard, pointed at the same model id. Evaluate in float32. The exported weights agree with the training checkpoint exactly.
+
+## Provenance
+
+This repository holds the training history. A checkpoint was pushed about every 30 minutes, and any of them can be retrieved by revision. The history starts at step 22,000, because pushes during the first half hour failed on a repository permission error.
+
+The commits for steps 662,000, 680,000, 700,000 and 718,000, pushed between 23:08 UTC on 23 September and 00:39 UTC on 24 September 2026, belong to the first cooldown attempt described above and were not used. The published line continues from step 644,000 with the commits from step 656,000 onward.
+
+## Limitations
+
+English only. 2,048 token context. No instruction tuning, no safety tuning, no RLHF. At 96M parameters it confabulates freely, as the samples show, and should not be relied on for factual questions. The mathematics ability measured by ArithMark is arithmetic and short symbolic work, not general mathematical reasoning.
+
+## License
+
+Apache-2.0. The training data is drawn from FineWeb-Edu and FineMath (ODC-By), DCLM-Baseline and OpenMathInstruct-2 (CC-BY-4.0), Cosmopedia v2 and SmolTalk (Apache-2.0).

config.json

@@ -0,0 +1,30 @@
+{
+  "architectures": [
+    "CagliostroForCausalLM"
+  ],
+  "bos_token_id": 0,
+  "canon": false,
+  "dtype": "float32",
+  "eos_token_id": 0,
+  "hidden_size": 640,
+  "intermediate_size": 1536,
+  "layer_repeat": 1,
+  "logit_cap": 15.0,
+  "max_position_embeddings": 2048,
+  "mlp_type": "swiglu",
+  "model_type": "cagliostro",
+  "num_attention_heads": 10,
+  "num_hidden_layers": 18,
+  "num_key_value_heads": 5,
+  "rms_norm_eps": 1e-06,
+  "rope_theta": 100000.0,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.17.0",
+  "value_residual": false,
+  "vocab_size": 32768,
+  "xsa": true,
+  "auto_map": {
+    "AutoConfig": "configuration_cagliostro.CagliostroConfig",
+    "AutoModelForCausalLM": "modeling_cagliostro.CagliostroForCausalLM"
+  }
+}

configuration_cagliostro.py

@@ -0,0 +1,43 @@
+from transformers import PretrainedConfig
+
+
+class CagliostroConfig(PretrainedConfig):
+    model_type = "cagliostro"
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+    def __init__(
+        self,
+        vocab_size=32768,
+        hidden_size=640,
+        num_hidden_layers=30,
+        num_attention_heads=10,
+        num_key_value_heads=5,
+        intermediate_size=1600,
+        max_position_embeddings=4096,
+        rope_theta=100000.0,
+        rms_norm_eps=1e-6,
+        tie_word_embeddings=True,
+        mlp_type="swiglu",
+        xsa=False,
+        canon=False,
+        value_residual=False,
+        layer_repeat=1,
+        logit_cap=0.0,
+        **kwargs,
+    ):
+        self.vocab_size = vocab_size
+        self.hidden_size = hidden_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.num_key_value_heads = num_key_value_heads
+        self.intermediate_size = intermediate_size
+        self.max_position_embeddings = max_position_embeddings
+        self.rope_theta = rope_theta
+        self.rms_norm_eps = rms_norm_eps
+        self.mlp_type = mlp_type
+        self.xsa = xsa
+        self.canon = canon
+        self.value_residual = value_residual
+        self.layer_repeat = layer_repeat
+        self.logit_cap = logit_cap
+        super().__init__(tie_word_embeddings=tie_word_embeddings, **kwargs)

generation_config.json

@@ -0,0 +1,8 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 0,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "transformers_version": "5.17.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e6dfb66d0c7dc41fefb5ab0f7d4846cb0d84b4eab2f4413ff9991d899695e52b
+size 384820248

modeling_cagliostro.py

@@ -0,0 +1,270 @@
+import math
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from transformers import PreTrainedModel
+from transformers.generation import GenerationMixin
+from transformers.modeling_outputs import CausalLMOutputWithPast
+
+try:
+    from .configuration_cagliostro import CagliostroConfig
+except ImportError:
+    from configuration_cagliostro import CagliostroConfig
+
+
+class RMSNorm(nn.Module):
+    def __init__(self, dim, eps=1e-6):
+        super().__init__()
+        self.eps = eps
+        self.weight = nn.Parameter(torch.ones(dim))
+
+    def forward(self, x):
+        dtype = x.dtype
+        x = x.float()
+        x = x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)
+        return x.to(dtype) * self.weight
+
+
+def rope_freqs(seq_len, head_dim, theta, device=None):
+    inv_freq = 1.0 / (theta ** (torch.arange(0, head_dim, 2, device=device).float() / head_dim))
+    t = torch.arange(seq_len, device=device).float()
+    freqs = torch.outer(t, inv_freq)
+    return torch.cat([freqs, freqs], dim=-1)
+
+
+def rotate_half(x):
+    x1, x2 = x.chunk(2, dim=-1)
+    return torch.cat([-x2, x1], dim=-1)
+
+
+def apply_rope(x, cos, sin):
+    return x * cos + rotate_half(x) * sin
+
+
+class Attention(nn.Module):
+    def __init__(self, dim, n_heads, n_kv_heads, head_dim, norm_eps=1e-6,
+                 xsa=False, value_residual=False):
+        super().__init__()
+        if n_heads % n_kv_heads != 0:
+            raise ValueError(
+                f"num_attention_heads ({n_heads}) must be divisible by "
+                f"num_key_value_heads ({n_kv_heads})")
+        self.n_heads = n_heads
+        self.n_kv_heads = n_kv_heads
+        self.head_dim = head_dim
+        self.n_rep = n_heads // n_kv_heads
+        self.xsa = xsa
+
+        self.q_proj = nn.Linear(dim, n_heads * head_dim, bias=False)
+        self.k_proj = nn.Linear(dim, n_kv_heads * head_dim, bias=False)
+        self.v_proj = nn.Linear(dim, n_kv_heads * head_dim, bias=False)
+        self.o_proj = nn.Linear(n_heads * head_dim, dim, bias=False)
+
+        self.q_norm = RMSNorm(head_dim, eps=norm_eps)
+        self.k_norm = RMSNorm(head_dim, eps=norm_eps)
+
+        self.value_residual = value_residual
+        if value_residual:
+            self.v_mix = nn.Parameter(torch.full((1,), -4.0))
+
+    def forward(self, x, cos, sin, attn_mask=None, v_first=None):
+        B, T, _ = x.shape
+        q = self.q_proj(x).view(B, T, self.n_heads, self.head_dim)
+        k = self.k_proj(x).view(B, T, self.n_kv_heads, self.head_dim)
+        v = self.v_proj(x).view(B, T, self.n_kv_heads, self.head_dim)
+
+        q = self.q_norm(q)
+        k = self.k_norm(k)
+        q = apply_rope(q, cos, sin)
+        k = apply_rope(k, cos, sin)
+
+        v_raw = v
+        if self.value_residual and v_first is not None:
+            lam = torch.sigmoid(self.v_mix)
+            v = (1 - lam) * v + lam * v_first
+
+        q = q.transpose(1, 2)
+        k = k.transpose(1, 2).repeat_interleave(self.n_rep, dim=1)
+        v = v.transpose(1, 2).repeat_interleave(self.n_rep, dim=1)
+
+        if attn_mask is None:
+            out = F.scaled_dot_product_attention(q, k, v, is_causal=True)
+        else:
+            out = F.scaled_dot_product_attention(q, k, v, attn_mask=attn_mask)
+
+        if self.xsa:
+            num = (out * v).sum(dim=-1, keepdim=True)
+            den = (v * v).sum(dim=-1, keepdim=True).clamp_min(1e-6)
+            out = out - (num / den) * v
+
+        out = out.transpose(1, 2).contiguous().view(B, T, -1)
+        return self.o_proj(out), v_raw
+
+
+class SwiGLU(nn.Module):
+    def __init__(self, dim, hidden):
+        super().__init__()
+        self.gate_proj = nn.Linear(dim, hidden, bias=False)
+        self.up_proj = nn.Linear(dim, hidden, bias=False)
+        self.down_proj = nn.Linear(hidden, dim, bias=False)
+
+    def forward(self, x):
+        return self.down_proj(F.silu(self.gate_proj(x)) * self.up_proj(x))
+
+
+class ReLU2MLP(nn.Module):
+    def __init__(self, dim, hidden):
+        super().__init__()
+        self.up_proj = nn.Linear(dim, hidden, bias=False)
+        self.down_proj = nn.Linear(hidden, dim, bias=False)
+
+    def forward(self, x):
+        return self.down_proj(F.relu(self.up_proj(x)).square())
+
+
+class Canon(nn.Module):
+    def __init__(self, dim, kernel=4):
+        super().__init__()
+        self.kernel = kernel
+        self.conv = nn.Conv1d(dim, dim, kernel, groups=dim, bias=False)
+
+    def forward(self, x):
+        y = F.pad(x.transpose(1, 2), (self.kernel - 1, 0))
+        return x + self.conv(y).transpose(1, 2)
+
+
+class DecoderLayer(nn.Module):
+    def __init__(self, dim, n_heads, n_kv_heads, head_dim, mlp_hidden, norm_eps,
+                 xsa, mlp_type, canon, value_residual):
+        super().__init__()
+        self.attn_norm = RMSNorm(dim, eps=norm_eps)
+        self.attn = Attention(dim, n_heads, n_kv_heads, head_dim, norm_eps,
+                              xsa=xsa, value_residual=value_residual)
+        self.mlp_norm = RMSNorm(dim, eps=norm_eps)
+        self.mlp = ReLU2MLP(dim, mlp_hidden) if mlp_type == "relu2" else SwiGLU(dim, mlp_hidden)
+        self.canon = Canon(dim) if canon else None
+
+    def forward(self, x, cos, sin, attn_mask=None, v_first=None):
+        h = self.attn_norm(x)
+        if self.canon is not None:
+            h = self.canon(h)
+        a, v = self.attn(h, cos, sin, attn_mask, v_first)
+        x = x + a
+        x = x + self.mlp(self.mlp_norm(x))
+        return x, v
+
+
+class CagliostroPreTrainedModel(PreTrainedModel):
+    config_class = CagliostroConfig
+    base_model_prefix = "model"
+    supports_gradient_checkpointing = False
+    _no_split_modules = ["DecoderLayer"]
+    _tied_weights_keys = {"lm_head.weight": "embed_tokens.weight"}
+
+    def _init_weights(self, module):
+        if isinstance(module, nn.Linear):
+            nn.init.normal_(module.weight, mean=0.0, std=0.02)
+        elif isinstance(module, nn.Embedding):
+            nn.init.normal_(module.weight, mean=0.0, std=0.02)
+
+
+class CagliostroForCausalLM(CagliostroPreTrainedModel, GenerationMixin):
+    def __init__(self, config):
+        super().__init__(config)
+        dim = config.hidden_size
+        self.head_dim = dim // config.num_attention_heads
+        self.layer_repeat = max(1, config.layer_repeat)
+        self.logit_cap = config.logit_cap
+
+        mlp_hidden = config.intermediate_size
+        if config.mlp_type == "relu2":
+            mlp_hidden = int(round(mlp_hidden * 1.5 / 64) * 64)
+
+        self.embed_tokens = nn.Embedding(config.vocab_size, dim)
+        self.layers = nn.ModuleList([
+            DecoderLayer(
+                dim, config.num_attention_heads, config.num_key_value_heads,
+                self.head_dim, mlp_hidden, config.rms_norm_eps,
+                config.xsa, config.mlp_type, config.canon,
+                config.value_residual and i > 0,
+            )
+            for i in range(config.num_hidden_layers)
+        ])
+        self.norm_out = RMSNorm(dim, eps=config.rms_norm_eps)
+        self.lm_head = nn.Linear(dim, config.vocab_size, bias=False)
+
+        self.rope_theta = config.rope_theta
+        self._rope_cache = None
+
+        self.post_init()
+
+    def _rope(self, seq_len, device, dtype):
+        key = (seq_len, str(device), str(dtype))
+        if self._rope_cache is None or self._rope_cache[0] != key:
+            freqs = rope_freqs(seq_len, self.head_dim, self.rope_theta, device=device)
+            self._rope_cache = (key,
+                                freqs.cos()[None, :, None, :].to(dtype),
+                                freqs.sin()[None, :, None, :].to(dtype))
+        return self._rope_cache[1], self._rope_cache[2]
+
+    def get_input_embeddings(self):
+        return self.embed_tokens
+
+    def set_input_embeddings(self, value):
+        self.embed_tokens = value
+
+    def get_output_embeddings(self):
+        return self.lm_head
+
+    def set_output_embeddings(self, value):
+        self.lm_head = value
+
+    def _causal_mask(self, attention_mask, B, T, dtype, device):
+        causal = torch.ones(T, T, dtype=torch.bool, device=device).tril()
+        mask = causal[None, None, :, :].expand(B, 1, T, T)
+        if attention_mask is not None:
+            pad = attention_mask.to(torch.bool)[:, None, None, :]
+            mask = mask & pad
+        out = torch.zeros(mask.shape, dtype=dtype, device=device)
+        return out.masked_fill(~mask, torch.finfo(dtype).min)
+
+    def forward(self, input_ids=None, attention_mask=None, labels=None,
+                inputs_embeds=None, use_cache=None, output_attentions=None,
+                output_hidden_states=None, return_dict=None, **kwargs):
+        if inputs_embeds is None:
+            x = self.embed_tokens(input_ids)
+        else:
+            x = inputs_embeds
+        B, T, _ = x.shape
+
+        cos, sin = self._rope(T, x.device, x.dtype)
+
+        attn_mask = None
+        if attention_mask is not None and attention_mask.dim() == 2 and not bool(attention_mask.all()):
+            attn_mask = self._causal_mask(attention_mask, B, T, x.dtype, x.device)
+
+        v_first = None
+        for layer in self.layers:
+            for _ in range(self.layer_repeat):
+                x, v = layer(x, cos, sin, attn_mask, v_first)
+                if v_first is None:
+                    v_first = v
+
+        x = self.norm_out(x)
+        logits = self.lm_head(x)
+        if self.logit_cap:
+            logits = self.logit_cap * torch.tanh(logits / self.logit_cap)
+
+        loss = None
+        if labels is not None:
+            shift_logits = logits[:, :-1].float().reshape(-1, logits.size(-1))
+            shift_labels = labels[:, 1:].reshape(-1)
+            loss = F.cross_entropy(shift_logits, shift_labels, ignore_index=-100)
+
+        if return_dict is False:
+            return (loss, logits) if loss is not None else (logits,)
+        return CausalLMOutputWithPast(loss=loss, logits=logits)
+
+    def prepare_inputs_for_generation(self, input_ids, attention_mask=None, **kwargs):
+        return {"input_ids": input_ids, "attention_mask": attention_mask}

tinctura-v1.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:59475dbfb89b1ef9235a3bbe38b89de204ddc42d0194782c4fc7f364a239371f
+size 13674126

tokenizer_config.json

@@ -0,0 +1,7 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>",
+  "model_max_length": 2048
+}