Text Generation
Transformers
Safetensors
English
rrt
recursive-transformer
small-language-model
knowledge-distillation
base-model
custom_code
Instructions to use sz14/cRia-LM-75M with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use sz14/cRia-LM-75M with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="sz14/cRia-LM-75M", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("sz14/cRia-LM-75M", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use sz14/cRia-LM-75M with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "sz14/cRia-LM-75M" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "sz14/cRia-LM-75M", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/sz14/cRia-LM-75M
- SGLang
How to use sz14/cRia-LM-75M with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "sz14/cRia-LM-75M" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "sz14/cRia-LM-75M", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "sz14/cRia-LM-75M" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "sz14/cRia-LM-75M", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use sz14/cRia-LM-75M with Docker Model Runner:
docker model run hf.co/sz14/cRia-LM-75M
|
Download README.md from sz14/cRia-LM-75M: direct link, hf CLI and curl.
- Browser
- Download file 17.1 kB
-
https://huggingface.co/sz14/cRia-LM-75M/resolve/main/README.md
- Command line
-
hf download hf://sz14/cRia-LM-75M/README.md
-
curl -L -o README.md https://huggingface.co/sz14/cRia-LM-75M/resolve/main/README.md
17.1 kB
| license: apache-2.0 | |
| language: | |
| - en | |
| library_name: transformers | |
| pipeline_tag: text-generation | |
| tags: | |
| - recursive-transformer | |
| - small-language-model | |
| - knowledge-distillation | |
| - base-model | |
| - custom_code | |
| datasets: | |
| - HuggingFaceFW/fineweb-edu | |
| - HuggingFaceTB/dclm-edu | |
| - HuggingFaceTB/smollm-corpus | |
| - HuggingFaceTB/cosmopedia | |
| - HuggingFaceTB/finemath | |
| - bigcode/starcoderdata | |
| - epfml/FineWeb-HQ | |
| - wikimedia/wikipedia | |
| - nvidia/Nemotron-CC-Math-v1 | |
| - codelion/sutra-1B | |
| # CRIA-LM-75M | |
| > **Cria** *(noun)*: a baby llama, alpaca, vicuña, or guanaco. Pronounced ˈkrē-ə. | |
| > | |
| > ~ *Merriam-Webster* | |
| *** | |
|  | |
| ___ | |
| **cRia-LM-75M** is a 75.7M-parameter base language model built as a Relaxed Recursive Transformer (RRT). It uses a shared 11-layer recurrent block evaluated twice, with pass-specific LoRA parameters on the second traversal. | |
| Training was carried out in three stages. Stage 1 established the 2K base model over 10B tokens. Stage 2 continued training with a 2B-token budget and a capability-focused data curriculum; the released Stage 2 checkpoint is step 10,000, corresponding to about 1.31B continuation tokens. Stage 3 extended the context window from 2,048 to 4,096 tokens with a 50M-token run on `codelion/sutra-1B`. The released checkpoint continues from that long-context stage with 35 learned residual-gain scalars and one epoch over the HellaSwag, ARC-Easy, ARC-Challenge, PIQA, SciQ and SWAG training splits. | |
| This is a base model. It has not been instruction tuned and does not use a chat template. The Transformer layer design follows `HuggingFaceTB/SmolLM2-135M`, with QK-Norm added for training stability. | |
| ## Model Details | |
| | Specification | Value | | |
| | --- | --- | | |
| | Parameters | 75.7M | | |
| | Architecture | Relaxed Recursive Transformer | | |
| | Unique Transformer layers | 13 | | |
| | Effective depth | 24 | | |
| | Prelude layers | 1 | | |
| | Shared recurrent layers | 11 | | |
| | Recurrent passes | 2 | | |
| | Coda layers | 1 | | |
| | Hidden size | 576 | | |
| | MLP intermediate size | 1,536 | | |
| | Attention heads | 9 query heads | | |
| | KV heads | 3 | | |
| | Attention type | Grouped-query attention | | |
| | Head dimension | 64 | | |
| | MLP | SwiGLU | | |
| | Normalization | RMSNorm | | |
| | Attention normalization | QK-Norm | | |
| | Position encoding | RoPE | | |
| | RoPE theta | 100,000 | | |
| | Context length | 4,096 tokens | | |
| | Vocabulary size | 49,152 | | |
| | Tokenizer | SmolLM2 BPE tokenizer, per-digit tokenized | | |
| | Token embedding | Tied, factorized | | |
| | Embedding rank | 210 | | |
| | Recurrent LoRA rank | 172 | | |
| | Residual gains | 35 learned scalars | | |
| | KV cache | Recursion-wise, one slot per effective layer/pass | | |
| | Model type | Base causal language model | | |
| ## Architecture | |
| cRia-LM-75M uses 13 unique Transformer layers arranged as: | |
| ```text | |
| 1 prelude + (11 shared layers x 2 recurrent passes) + 1 coda | |
| ``` | |
| This gives an effective depth of 24 Transformer layers while storing the main parameters for only 13 unique layers. | |
| The 11-layer recurrent block is shared across both passes. On the second pass, the recurrent linear projections receive rank-172 LoRA updates. The two traversals therefore share the base weights while retaining pass-specific capacity. | |
| Each Transformer layer uses: | |
| - hidden size 576 | |
| - 9 query heads and 3 KV heads with grouped-query attention | |
| - head dimension 64 | |
| - SwiGLU feed-forward network with intermediate size 1,536 | |
| - RMSNorm | |
| - rotary position embeddings | |
| - QK-Norm | |
| <a href="https://hfviewer.com/sz14/cRia-LM-75M?utm_source=huggingface&utm_medium=embedded_model_card&utm_campaign=sz14__cRia-LM-75M_card&utm_content=embedded_card_open_viewer&from=embedded-model-card" target="_blank" rel="noopener"> | |
| <img | |
| src="https://hfviewer.com/api/card.svg?source=sz14%2FcRia-LM-75M&granularity=auto&v=20260516-title-pills-card" | |
| alt="Architecture graph for sz14/cRia-LM-75M. Open in hfviewer" | |
| width="100%" | |
| /> | |
| </a> | |
| ### Factorized Tied Embedding | |
| The input embedding and language-model readout are tied through a rank-210 factorization: | |
| ```text | |
| 49,152 x 210 | |
| 210 x 576 | |
| ``` | |
| The same factors are used for token lookup and output projection. This keeps the full 49,152-token vocabulary while reducing the parameter cost of the embedding and output head. | |
| ## Training | |
| ### Stage 1 — Base pretraining | |
| Stage 1 trained the model for **10B tokens** at a sequence length of **2,048**. The objective combined next-token cross-entropy with logit-level knowledge distillation from `HuggingFaceTB/SmolLM2-360M`, with dynamic CE/KD scale balancing. | |
| The Stage 1 sampling mix was: | |
| | Data source | Sampling weight | | |
| | --- | ---: | | |
| | FineWeb-Edu | 48% | | |
| | DCLM-Edu | 32% | | |
| | Cosmopedia-v2 | 12% | | |
| | FineMath-4+ | 5% | | |
| | StarCoder Python | 3% | | |
| Stage 1 used Muon for matrix parameters and AdamW for the remaining parameter groups. The learning-rate schedule was warmup-stable-decay: 1% warmup, decay beginning at 80% of the run, and the final 20% used for decay. | |
| ### Stage 2 — Capability mid-training | |
| Stage 2 used a **2B-token training budget** at 2,048-token context. The released Stage 2 checkpoint is **step 10,000**, or **1,310,720,000 tokens** at 131,072 tokens per optimizer step. | |
| The phase used two-teacher distillation: same-tokenizer KL from `HuggingFaceTB/SmolLM2-360M` and cross-tokenizer supervision from `Qwen/Qwen3.5-0.8B-Base` using [X-Token](https://arxiv.org/abs/2605.21699). The run used the H-KL cross-tokenizer objective after the tokenizer-coverage audit, with dynamic CE/KD balancing retained around the combined distillation loss. | |
| #### Phase 2A mix | |
| Phase 2A is the capability-injection mix. It remains active through 65% of the planned Stage 2 run. The original configuration assigned 8% to Nemotron QA, but that source was unavailable when the run started. The loader dropped it and renormalized the remaining weights; the table below shows the effective mix that was actually sampled. | |
| | Data source | Effective sampling weight | | |
| | --- | ---: | | |
| | FineWeb-Edu | 19.57% | | |
| | DCLM-Edu | 13.04% | | |
| | Cosmopedia-v2 | 10.87% | | |
| | Cosmopedia Stories | 8.70% | | |
| | Cosmopedia WikiHow | 7.61% | | |
| | Cosmopedia Science | 9.78% | | |
| | Wikipedia | 10.87% | | |
| | FineMath | 5.43% | | |
| | Nemotron Math | 3.26% | | |
| | InfiWebMath | 3.26% | | |
| | StarCoder Python | 4.35% | | |
| | FineWeb-HQ | 3.26% | | |
| #### Phase 2B mix | |
| Phase 2B is the cooldown mix. It removes the noisier sources and shifts more weight toward higher-quality web, educational, encyclopedic, science, and math data. | |
| | Data source | Sampling weight | | |
| | --- | ---: | | |
| | FineWeb-HQ | 18% | | |
| | Cosmopedia-v2 | 14% | | |
| | Cosmopedia Science | 14% | | |
| | Wikipedia | 14% | | |
| | FineMath | 12% | | |
| | Cosmopedia WikiHow | 8% | | |
| | Cosmopedia Stories | 8% | | |
| | DCLM-Edu | 8% | | |
| | StarCoder Python | 4% | | |
| #### Stage 2 WSD schedule | |
| Stage 1 had already completed a full decay, so Stage 2 re-warmed to a lower peak rather than returning to the original Stage 1 learning rate. | |
| | Setting | Value | | |
| | --- | ---: | | |
| | Stage 2 token budget | 2.00B | | |
| | Tokens per optimizer step | 131,072 | | |
| | Planned optimizer steps | 15,258 | | |
| | AdamW base LR | 1e-4 | | |
| | Muon base LR | 0.006 | | |
| | Re-warm | first 3% | | |
| | Stable region | through 55% | | |
| | WSD decay start | 55% (~1.10B tokens) | | |
| | Phase 2B mix switch | 65% (~1.30B tokens) | | |
| | Minimum LR fraction | 2% of the Stage 2 base LR | | |
| | Released checkpoint | step 10,000 (~1.31B tokens) | | |
| After the stable region, the LR multiplier follows the run's square-root WSD decay: | |
| ```text | |
| lr_scale = max(0.02, 1 - sqrt(decay_progress)) | |
| ``` | |
| Because the data-mix switch occurs at 65%, the final portion of Phase 2A already runs under LR decay. The selected step-10,000 checkpoint lands just after the switch to Phase 2B. | |
| ### Stage 3 — 2K to 4K context extension | |
| Stage 3 increases the supported context length from **2,048 to 4,096 tokens**. The run was configured for **50M tokens** on `codelion/sutra-1B`, with the long-context checkpoint used for this release selected at **step 100**, roughly **15M tokens** into the run. The released model then continued with 35 learned residual-gain scalars for one epoch over the HellaSwag, ARC-Easy, ARC-Challenge, PIQA, SWAG, and SciQ training splits. | |
| The context-length curriculum used the following token fractions: | |
| | Sequence length | Token fraction | | |
| | --- | ---: | | |
| | 1,024 | 25% | | |
| | 2,048 | 40% | | |
| | 4,096 | 35% | | |
| The training path uses Apple's [Cut Cross Entropy](https://github.com/apple/ml-cross-entropy) directly against cRia's factorized tied output head. The `cce_exact` implementation was used to avoid gradient filtering while reducing the memory cost of materializing full-vocabulary logits for CE computation. | |
| RoPE theta remains **100,000** and no RoPE scaling is applied. The model has no learned absolute position table, so moving from 2K to 4K does not add parameters. | |
| ## Evaluation | |
| The complete standard suite was evaluated zero-shot with FP32 model tensors and FP32 log-softmax. CUDA TF32 matmuls were enabled. | |
| | Benchmark | Metric | Score | Std. error | | |
| | --- | ---: | ---: | ---: | | |
| | ARC-Challenge | acc_norm | 30.46 | 0.0134 | | |
| | ARC-Easy | acc_norm | 58.16 | 0.0101 | | |
| | ArithMark-3 | acc_norm | 37.30 | 0.0153 | | |
| | Balanced COPA | acc | 58.70 | 0.0156 | | |
| | BananaMind Base Bench 1.1 | raw accuracy | 60.86 | — | | |
| | BLiMP | acc | 79.06 | 0.0015 | | |
| | BoolQ | acc | 53.82 | 0.0087 | | |
| | CommonsenseQA | acc | 20.48 | 0.0116 | | |
| | HellaSwag | acc_norm | 38.95 | 0.0049 | | |
| | LAMBADA | acc | 38.17 | 0.0068 | | |
| | LAMBADA | perplexity | 43.67 | 1.9797 | | |
| | MMLU (continuation) | acc | 26.66 | 0.0037 | | |
| | OpenBookQA | acc_norm | 34.20 | 0.0212 | | |
| | PIQA | acc_norm | 64.74 | 0.0111 | | |
| | SciQ | acc_norm | 90.00 | 0.0095 | | |
| | TruthfulQA MC2 | acc | 41.74 | 0.0153 | | |
| | Winogrande | acc | 52.17 | 0.0140 | | |
| | **Average (accuracy metrics only)** | — | **49.09** | — | | |
| The average is the unweighted arithmetic mean of the 14 standard accuracy scores, ArithMark-3 normalized accuracy, and BananaMind raw accuracy. | |
| ### Comparison with established small language models | |
| | Benchmark | Metric | GPT-2 | OPT-125M | AMD-Llama-135M | SmolLM-135M | SmolLM2-135M | | |
| | --- | ---: | ---: | ---: | ---: | ---: | ---: | | |
| | ARC-Challenge | acc_norm | 22.70 | 22.53 | 26.02 | 28.75 | 29.61 | | |
| | ARC-Easy | acc_norm | 39.52 | 40.03 | 31.82 | 56.48 | 58.54 | | |
| | ArithMark-3 | acc_norm | 37.00 | 35.30 | 35.70 | 36.80 | 38.80 | | |
| | Balanced COPA | acc | 54.80 | 56.90 | 50.50 | 58.10 | 61.20 | | |
| | BananaMind Base Bench 1.1 | raw accuracy | 50.57 | 47.71 | 53.14 | 67.71 | 66.00 | | |
| | BLiMP | acc | 82.13 | 81.64 | 58.16 | 78.53 | 78.96 | | |
| | BoolQ | acc | 49.11 | 56.21 | 61.83 | 60.18 | 60.31 | | |
| | CommonsenseQA | acc | 19.57 | 19.98 | 19.57 | 19.90 | 19.57 | | |
| | HellaSwag | acc_norm | 31.12 | 31.60 | 27.15 | 42.61 | 43.08 | | |
| | LAMBADA | acc | 32.56 | 38.81 | 21.19 | 37.73 | 42.89 | | |
| | LAMBADA | perplexity | 40.07 | 23.13 | 1,074.67 | 32.33 | 19.26 | | |
| | MMLU (continuation) | acc | 25.47 | 25.15 | 25.42 | 28.92 | 29.90 | | |
| | OpenBookQA | acc_norm | 27.20 | 28.80 | 26.00 | 34.40 | 33.40 | | |
| | PIQA | acc_norm | 62.57 | 62.02 | 60.66 | 68.17 | 68.50 | | |
| | SciQ | acc_norm | 64.30 | 70.40 | 47.20 | 74.80 | 78.20 | | |
| | TruthfulQA MC2 | acc | 40.68 | 42.87 | 42.73 | 39.28 | 38.79 | | |
| | Winogrande | acc | 51.46 | 51.38 | 51.93 | 53.28 | 53.04 | | |
| | **Average (accuracy metrics only)** | — | **43.17** | **44.46** | **39.94** | **49.10** | **50.05** | | |
| We don't make any claims about inferiority or superiority of any model. | |
| While we took care of avoiding leakage from validation and test sets, we DID use the train splits of several of these benchmarks in our pre-training corpus. | |
| Compared to other models who had a longer training regime on non-benchmark data, it is possible real performance may lag behind what these benchmarks indicate (LAMBADA perplexity and MMLU already shows this trend somewhat). | |
| ## Inference Performance | |
| Measurements used FP32 weights, batch size 1, synchronized CUDA medians, PyTorch 2.12.0+cu130, WSL2, and an RTX 4070 Ti. | |
| | Measurement | Result | | |
| | --- | ---: | | |
| | 128-token cached prefill | 17.85 ms | | |
| | 512-token cached prefill | 17.49 ms | | |
| | 2,048-token cached prefill | 49.54 ms | | |
| | 128-token prompt + 128-token decode | 62.97 tokens/s | | |
| | Resident CUDA allocation after warm-up | 305.20 MiB | | |
| | Incremental peak CUDA allocation, 128 + 128 | 10.358 MiB | | |
| | Reserved CUDA memory | 328 MiB | | |
| For repeated unpadded greedy requests of one fixed shape, the reusable CUDA graph runner reached **318.77 tokens/s** with a 128-token prompt and 64 generated tokens. The graph is inference-only, is fixed to its configured batch/prompt/output shape, and must be rebuilt after training changes model parameters. | |
| ## Usage | |
| The model ships with a custom Transformers implementation, so `trust_remote_code=True` is required for the model. The tokenizer is loaded directly from the cRia repository. | |
| `model.generate()` uses the recursion-wise KV cache by default. `use_cache=True` is shown explicitly below so the inference path is clear. | |
| ```python | |
| import torch | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| model_id = "sz14/cRia-LM-75M" | |
| device = "cuda" if torch.cuda.is_available() else "cpu" | |
| tokenizer = AutoTokenizer.from_pretrained(model_id) | |
| model = AutoModelForCausalLM.from_pretrained( | |
| model_id, | |
| trust_remote_code=True, | |
| dtype=torch.float32, | |
| ).to(device).eval() | |
| prompt = "In mathematics, a vector is" | |
| inputs = tokenizer(prompt, return_tensors="pt").to(device) | |
| with torch.no_grad(): | |
| output = model.generate( | |
| **inputs, | |
| max_new_tokens=64, | |
| do_sample=False, | |
| use_cache=True, | |
| ) | |
| print(tokenizer.decode(output[0], skip_special_tokens=True)) | |
| ``` | |
| ### Explicit KV-cache usage | |
| For custom decode loops, a cache can also be created and passed through `past_key_values` directly: | |
| ```python | |
| prompt = "In mathematics, a vector is" | |
| inputs = tokenizer(prompt, return_tensors="pt").to(device) | |
| cache = model.new_cache(batch=inputs.input_ids.shape[0]) | |
| with torch.no_grad(): | |
| # Prefill the prompt and populate the cache. | |
| out = model( | |
| **inputs, | |
| past_key_values=cache, | |
| use_cache=True, | |
| ) | |
| # Decode one token using only the new token plus cached K/V states. | |
| next_token = out.logits[:, -1].argmax(dim=-1, keepdim=True) | |
| out = model( | |
| input_ids=next_token, | |
| past_key_values=cache, | |
| use_cache=True, | |
| ) | |
| print("cached positions:", cache.pos) | |
| ``` | |
| Pass `use_cache=False` to `model.generate()` to fall back to full-prefix recomputation. | |
| ### KV-cache implementation | |
| The recurrent block shares weights across passes, but the two passes do not share keys and values. The second traversal uses its own LoRA-adapted projections, so each recurrent pass receives a separate cache slot. With the current architecture this produces **24 cache slots**, matching the model's 24 effective layers. | |
| The current implementation supports: | |
| - cached prompt prefill and token-by-token decode | |
| - batched generation with left padding | |
| - attention masks over the full cached key length | |
| - RoPE positions derived from the cumulative attention mask for padded batches | |
| - beam-search cache reordering | |
| - explicit cache cropping/resetting | |
| - a hard 4,096-token cache/context bound with clear errors beyond the configured window | |
| A standalone `model.generate_cached(...)` helper is also available for direct incremental decoding outside the standard Transformers generation path. | |
| ## Intended Use | |
| cRia-LM-75M is intended primarily for: | |
| - research on recursive parameter sharing | |
| - experiments with compact language models | |
| - further pretraining and domain adaptation | |
| - supervised fine-tuning | |
| - small language-model backbones | |
| - architecture and knowledge-distillation research | |
| Because this is a base model, prompts are treated as ordinary text continuation rather than instructions. | |
| ## Limitations | |
| cRia-LM-75M is a small base language model and should not be expected to match substantially larger pretrained models. | |
| Known limitations include: | |
| - weak multi-step reasoning | |
| - limited factual knowledge | |
| - potential factual errors and hallucinations | |
| - no instruction-following training | |
| - no chat template | |
| - English-focused training | |
| - a 4,096-token maximum context window | |
| The model should not be treated as a reliable source of factual information or used without additional validation in high-stakes applications. | |
| ## Architecture and Training Lineage | |
| The Transformer layer design follows `HuggingFaceTB/SmolLM2-135M`, released by Hugging Face under the Apache 2.0 license. | |
| `HuggingFaceTB/SmolLM2-360M` was used for same-tokenizer logit distillation in the initial training and Stage 2. Stage 2 additionally used `Qwen/Qwen3.5-0.8B-Base` for cross-tokenizer X-Token distillation. | |
| The recursive parameter-sharing approach is based on: | |
| **Bae et al., "Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA," arXiv:2410.20672.** | |
| Stage 2 cross-tokenizer distillation uses: | |
| **Sreenivas et al., "X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation," arXiv:2605.21699.** | |
| Stage 3 uses Apple's Cut Cross Entropy implementation from **Wijmans et al., "Cut Your Losses in Large-Vocabulary Language Models," arXiv:2411.09009v2.** | |
| ## License | |
| cRia-LM-75M is released under the **Apache License 2.0**. |