raincandy-u commited on
Commit
ee26b01
·
verified ·
1 Parent(s): 5a82204

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +47 -0
README.md CHANGED
@@ -8,6 +8,53 @@ pipeline_tag: text-generation
8
  library_name: transformers
9
  ---
10
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
11
  # 🩵 Rain-100M — Model Card
12
 
13
  **Rain-100M** 是一个基于 **Qwen3 架构** 从零训练的实验语言模型。
 
8
  library_name: transformers
9
  ---
10
 
11
+
12
+ # 🩵 Rain-100M — Model Card
13
+
14
+ **Rain-100M** is an experimental language model trained from scratch based on the **Qwen3 architecture**.
15
+
16
+ ## 🧠 Training & Data
17
+
18
+ * **Training corpus**: `HuggingFaceFW/fineweb-edu`
19
+ * **Total tokens**: ~**3B**
20
+ * **Language**: English only
21
+ * **Tokenizer**: Newly trained **16k BPE** (optimized for small/compact models)
22
+ * **Max sequence length**: 4096
23
+
24
+ **Sample training metrics**:
25
+
26
+ ```text
27
+ train/grad_norm: 0.6640625
28
+ train/learning_rate: 0.00000000002171853813
29
+ train/loss: 3.4459
30
+ ```
31
+
32
+ ## 🏗️ Architecture (Qwen3-style)
33
+
34
+ * **Parameters**: ~100M
35
+ * **Layers**: 12 Transformer layers
36
+ * **Hidden size**: 768
37
+ * **Attention heads**: 12
38
+ * **MLP dimension**: 2048
39
+ * **Activation**: SiLU
40
+ * **Weight dtype**: bfloat16
41
+ * **RMSNorm eps**: 1e-6
42
+ * **RoPE θ**: 10000
43
+ * **Inference framework**: `transformers`
44
+
45
+ ## ⚠️ Limitations
46
+
47
+ * Trained only on English data; weak or no capabilities in other languages, not suitable as a general-purpose chat model or for safety-critical use cases.
48
+ * No system-level alignment or safety fine-tuning has been applied.
49
+
50
+ ## 📄 License
51
+
52
+ When using this model locally, please also comply with the licenses of the `fineweb-edu` dataset and the `transformers` / Qwen3-related components.
53
+
54
+ ---
55
+
56
+ # 中文版本
57
+
58
  # 🩵 Rain-100M — Model Card
59
 
60
  **Rain-100M** 是一个基于 **Qwen3 架构** 从零训练的实验语言模型。