Spaces:
Running
Running
| <html lang="en"> | |
| <head> | |
| <meta charset="UTF-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> | |
| <title>SupraLabs | Information Routing: Dynamic Curriculum Shifting</title> | |
| <script src="https://cdn.jsdelivr.net/npm/chart.js"></script> | |
| <style> | |
| :root { | |
| --bg: #0f0f0f; | |
| --surface: #1a1a1a; | |
| --border: #333; | |
| --text: #e0e0e0; | |
| --accent: #536bfe; /* Supra Blue */ | |
| --muted: #888; | |
| --success: #00e676; | |
| --warning: #ffb300; | |
| --font-mono: 'JetBrains Mono', 'Fira Code', monospace; | |
| } | |
| * { margin: 0; padding: 0; box-sizing: border-box; } | |
| body { | |
| background-color: var(--bg); | |
| color: var(--text); | |
| font-family: 'Inter', -apple-system, sans-serif; | |
| line-height: 1.6; | |
| padding: 2rem; | |
| } | |
| code, pre, .mono { font-family: var(--font-mono); } | |
| .container { max-width: 1000px; margin: 0 auto; } | |
| header { | |
| border-bottom: 2px solid var(--border); | |
| padding-bottom: 2rem; | |
| margin-bottom: 3rem; | |
| display: flex; | |
| justify-content: space-between; | |
| align-items: flex-end; | |
| } | |
| .logo-area a { text-decoration: none; color: inherit; } | |
| .logo-area h1 { | |
| font-size: 1.2rem; | |
| text-transform: uppercase; | |
| letter-spacing: 2px; | |
| color: var(--accent); | |
| line-height: 1; | |
| display: flex; | |
| align-items: center; | |
| gap: 10px; | |
| } | |
| nav a { | |
| color: var(--text); | |
| text-decoration: none; | |
| margin-left: 1.5rem; | |
| font-size: 0.9rem; | |
| border-bottom: 1px solid transparent; | |
| } | |
| nav a:hover { border-bottom: 1px solid var(--accent); } | |
| .hero { margin-bottom: 4rem; } | |
| .hero h2 { font-size: 3rem; line-height: 1.1; margin-bottom: 1.5rem; font-weight: 800; } | |
| .hero p { font-size: 1.2rem; color: var(--muted); max-width: 750px; } | |
| .section-label { | |
| display: block; | |
| font-family: var(--font-mono); | |
| color: var(--accent); | |
| font-size: 0.8rem; | |
| margin-top: 3rem; | |
| margin-bottom: 1rem; | |
| text-transform: uppercase; | |
| } | |
| .card { background: var(--surface); border: 1px solid var(--border); padding: 2.5rem; margin-bottom: 2rem; } | |
| h3 { font-size: 1.6rem; margin-bottom: 1rem; font-weight: 700; } | |
| p { margin-bottom: 1rem; color: #ccc; } | |
| ul { margin-left: 1.5rem; margin-bottom: 1.5rem; } | |
| li { margin-bottom: 0.5rem; } | |
| .winner-badge { color: var(--success); font-weight: bold; font-family: var(--font-mono); } | |
| .table-container { overflow-x: auto; margin: 2rem 0; border: 1px solid var(--border); } | |
| table { width: 100%; border-collapse: collapse; text-align: left; font-size: 0.95rem; } | |
| th, td { padding: 1rem; border-bottom: 1px solid var(--border); } | |
| th { background-color: rgba(26, 26, 26, 0.8); font-family: var(--font-mono); color: var(--accent); font-size: 0.85rem; text-transform: uppercase; } | |
| tr:hover { background-color: rgba(83, 107, 254, 0.05); } | |
| .highlight-row { border-left: 4px solid var(--success); background-color: rgba(0, 230, 118, 0.02); } | |
| .chart-box { background: var(--surface); border: 1px solid var(--border); padding: 2rem; margin-bottom: 2rem; } | |
| .stats-grid { | |
| display: grid; | |
| grid-template-columns: 1fr 1fr 1fr; | |
| gap: 1rem; | |
| margin-top: 4rem; | |
| border-top: 1px solid var(--border); | |
| padding-top: 2rem; | |
| } | |
| .stat-box { padding: 1rem; border-left: 2px solid var(--accent); } | |
| .stat-box small { display: block; color: var(--muted); font-family: var(--font-mono); } | |
| footer { margin-top: 6rem; padding-bottom: 2rem; font-size: 0.8rem; color: var(--muted); text-align: center; } | |
| @media (max-width: 768px) { | |
| .hero h2 { font-size: 2rem; } | |
| header { flex-direction: column; align-items: flex-start; gap: 1rem; } | |
| nav a { margin-left: 0; margin-right: 1rem; } | |
| .stats-grid { grid-template-columns: 1fr; } | |
| } | |
| </style> | |
| </head> | |
| <body> | |
| <div class="container"> | |
| <header> | |
| <div class="logo-area" style="font-size: 1.5em;"> | |
| <a href="index.html"><h1><img src="./image.png" style="height: 2em" alt="Logo"> SupraLabs_</h1></a> | |
| </div> | |
| <nav> | |
| <a href="#summary">Core Learnings</a> | |
| <a href="#benchmarks">Routing Matrix</a> | |
| <a href="#charts">Visualizations</a> | |
| <a href="https://huggingface.co/SupraLabs" target="_blank">HuggingFace</a> | |
| </nav> | |
| </header> | |
| <section class="hero"> | |
| <h2>Experiment #7:<br>Information Routing — Curriculum Shifting</h2> | |
| <p>An empirical evaluation of structured knowledge injection schedules inside parameter-constrained spaces. We subjected a optimized <strong>4,196,096 parameter architecture</strong> to a rigid budget of <strong>200,000,000 total tokens</strong> to test multi-distribution blending against raw uniform pooling.</p> | |
| </section> | |
| <span class="section-label" id="summary">// Syntactic_Preservation_&_Memory_Retainment</span> | |
| <div class="card"> | |
| <h3>Curing Catastrophic Forgetting via Shifting Weights</h3> | |
| <p>When training extremely compact language models, data sequencing determines memory stabilization boundaries. Our comparative routing sweep demonstrates that standard dataset pooling bottlenecks the structural capability of micro-networks:</p> | |
| <ul> | |
| <li><strong>The Failure of Rigid Sequential Partitioning:</strong> Processing data in unblended sequential stages (Hard Blocks) triggers devastating memory overwrites. As the training block moves from base syntax to intense facts, the network suffers a 39% language accuracy collapse (Wikitext PPL: 632.9).</li> | |
| <li><strong>The Naive Pooling Limitation:</strong> Shoving all token distributions into a single, uniform mix from the start (Baseline) forces the small weights matrix to stabilize multiple contradictory targets simultaneously, degrading the final linguistic resolution (PPL: 556.7).</li> | |
| <li><strong>The Shifting Weight Solution:</strong> Launching with structured syntax (TinyStories/Cosmo), moving into reasoning (Math), and finalizing with facts while preserving a mathematical logic holding current protects the weight paths. Language Perplexity drops cleanly to <strong>453.6</strong>.</li> | |
| </ul> | |
| </div> | |
| <span class="section-label" id="benchmarks">// Knowledge_Injection_Routing_Matrix</span> | |
| <div class="card" style="padding: 1.5rem;"> | |
| <h3>Empirical Scheduling Analytics</h3> | |
| <p>All training pipelines utilize an identical 4096 vocabulary architecture. Downstream tracking processed zero-shot. Deceptive optimization targets (apparent pretrain loss) are directly falsified by true language modeling benchmarks.</p> | |
| <div class="table-container"> | |
| <table> | |
| <thead> | |
| <tr> | |
| <th>Benchmark / Metric</th> | |
| <th style="color: var(--success)">Run 1: Shifting Weights (🏆 Win)</th> | |
| <th>Run 2: Hard Sequential Blocks</th> | |
| <th>Run 3: Uniform Baseline Mix</th> | |
| </tr> | |
| </thead> | |
| <tbody> | |
| <tr> | |
| <td class="mono">Routing Architecture</td> | |
| <td style="color: var(--success)">Dynamic Weight Stepper</td> | |
| <td>Static Linear Switches</td> | |
| <td>Star Blend Pooling</td> | |
| </tr> | |
| <tr> | |
| <td class="mono">Final Pretrain Step Loss (↓)</td> | |
| <td>2.770</td> | |
| <td class="winner-badge">2.458</td> | |
| <td>2.690</td> | |
| </tr> | |
| <tr> | |
| <td class="mono">Final Pretrain Train Loss (↓)</td> | |
| <td>3.151</td> | |
| <td class="winner-badge">2.817</td> | |
| <td>3.114</td> | |
| </tr> | |
| <tr> | |
| <td class="mono">ARC-Easy Zero-Shot (↑)</td> | |
| <td>29.71%</td> | |
| <td class="winner-badge">30.22%</td> | |
| <td>30.18%</td> | |
| </tr> | |
| <tr> | |
| <td class="mono">ARC-Easy Acc Norm (↑)</td> | |
| <td>30.26%</td> | |
| <td class="winner-badge">30.64%</td> | |
| <td>30.47%</td> | |
| </tr> | |
| <tr> | |
| <td class="mono">Wikitext Byte PPL (↓)</td> | |
| <td class="winner-badge">3.1392</td> | |
| <td>3.3410</td> | |
| <td>3.2618</td> | |
| </tr> | |
| <tr> | |
| <td class="mono">Wikitext Word PPL (↓)</td> | |
| <td class="winner-badge" style="color: var(--success); font-weight: bold;">453.6715</td> | |
| <td>632.9729</td> | |
| <td>556.7454</td> | |
| </tr> | |
| <tr class="highlight-row"> | |
| <td style="font-weight: bold;">PRETRAIN COHERENCE</td> | |
| <td style="color: var(--success); font-weight: bold;">SOTA BALANCE ARCHIEVED</td> | |
| <td style="color: var(--warning)">Severe Memory Corruption</td> | |
| <td>Information Dilution</td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| </div> | |
| </div> | |
| <span class="section-label" id="charts">// Visualizing_Routing_Mechanics</span> | |
| <div class="chart-box"> | |
| <h3>Linguistic Perplexity Splintering Across Routing Approaches</h3> | |
| <div style="position: relative; height:350px; width:100%"> | |
| <canvas id="curriculumPplChart"></canvas> | |
| </div> | |
| </div> | |
| <div class="chart-box"> | |
| <h3>The Apparent Loss Paradox: Pretrain Loss vs. True Structural Perplexity</h3> | |
| <p style="font-size: 0.85rem; color: var(--muted); margin-bottom: 1.5rem;">Critical discovery: Rigid data blocking pushes the localized training loss down artificially, but causes severe out-of-distribution language collapse on core evaluation matrices.</p> | |
| <div style="position: relative; height:350px; width:100%"> | |
| <canvas id="curriculumParadoxChart"></canvas> | |
| </div> | |
| </div> | |
| <section class="stats-grid" id="hardware"> | |
| <div class="stat-box"> | |
| <small>TOTAL COMPUTE EXPOSURE</small> | |
| <strong>200M Unified Steps</strong> | |
| </div> | |
| <div class="stat-box"> | |
| <small>MODEL GEOMETRY PROFILE</small> | |
| <strong>3-Layer Shallow Wide SOTA</strong> | |
| </div> | |
| <div class="stat-box"> | |
| <small>SCHEDULING RESOLUTION</small> | |
| <strong>4096 Vocab Pareto Bound</strong> | |
| </div> | |
| </section> | |
| <footer> | |
| <p>© 2026 SupraLabs. High performance. Small footprints. Proudly open-source.</p> | |
| </footer> | |
| </div> | |
| <script> | |
| // Curriculum Perplexity Chart | |
| const ctxCurrPpl = document.getElementById('curriculumPplChart').getContext('2d'); | |
| new Chart(ctxCurrPpl, { | |
| type: 'bar', | |
| data: { | |
| labels: ['Run 1: Dynamic Shifting Weights', 'Run 2: Hard Blocks', 'Run 3: Uniform Baseline Mix'], | |
| datasets: [{ | |
| label: 'Wikitext Word Perplexity (Lower = Better)', | |
| data: [453.67, 632.97, 556.74], | |
| backgroundColor: ['rgba(0, 230, 118, 0.3)', 'rgba(229, 57, 53, 0.2)', 'rgba(255, 179, 0, 0.2)'], | |
| borderColor: ['#00e676', '#e53935', '#ffb300'], | |
| borderWidth: 2 | |
| }] | |
| }, | |
| options: { | |
| responsive: true, | |
| maintainAspectRatio: false, | |
| plugins: { legend: { labels: { color: '#bbb' } } }, | |
| scales: { | |
| y: { grid: { color: '#222' }, ticks: { color: '#888' } }, | |
| x: { grid: { display: false }, ticks: { color: '#aaa' } } | |
| } | |
| } | |
| }); | |
| // Paradox Dual Axis Curve | |
| const ctxCurrParadox = document.getElementById('curriculumParadoxChart').getContext('2d'); | |
| new Chart(ctxCurrParadox, { | |
| type: 'line', | |
| data: { | |
| labels: ['Dynamic Shifting', 'Hard Sequential Blocks', 'Uniform Baseline Mix'], | |
| datasets: [ | |
| { | |
| label: 'Apparent Pretrain Train Loss (Lower = Deceptive ↓)', | |
| data: [3.151, 2.817, 3.114], | |
| borderColor: 'rgba(229, 57, 53, 0.5)', | |
| backgroundColor: 'transparent', | |
| yAxisID: 'yLoss', | |
| tension: 0.1, | |
| borderWidth: 2 | |
| }, | |
| { | |
| label: 'True Linguistic Word Perplexity (Lower = Superior ↑)', | |
| data: [453.67, 632.97, 556.74], | |
| borderColor: '#536bfe', | |
| backgroundColor: 'transparent', | |
| yAxisID: 'yPpl', | |
| tension: 0.1, | |
| borderWidth: 3, | |
| borderDash: [3, 3] | |
| } | |
| ] | |
| }, | |
| options: { | |
| responsive: true, | |
| maintainAspectRatio: false, | |
| plugins: { legend: { labels: { color: '#bbb' } } }, | |
| scales: { | |
| x: { grid: { display: false }, ticks: { color: '#aaa' } }, | |
| yLoss: { | |
| type: 'linear', | |
| position: 'left', | |
| title: { display: true, text: 'Pretrain Loss Baseline', color: 'rgba(229, 57, 53, 0.8)' }, | |
| grid: { color: '#222' }, | |
| ticks: { color: '#888' } | |
| }, | |
| yPpl: { | |
| type: 'linear', | |
| position: 'right', | |
| title: { display: true, text: 'Linguistic Word Perplexity', color: '#536bfe' }, | |
| grid: { display: false }, | |
| ticks: { color: '#888' } | |
| } | |
| } | |
| } | |
| }); | |
| </script> | |
| </body> | |
| </html> |