Add files using upload-large-folder tool

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+chapter-04-efficient-and-scalable-vision-transformers/results/figures/validation_accuracy.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,90 @@
+---
+language:
+  - en
+tags:
+  - book
+  - computer-vision
+  - transformers
+  - pytorch
+  - reproducibility
+---
+
+# Transformer Vision: From Image, Video to World Models & Robotics — Book
+
+This repository is the model-artifact companion to the technical book **Transformer Vision: From
+Image, Video to World Models & Robotics**. It keeps the book's trained weights, experiment
+configuration, evaluation tables, and usage notes together without pretending that every chapter
+uses one interchangeable model architecture.
+
+## Why the repository is organized by chapter
+
+Each chapter asks a different technical question and may use a different architecture, dataset,
+checkpoint format, preprocessing pipeline, or output type. Chapter-level folders create a stable
+boundary around those differences:
+
+```text
+chapter-NN-short-title/
+├── README.md                 # chapter-specific model card and usage
+├── configs/                  # resolved experiment configuration
+├── checkpoints/              # model, treatment, and seed subfolders
+├── results/                  # metrics, profiles, and figures
+├── EXPERIMENT_REPORT.md      # interpreted experiment narrative
+└── manifest.json             # machine-readable provenance
+```
+
+Inside a chapter, checkpoint folders are further separated by model, training treatment, and seed.
+This prevents a ViT checkpoint from being confused with a Swin, language, video, world-model, or
+robotics checkpoint from another part of the book.
+
+## Available chapters
+
+| Chapter | Artifact folder | Status |
+| --- | --- | --- |
+| 4 — Efficient and scalable vision transformers | [`chapter-04-efficient-and-scalable-vision-transformers`](chapter-04-efficient-and-scalable-vision-transformers) | Complete |
+
+Folders for other chapters will be added only after their experiments and validation are complete.
+Empty placeholder folders are intentionally avoided.
+
+## Download one chapter
+
+Using the Hugging Face CLI:
+
+```bash
+hf download \
+  Mayank022/Transformer-Vision-From-Image-Video-to-World-Models-and-Robotics-Book \
+  --include "chapter-04-efficient-and-scalable-vision-transformers/*" \
+  --local-dir transformer-vision-book
+```
+
+Readers who need only one seed can download a single file with `hf_hub_download`; each chapter card
+contains exact examples.
+
+## File-format policy
+
+Published inference weights use **Safetensors**. They contain the model state only and exclude
+optimizer, scheduler, gradient-scaler, and random-number-generator state. This makes the public
+artifact smaller and avoids pickle execution during weight loading. Full resumable training
+checkpoints remain in the experiment's controlled storage and are not required for inference.
+
+Every chapter includes SHA-256 checksums and a JSON manifest connecting each exported file to:
+
+- the exact upstream checkpoint identifier;
+- architecture and training mode;
+- random seed and selected epoch;
+- validation and test metrics;
+- configuration fingerprint; and
+- the checksum of the original training checkpoint.
+
+## Important scope note
+
+This is a **multi-model book archive**, not a single `from_pretrained()` repository. Always open the
+README inside the relevant chapter before loading a file. Model construction and preprocessing are
+chapter-specific.
+
+## License and attribution
+
+The repository contains fine-tuned derivatives of upstream research checkpoints. Each chapter lists
+the exact upstream model identifiers and papers. Before making a new public redistribution or a
+commercial derivative, review the applicable upstream model and dataset terms. The book's
+documentation does not replace those licenses.
+

chapter-04-efficient-and-scalable-vision-transformers/EXPERIMENT_REPORT.md

@@ -0,0 +1,661 @@
+# What actually wins under a fixed vision-transformer budget?
+
+## A controlled CIFAR-100 study of ViT, DeiT, Swin, EfficientFormer, and distillation
+
+This report records the completed experiment for the chapter project **Efficient and Scalable
+Vision Transformers**. The goal was not to reproduce a collection of headline numbers from model
+cards. It was to put several important architecture and training ideas into the same experimental
+frame: the same image-classification task, the same train/validation/test policy, the same
+fine-tuning budget, and the same inference hardware.
+
+The result is a practical answer rather than a universal leaderboard. On this task, a small plain
+ViT was already an excellent L4 throughput baseline; Swin's hierarchy did not make it faster;
+EfficientFormer dramatically reduced parameter count and MACs but gave up accuracy; and active
+hard distillation produced a small, repeatable accuracy gain at training time without changing
+deployment cost. Just as importantly, the benchmark showed why parameter count, theoretical
+operations, latency, and throughput must be measured separately.
+
+> **Experiment status:** complete. All 13 training runs reached 20 epochs, all 24 inference cases
+> completed, the report artifacts were generated, and all 13 validation-selected checkpoints were
+> downloaded and verified locally.
+
+---
+
+## Contents
+
+- [Executive summary](#executive-summary)
+- [1. Objective and research questions](#1-objective-and-research-questions)
+- [2. Models and why they were selected](#2-models-and-why-they-were-selected)
+- [3. Experimental design](#3-experimental-design)
+- [4. How the experiment was executed](#4-how-the-experiment-was-executed)
+- [5. Accuracy and training results](#5-accuracy-and-training-results)
+- [6. What did distillation buy?](#6-what-did-distillation-buy)
+- [7. Parameters and operation counts](#7-parameters-and-operation-counts)
+- [8. Inference benchmark](#8-inference-benchmark)
+- [9. Interpretation by model family](#9-interpretation-by-model-family)
+- [10. Decision guide](#10-decision-guide)
+- [11. Threats to validity and limitations](#11-threats-to-validity-and-limitations)
+- [12. Reproducibility record and artifact map](#12-reproducibility-record-and-artifact-map)
+- [13. Reproducing the experiment](#13-reproducing-the-experiment)
+- [14. Loading a downloaded checkpoint](#14-loading-a-downloaded-checkpoint)
+- [15. Final takeaway](#15-final-takeaway)
+- [16. Primary references](#16-primary-references)
+
+## Executive summary
+
+- **Best single-run test accuracy:** hard-distilled DeiT-S/16, seed 42, at **89.83%** top-1.
+- **Best multi-seed transformer result:** hard-distilled DeiT-S/16 at
+  **89.72% ± 0.11** top-1 across three seeds.
+- **Best single-seed reference row:** ConvNeXt-T at **89.78%**, although its one run does not
+  provide an uncertainty estimate.
+- **Controlled distillation result:** hard KD improved Distilled DeiT by **+0.09, +0.40, and
+  +0.30 percentage points** for seeds 42, 43, and 44. The mean paired gain was **+0.26 points**.
+- **Fastest FP16 batch-64 result on the NVIDIA L4:** ViT-S/16 at approximately
+  **1,995 images/s**. DeiT-S/16 and Distilled DeiT were effectively in the same throughput band.
+- **Smallest candidate:** EfficientFormer-L1 at **11.48M parameters** and **1.32 GMACs**. It
+  reached **86.83%** top-1 and about **1,961 FP16 images/s** at batch 64.
+- **Most expensive model in this training setup:** Swin-T, at about **60.5 minutes** per run and
+  **4.84 GB** peak CUDA allocator memory. It also had the lowest measured FP16 throughput.
+- **Most useful systems lesson:** MACs did not predict the runtime ordering. EfficientFormer used
+  roughly 69% fewer measured MACs than ViT, yet their FP16 batch-64 throughput was nearly the same
+  on this GPU.
+
+These findings are specific to this dataset, software stack, and NVIDIA L4. They are evidence for
+how to make a deployment decision, not proof that one architecture always dominates another.
+
+## 1. Objective and research questions
+
+The project asks one central question:
+
+> When data, fine-tuning budget, input resolution, evaluation rules, and hardware are held as
+> constant as practical, what do ViT, DeiT, Swin, and a compact efficient model trade in accuracy,
+> training behavior, model size, operation count, memory, latency, and throughput?
+
+That question is divided into six measurable subquestions:
+
+1. How competitive is a plain global-attention ViT after transfer learning on a modest dataset?
+2. Does active teacher distillation improve a distilled DeiT when architecture, seed, and data are
+   held fixed?
+3. Does Swin's windowed hierarchical design translate into lower memory use or higher throughput in
+   this implementation?
+4. How much accuracy does a compact EfficientFormer exchange for fewer parameters and MACs?
+5. Do MAC counts predict real latency and throughput?
+6. Which conclusions survive a multi-seed comparison, and which remain tentative single-seed
+   observations?
+
+The project fine-tunes pretrained models. It does **not** train these architectures from scratch.
+That choice matches the workflow most readers will use when compute and labeled data are limited.
+
+## 2. Models and why they were selected
+
+| Project alias | Exact `timm` checkpoint | Role in the chapter |
+| --- | --- | --- |
+| `vit` | `vit_small_patch16_224.augreg_in1k` | Plain global-attention baseline |
+| `deit` | `deit_small_patch16_224.fb_in1k` | Data-efficient ViT recipe |
+| `deit_distilled` | `deit_small_distilled_patch16_224.fb_in1k` | Controlled standard-versus-hard-KD comparison |
+| `swin` | `swin_tiny_patch4_window7_224.ms_in1k` | Shifted-window hierarchical transformer |
+| `efficientformer` | `efficientformer_l1.snap_dist_in1k` | Compact, latency-oriented candidate |
+| `convnext` | `convnext_tiny.fb_in1k` | CNN-style reference and frozen teacher |
+
+These are checkpoint-level comparisons: an architecture, its upstream training recipe, and its
+published weights arrive together in practice. For that reason, a broad ViT-versus-DeiT comparison
+cannot isolate one causal mechanism. The narrower Distilled DeiT experiment can: standard and
+hard-KD runs use the same student architecture, initialization family, split, and seeds; only the
+teacher loss changes.
+
+The model selection also separates **load-bearing ideas** from a larger paper map:
+
+- ViT supplies patch tokens and global self-attention.
+- DeiT shows that the training recipe and teacher supervision can matter as much as a new block.
+- Swin supplies local windows, shifted-window communication, patch merging, and a feature hierarchy.
+- EfficientFormer represents compact, latency-oriented hybrid design.
+- ConvNeXt provides a strong modern convolutional reference and a teacher whose inference graph is
+  not needed after student training.
+
+MobileViT, TinyViT, and MaxViT remain valuable related work, but adding every family would make this
+chapter project slower and less controlled. The selected set is large enough to expose the central
+trade-offs while remaining reproducible within a few hours of rented GPU time.
+
+## 3. Experimental design
+
+### 3.1 Dataset and split discipline
+
+The task is CIFAR-100 classification. CIFAR-100 contains 100 classes and 32×32 source images. The
+official 50,000-image training split was divided once, with split seed `2027`, into:
+
+- **45,000 training images:** 450 images per class;
+- **5,000 validation images:** 50 images per class; and
+- **10,000 test images:** the untouched official test split.
+
+Every run used exactly these indices. Checkpoint selection used validation accuracy only, with
+validation loss as the tie-breaker. The official test split was evaluated after training and was not
+used for early stopping or hyperparameter selection. This separation matters: repeatedly choosing
+settings from test accuracy would turn the test set into another validation set.
+
+All inputs were resized to 224×224 to match the pretrained checkpoints. Training augmentation was:
+
+1. random 32×32 crop with four pixels of reflected padding;
+2. random horizontal flip;
+3. RandAugment with `num_ops=2` and `magnitude=9`;
+4. bicubic resize to 224×224;
+5. checkpoint-native normalization; and
+6. random erasing with probability `0.25`.
+
+Validation and test preprocessing was deterministic: resize followed by the same checkpoint-native
+normalization. Resizing a 32×32 dataset to 224×224 is intentionally a transfer-learning exercise;
+it should not be confused with evidence from naturally high-resolution data.
+
+### 3.2 Run matrix
+
+The complete preset contains 13 training runs:
+
+| Model and mode | Seeds | Runs | Purpose |
+| --- | --- | ---: | --- |
+| ConvNeXt-T, standard | 42 | 1 | Teacher and CNN reference |
+| ViT-S/16, standard | 42 | 1 | Plain ViT baseline |
+| Swin-T, standard | 42 | 1 | Hierarchical/windowed baseline |
+| EfficientFormer-L1, standard | 42 | 1 | Compact candidate |
+| DeiT-S/16, standard | 42, 43, 44 | 3 | Multi-seed DeiT estimate |
+| Distilled DeiT-S/16, standard | 42, 43, 44 | 3 | No active teacher during fine-tuning |
+| Distilled DeiT-S/16, hard KD | 42, 43, 44 | 3 | Paired active-distillation treatment |
+
+Only the DeiT family received three seeds because the controlled distillation question is the main
+training ablation. The other model rows are useful engineering observations, but one seed cannot
+support strong claims about small accuracy differences.
+
+### 3.3 Fine-tuning recipe
+
+All runs used the `full` preset:
+
+| Setting | Value |
+| --- | --- |
+| Epochs | 20 |
+| Batch size | 64 |
+| Optimizer | AdamW |
+| Pretrained backbone learning rate | `5e-5` |
+| Classification-head learning rate | `5e-4` |
+| Weight decay | `0.05` |
+| Schedule | 2-epoch warmup, then cosine decay |
+| Minimum learning rate | `1e-6` |
+| Label smoothing | `0.1` |
+| Gradient clipping | `1.0` |
+| Automatic mixed precision | Enabled |
+| Input size | 224×224 |
+
+The 10× higher head learning rate lets the new 100-class classifier adapt more quickly while
+preserving useful pretrained features. Each checkpoint contains the model, optimizer, scheduler,
+random-number-generator, and data-generator state, so an interrupted run can resume without silently
+changing its planned data order.
+
+### 3.4 Hard distillation
+
+The ConvNeXt-T teacher was fine-tuned first and then frozen. During a hard-KD student run, the
+teacher's predicted class becomes a target for the distillation head. With `alpha = 0.5`, the loss is
+
+\[
+\mathcal{L} = (1-\alpha)\,\mathcal{L}_{\text{class}}
+             + \alpha\,\mathcal{L}_{\text{teacher}}.
+\]
+
+The teacher is in evaluation mode and receives no gradients. At student evaluation time, the class
+and distillation-head predictions are averaged. The teacher adds work only while training; it is not
+part of the deployed student.
+
+### 3.5 How to read the metrics
+
+| Metric | Meaning in this report |
+| --- | --- |
+| Top-1 accuracy | Percentage for which the highest-scoring class is correct |
+| Top-5 accuracy | Percentage for which the correct class appears among the five highest scores |
+| Best validation accuracy | Selection metric used to choose `best.pt`; not the final test score |
+| Parameters | Live parameter count after replacing the classifier with a 100-class head |
+| GMACs | Traced multiply-accumulate estimate for one 224×224 image |
+| Peak memory | Maximum PyTorch accelerator allocator usage inside the measured operation |
+| Batch latency | Synchronized time for one complete model forward pass at the stated batch size |
+| Images per second | `batch_size × 1000 / mean_latency_ms`; higher is better |
+
+Accuracy measures predictive quality; parameters approximate storage capacity; MACs approximate
+work; and latency and throughput measure the realized implementation. They answer different
+questions and should not be collapsed into one vague notion of “efficiency.”
+
+## 4. How the experiment was executed
+
+The final run used Modal with a persistent volume named
+`vision-transformer-benchmark-data`. The successful artifacts live remotely under
+`/artifacts/full` and have also been copied into this repository where noted below.
+
+| Provenance field | Recorded value |
+| --- | --- |
+| Completion date | 16 August 2026 |
+| Modal app ID | `ap-731KstAj4aSwjYegWlbGfo` |
+| State at handoff | Stopped; zero running tasks |
+| Training/benchmark accelerator | NVIDIA L4 |
+| Persistent volume | `vision-transformer-benchmark-data` |
+| Remote result root | `/artifacts/full` |
+| Local result root | `modal_full_results_20260816` |
+
+Execution was staged deliberately:
+
+1. Train the ConvNeXt teacher first.
+2. Launch the remaining student jobs with bounded concurrency of at most four NVIDIA L4
+   containers.
+3. Benchmark every architecture sequentially in one L4 container, so devices are not mixed within
+   the timing table.
+4. Generate CSV tables, figures, and the machine-generated report on a CPU container.
+
+The Modal functions commit the persistent volume after every checkpoint and automatically resume
+from the latest committed epoch after a retry or restart. This was important in practice because an
+earlier execution stopped between stages; the final invocation reused completed work rather than
+starting it again. The completed app has been stopped, so it is no longer consuming compute, while
+the volume retains the artifacts.
+
+The completed run is identified by configuration fingerprint:
+
+```text
+7b87d57ae5546600c460245a81431288db34c3e40a44c2e83da01acc4160d731
+```
+
+That fingerprint appears in the report and checkpoints and prevents results from an incompatible
+configuration from being combined accidentally.
+
+### 4.1 Compute accounting
+
+The recorded training durations sum to **27,362.9 seconds**, or approximately **7.60 L4
+GPU-hours**, across all 13 runs. Parallel student jobs reduced the successful workflow's elapsed
+wall-clock time to roughly **2 hours 4 minutes**; wall time is therefore not the same as total GPU
+consumption. Based on the recorded latency means, the configured inference warm-ups and timed
+forwards account for about another **18 minutes** of L4 execution, before model loading, MAC tracing,
+and container overhead.
+
+These are workload measurements, not a billing statement. Provider charges can also include
+container startup, downloads, CPU, memory, retries, and rounding, and the applicable unit price can
+change. For planning a repeat, budget approximately eight L4 GPU-hours plus setup and benchmark
+overhead, while expecting much less elapsed time when four training containers are available.
+
+## 5. Accuracy and training results
+
+### 5.1 Aggregate comparison
+
+| Model | Mode | Seeds | Test top-1 (%) | Best val top-1 (%) | Params (M) | Mean train time (min) | Peak train memory (MB) |
+| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| ConvNeXt-T | Standard | 1 | **89.78** | 89.86 | 27.90 | 46.49 | 4,245.67 |
+| ViT-S/16 | Standard | 1 | 89.22 | 89.60 | 21.70 | 27.89 | 2,563.02 |
+| Swin-T | Standard | 1 | 89.54 | 89.54 | 27.60 | 60.52 | 4,843.58 |
+| EfficientFormer-L1 | Standard | 1 | 86.83 | 86.86 | **11.48** | 28.35 | 2,633.86 |
+| DeiT-S/16 | Standard | 3 | 88.86 ± 0.03 | 89.72 | 21.70 | 27.94 | 2,563.02 |
+| Distilled DeiT-S/16 | Standard | 3 | 89.46 ± 0.27 | 90.17 | 21.74 | 28.36 | 2,566.76 |
+| Distilled DeiT-S/16 | Hard KD | 3 | **89.72 ± 0.11** | **90.27** | 21.74 | 41.29 | 3,051.61 |
+
+The `±` values are sample standard deviations across seeds. A blank uncertainty estimate for a
+single-seed row should not be read as zero variance.
+
+![Validation accuracy curves for all completed runs](results/figures/validation_accuracy.png)
+
+Several best checkpoints occurred at epoch 19 or 20, and only ConvNeXt selected substantially
+earlier, at epoch 17. The 20-epoch budget therefore appears close to the useful end of this preset,
+but the experiment does not prove that every model was fully saturated. Extending training would be
+a new experiment and should preserve the same selection discipline.
+
+### 5.2 Every completed run
+
+| Model | Mode | Seed | Best epoch | Best val top-1 | Test top-1 | Test top-5 |
+| --- | --- | ---: | ---: | ---: | ---: | ---: |
+| ConvNeXt-T | Standard | 42 | 17 | 89.86 | 89.78 | 98.86 |
+| ViT-S/16 | Standard | 42 | 20 | 89.60 | 89.22 | 98.32 |
+| Swin-T | Standard | 42 | 20 | 89.54 | 89.54 | 98.65 |
+| EfficientFormer-L1 | Standard | 42 | 20 | 86.86 | 86.83 | 98.26 |
+| DeiT-S/16 | Standard | 42 | 20 | 89.80 | 88.89 | 98.09 |
+| DeiT-S/16 | Standard | 43 | 20 | 89.70 | 88.86 | 98.22 |
+| DeiT-S/16 | Standard | 44 | 18 | 89.66 | 88.83 | 98.28 |
+| Distilled DeiT-S/16 | Standard | 42 | 19 | 90.04 | 89.74 | 98.25 |
+| Distilled DeiT-S/16 | Standard | 43 | 20 | 90.16 | 89.21 | 98.24 |
+| Distilled DeiT-S/16 | Standard | 44 | 19 | 90.30 | 89.42 | 98.31 |
+| Distilled DeiT-S/16 | Hard KD | 42 | 20 | 90.46 | **89.83** | 98.88 |
+| Distilled DeiT-S/16 | Hard KD | 43 | 20 | 90.06 | 89.61 | **98.94** |
+| Distilled DeiT-S/16 | Hard KD | 44 | 19 | 90.28 | 89.72 | 98.85 |
+
+The best individual result, 89.83%, came from hard-KD seed 42. ConvNeXt's 89.78% is the strongest
+single-seed reference row, while the hard-KD aggregate is the strongest multi-seed transformer
+result. Their 0.06-point difference is too small to treat as meaningful without matched repetitions.
+
+## 6. What did distillation buy?
+
+The most defensible causal result in the project is the paired standard-versus-hard-KD comparison:
+
+| Seed | Standard Distilled DeiT | Hard KD | Paired gain |
+| ---: | ---: | ---: | ---: |
+| 42 | 89.74 | 89.83 | +0.09 points |
+| 43 | 89.21 | 89.61 | +0.40 points |
+| 44 | 89.42 | 89.72 | +0.30 points |
+| **Mean** | **89.46** | **89.72** | **+0.26 points** |
+
+![Paired hard-distillation gains](results/figures/distillation_gain.png)
+
+All three paired differences are positive. The treatment also reduced the observed test-accuracy
+standard deviation from 0.27 to 0.11 points in these three runs. Three seeds are too few to claim a
+general variance reduction, but the direction is worth following up.
+
+The gain was not free during training:
+
+- mean training time rose from **28.36 to 41.29 minutes**, an increase of about **45.6%**;
+- peak CUDA allocator memory rose from roughly **2,566.76 to 3,051.61 MB**, about **18.9%**; and
+- inference cost stayed unchanged because both treatments deploy the same student architecture.
+
+This is the classic attraction of offline distillation: pay for a teacher during training, then
+retain only the student at inference. Whether +0.26 points is worth the added training cost depends
+on how often training happens and how valuable final accuracy is.
+
+The broader comparison between ordinary DeiT-S/16 (88.86%) and Distilled DeiT without an active
+teacher (89.46%) shows a +0.60-point difference, but it is **not** a clean estimate of distillation.
+Those rows use different upstream checkpoints and a different token/head structure. The paired
+hard-KD table is the appropriate evidence for the teacher's fine-tuning effect.
+
+## 7. Parameters and operation counts
+
+| Architecture | Task parameters (M) | Measured GMACs at 224×224 |
+| --- | ---: | ---: |
+| EfficientFormer-L1 | **11.48** | **1.317** |
+| ViT-S/16 | 21.70 | 4.250 |
+| DeiT-S/16 | 21.70 | 4.250 |
+| Distilled DeiT-S/16 | 21.74 | 4.272 |
+| ConvNeXt-T | 27.90 | 4.470 |
+| Swin-T | 27.60 | 4.509 |
+
+These are live counts after replacing the original classifier with a 100-class head. The operation
+counter uses `fvcore`'s MAC-style convention: one multiply-add is reported as one operation. It is
+better to call these **MACs**, not universal FLOPs. Some operators are not counted by the tracer;
+the exact unsupported-operator lists are preserved in
+[`model_profiles.csv`](results/model_profiles.csv).
+
+EfficientFormer used about 47% fewer parameters and 69% fewer measured MACs than ViT. Those are
+important advantages for storage, bandwidth, and some runtimes. They did not, however, produce a
+69% throughput improvement on the L4. Operation count describes computational work at one level;
+runtime also depends on kernel quality, memory traffic, parallelism, launch overhead, precision,
+and batch shape.
+
+## 8. Inference benchmark
+
+### 8.1 Measurement protocol
+
+All six architectures were benchmarked sequentially on one **NVIDIA L4** with:
+
+- PyTorch `2.9.1+cu128`, torchvision `0.24.1+cu128`, and `timm 1.0.28`;
+- FP32 and FP16;
+- batch sizes 1 and 64;
+- 50 warm-up iterations;
+- 200 timed iterations per repeat;
+- 5 repeats, or 1,000 timed samples per model/precision/batch case;
+- CUDA events plus synchronization; and
+- random input tensors already resident on the GPU.
+
+The timings include the model forward pass and exclude image decoding, preprocessing, data loading,
+host-to-device transfer, postprocessing, networking, and service overhead. Peak memory is PyTorch's
+CUDA allocator peak, not whole-system memory or energy use. All **24 of 24** configured benchmark
+cases completed.
+
+### 8.2 Main FP16 results
+
+| Model | Batch-1 latency (ms) | Batch-1 images/s | Batch-64 latency (ms) | Batch-64 images/s | Batch-64 peak MB |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| ConvNeXt-T | 5.377 | 185.98 | 57.764 | 1,107.96 | 728.76 |
+| ViT-S/16 | **4.694** | **213.04** | **32.072** | **1,995.49** | **320.05** |
+| Swin-T | 9.133 | 109.49 | 84.709 | 755.52 | 970.56 |
+| EfficientFormer-L1 | 6.209 | 161.06 | 32.634 | 1,961.13 | 354.11 |
+| DeiT-S/16 | 4.723 | 211.73 | 32.136 | 1,991.55 | **320.05** |
+| Distilled DeiT-S/16 | 4.723 | 211.73 | 32.342 | 1,978.86 | 321.56 |
+
+ViT and DeiT are effectively tied at batch 64: their reported means differ by less than 0.2%.
+EfficientFormer is also in the same broad FP16 throughput band despite having far fewer MACs. Swin
+is the slowest and has the largest inference allocator peak in this environment.
+
+![Accuracy versus FP16 batch-64 throughput](results/figures/accuracy_throughput.png)
+
+### 8.3 Precision changes the ranking
+
+| Model | FP32 b1 images/s | FP16 b1 images/s | FP32 b64 images/s | FP16 b64 images/s |
+| --- | ---: | ---: | ---: | ---: |
+| ConvNeXt-T | 209.30 | 185.98 | 500.35 | 1,107.96 |
+| ViT-S/16 | **233.61** | **213.04** | 598.39 | **1,995.49** |
+| Swin-T | 123.19 | 109.49 | 359.25 | 755.52 |
+| EfficientFormer-L1 | 173.24 | 161.06 | **1,128.16** | 1,961.13 |
+| DeiT-S/16 | 228.91 | 211.73 | 599.09 | 1,991.55 |
+| Distilled DeiT-S/16 | 228.13 | 211.73 | 607.31 | 1,978.86 |
+
+FP16 was slower than FP32 at batch 1 for every model. At such a small batch, conversion and kernel
+launch overhead can outweigh tensor-core savings. At batch 64, FP16 was substantially faster for
+every architecture. EfficientFormer led the FP32 batch-64 table, while ViT narrowly led the FP16
+table. A statement such as “model A is faster” is incomplete unless it names the hardware,
+precision, batch size, software stack, and timing boundary.
+
+The raw means, medians, p90 values, standard deviations, sample counts, and memory measurements are
+available in
+[`inference_benchmark.csv`](results/inference_benchmark.csv).
+
+## 9. Interpretation by model family
+
+### Plain ViT: stronger than its reputation under transfer learning
+
+ViT-S/16 reached 89.22% and delivered the best measured FP16 batch-64 throughput with low inference
+memory. This does not contradict the claim that plain ViT can be data hungry from scratch. It shows
+that a pretrained ViT can be a very practical small-data fine-tuning baseline. Global attention is
+not automatically too expensive at a 14×14 token grid.
+
+### DeiT: the recipe is a load-bearing idea
+
+Standard DeiT averaged 88.86% across three seeds, while the distilled checkpoint family did better.
+The central lesson is not that the DeiT block is universally superior to ViT—the architectures are
+close and the upstream checkpoints differ. The lesson is that augmentation, regularization,
+optimization, and teacher supervision deserve first-class experimental treatment.
+
+### Swin: hierarchy is useful, but it is not a free speedup
+
+Swin-T reached a competitive 89.54%, illustrating that windowed attention and hierarchical feature
+maps work. In this PyTorch/L4 setup, however, it trained longest, used the most peak memory, and had
+the lowest inference throughput. Window locality changes asymptotic attention behavior, but shifted
+windows, tensor layouts, and kernel implementations still determine wall-clock performance. Swin's
+hierarchy may be more valuable for detection and segmentation than this classification-only project
+can reveal.
+
+### EfficientFormer: compact does not mean universally fastest
+
+EfficientFormer-L1 was clearly the smallest model and had the fewest measured MACs. Its 86.83%
+accuracy was 2.39 points below ViT and 2.89 points below hard-KD Distilled DeiT. On the L4 it nearly
+matched ViT's FP16 batch-64 throughput, but did not beat it. The compact model may still be preferable
+when storage, download size, CPU/mobile execution, or energy dominates; this GPU experiment does not
+measure those targets. A real mobile recommendation requires benchmarking the exported model on the
+actual runtime and device.
+
+### ConvNeXt: a strong CNN remains a relevant baseline
+
+ConvNeXt-T reached 89.78%, the best single-seed architecture row, and served as an effective teacher.
+It was slower and larger than ViT/DeiT in the FP16 L4 benchmark. This is exactly why a CNN-style
+baseline belongs in the chapter: architectural novelty should be compared with a strong, modern
+convolutional model, not with an obsolete strawman.
+
+## 10. Decision guide
+
+For this exact task and L4 software stack:
+
+| Constraint | Sensible starting point | Reason |
+| --- | --- | --- |
+| Highest transformer accuracy | Hard-KD Distilled DeiT-S/16 | Best multi-seed transformer mean; no teacher at inference |
+| FP16 GPU throughput | ViT-S/16 or DeiT-S/16 | Approximately 2,000 images/s at batch 64 |
+| Lowest parameter/MAC budget | EfficientFormer-L1 | 11.48M parameters and 1.32 GMACs |
+| CNN-style accuracy reference | ConvNeXt-T | 89.78% in its single completed run |
+| Hierarchical features for downstream dense tasks | Swin-T, then benchmark again | Classification result alone does not value the full hierarchy |
+| Minimal training complexity | Plain ViT/DeiT fine-tuning | Avoids the teacher's training-time cost |
+
+This table is a starting point, not a deployment guarantee. If the target is a phone, browser, CPU,
+TensorRT service, or different accelerator, export the shortlisted models and rerun the benchmark on
+that exact stack.
+
+## 11. Threats to validity and limitations
+
+1. **Single-seed architecture rows.** ViT, Swin, EfficientFormer, and ConvNeXt were run once. Small
+   differences between those rows should not be interpreted as statistically settled rankings.
+2. **Checkpoint-level, not architecture-only, comparison.** The pretrained recipes and weights
+   differ. Only the paired standard-versus-hard-KD runs isolate the active teacher treatment.
+3. **Upsampled low-resolution data.** CIFAR-100 images begin at 32×32 and are resized to 224×224.
+   This is a controlled transfer task, not a substitute for naturally high-resolution benchmarks.
+4. **One GPU and software stack.** Kernel support can reorder models across hardware, framework
+   versions, compilers, and deployment runtimes.
+5. **No end-to-end serving measurement.** The inference benchmark excludes input pipelines, data
+   transfer, postprocessing, network latency, power, and energy.
+6. **No quantization or compiler optimization.** Results use eager PyTorch FP32/FP16, not INT8,
+   `torch.compile`, TensorRT, Core ML, TFLite, or vendor-specific graph optimization.
+7. **MAC tracing is incomplete by construction.** Unsupported operators are disclosed rather than
+   silently treated as proof of exact FLOPs.
+8. **Finite training budget.** Many runs selected epoch 19 or 20. Longer schedules could change the
+   result, but would require a fresh controlled run.
+9. **No dense prediction task.** Classification does not fully test the practical value of Swin's
+   multi-scale hierarchy.
+
+## 12. Reproducibility record and artifact map
+
+The completed local result bundle is
+[`results`](results). Important files are:
+
+| Artifact | Purpose |
+| --- | --- |
+| [`benchmark.json`](results/benchmark.json) | Complete hardware/software metadata and raw inference measurements |
+| [`run_results.csv`](results/run_results.csv) | One row for every training run |
+| [`epoch_metrics.csv`](results/epoch_metrics.csv) | Per-epoch learning curves and timing |
+| [`aggregate_results.csv`](results/aggregate_results.csv) | Grouped mean and standard deviation |
+| [`distillation_gains.csv`](results/distillation_gains.csv) | Seed-matched KD differences |
+| [`model_profiles.csv`](results/model_profiles.csv) | Parameters, MACs, and unsupported tracing operators |
+| [`inference_benchmark.csv`](results/inference_benchmark.csv) | All 24 timing and memory cases |
+| [`report.md`](results/report.md) | Automatically generated compact report |
+| [`checkpoints`](checkpoints) | Exported model-only Safetensors files |
+| [`SHA256SUMS`](checkpoints/SHA256SUMS) | Integrity manifest for all local checkpoints |
+
+The Hugging Face checkpoint directory contains 13 model-only
+`model.safetensors` files totaling approximately 1.06 GiB. The larger source
+`best.pt` files remain in controlled local and Modal storage because they also contain optimizer,
+scheduler, scaler, and random-state data used for exact training resume.
+
+To verify every downloaded model file:
+
+```bash
+cd checkpoints
+shasum -a 256 -c SHA256SUMS
+```
+
+Each line should end in `OK`. A mismatch means that file should be downloaded again before use.
+
+## 13. Reproducing the experiment
+
+From the project root, install and validate the environment as described in the README, then choose
+one of the following paths.
+
+### Run the complete workflow on Modal
+
+```bash
+uv run modal run modal_app.py --preset full --stage all
+```
+
+The teacher runs first, the students run with bounded concurrency, the inference suite uses one L4,
+and the report stage follows. Completed checkpoints are resumable, so rerunning after interruption
+does not intentionally discard finished epochs.
+
+Individual stages can also be run explicitly:
+
+```bash
+uv run modal run modal_app.py --preset full --stage train
+uv run modal run modal_app.py --preset full --stage benchmark
+uv run modal run modal_app.py --preset full --stage report
+```
+
+### Run on a local CUDA machine
+
+```bash
+uv run vision-bench all --preset full --device cuda
+```
+
+For a first instructional pass, use the shorter preset documented in the README before committing
+to the full multi-seed run.
+
+### Regenerate the report from existing artifacts
+
+```bash
+uv run vision-bench report --preset full
+```
+
+### Run the quality gates
+
+```bash
+uv run pytest -q
+uv run ruff check .
+uv run mypy src
+```
+
+At handoff, the project passed **41 tests with 1 hardware-dependent MPS test skipped**, and both
+Ruff and mypy completed cleanly.
+
+## 14. Loading a downloaded checkpoint
+
+The Hugging Face files contain model parameters only and use Safetensors. This example restores the
+validation-selected ViT on CPU after downloading the chapter folder:
+
+```python
+from pathlib import Path
+
+from safetensors.torch import load_file
+import timm
+
+weights_path = Path(
+    "checkpoints/vit-s16/standard/seed-42/model.safetensors"
+)
+state_dict = load_file(weights_path, device="cpu")
+
+model = timm.create_model(
+    "vit_small_patch16_224.augreg_in1k",
+    pretrained=False,
+    num_classes=100,
+)
+model.load_state_dict(state_dict, strict=True)
+model.eval()
+```
+
+For `deit-distilled-s16`, disable distilled-training output before ordinary evaluation if the
+installed `timm` model exposes the switch:
+
+```python
+if hasattr(model, "set_distilled_training"):
+    model.set_distilled_training(False)
+```
+
+Use `manifest.json` to map each published path to its exact `timm` identifier, training mode, seed,
+metrics, and source checksum. Use checkpoint-native preprocessing as shown in this chapter's
+README. These model-only exports cannot resume training because optimizer, scheduler, scaler, and
+random-number-generator state intentionally remain in the controlled source checkpoints.
+## 15. Final takeaway
+
+No single number answered this project. ViT and DeiT were excellent FP16 GPU throughput baselines;
+hard distillation bought a repeatable but modest accuracy gain; EfficientFormer minimized size and
+MACs without becoming universally faster; Swin's valuable hierarchy carried real implementation
+cost on this classifier; and ConvNeXt remained highly competitive. The practical method is the
+lasting result: select checkpoints without touching the test set, repeat the comparison that is
+meant to be causal, disclose uncertainty, profile live models, and benchmark the exact deployment
+shape instead of choosing from FLOPs alone.
+
+That is the chapter's main engineering lesson: efficient vision is not a property of an architecture
+name. It is an accuracy–training–memory–latency trade-off measured under a stated workload.
+
+## 16. Primary references
+
+- Dosovitskiy et al., [*An Image Is Worth 16×16 Words: Transformers for Image Recognition at
+  Scale*](https://arxiv.org/abs/2010.11929) — ViT.
+- Touvron et al., [*Training data-efficient image transformers & distillation through
+  attention*](https://arxiv.org/abs/2012.12877) — DeiT and the distillation token.
+- Liu et al., [*Swin Transformer: Hierarchical Vision Transformer using Shifted
+  Windows*](https://arxiv.org/abs/2103.14030) — window attention, shifts, and hierarchy.
+- Li et al., [*EfficientFormer: Vision Transformers at MobileNet
+  Speed*](https://arxiv.org/abs/2206.01191) — compact, latency-oriented design.
+- Liu et al., [*A ConvNet for the 2020s*](https://arxiv.org/abs/2201.03545) — ConvNeXt.
+- Mehta and Rastegari, [*MobileViT*](https://arxiv.org/abs/2110.02178); Wu et al.,
+  [*TinyViT*](https://arxiv.org/abs/2207.10666); and Tu et al.,
+  [*MaxViT*](https://arxiv.org/abs/2204.01697) — practical variants for the chapter's paper map.
+
+The exact executable checkpoint identifiers are pinned in
+[the chapter model table](README.md#exact-upstream-model-identifiers). Architecture names alone are not enough
+to reproduce a pretrained-checkpoint comparison.

chapter-04-efficient-and-scalable-vision-transformers/README.md

@@ -0,0 +1,184 @@
+# Chapter 4 — Efficient and scalable vision transformers
+
+## ViT, DeiT, Swin, EfficientFormer, and hard distillation on CIFAR-100
+
+This folder contains the validation-selected inference weights and measured artifacts for Chapter
+4's controlled image-classification project. Six architectures were fine-tuned from pinned `timm`
+checkpoints on the same fixed CIFAR-100 split. The experiment measured accuracy, training behavior,
+parameters, MAC-style operation counts, accelerator memory, latency, and throughput.
+
+The published `.safetensors` files contain **model parameters only**. They are derived from the
+verified `best.pt` files selected by validation top-1 accuracy; optimizer and resume state are not
+included.
+
+## Experiment summary
+
+- Dataset: CIFAR-100
+- Split: 45,000 train / 5,000 validation / 10,000 official test
+- Input: 224×224
+- Training: 20 epochs, AdamW, mixed precision, batch size 64
+- Hardware: NVIDIA L4
+- Configuration fingerprint:
+  `7b87d57ae5546600c460245a81431288db34c3e40a44c2e83da01acc4160d731`
+- Completed training runs: 13
+- Completed inference cases: 24
+
+## Accuracy results
+
+| Model | Mode | Seeds | Test top-1 (%) | Published checkpoint folder |
+| --- | --- | --- | ---: | --- |
+| ConvNeXt-T | Standard | 42 | 89.78 | `checkpoints/convnext-t/standard/seed-42` |
+| ViT-S/16 | Standard | 42 | 89.22 | `checkpoints/vit-s16/standard/seed-42` |
+| Swin-T | Standard | 42 | 89.54 | `checkpoints/swin-t/standard/seed-42` |
+| EfficientFormer-L1 | Standard | 42 | 86.83 | `checkpoints/efficientformer-l1/standard/seed-42` |
+| DeiT-S/16 | Standard | 42, 43, 44 | 88.86 ± 0.03 | `checkpoints/deit-s16/standard/seed-*` |
+| Distilled DeiT-S/16 | Standard | 42, 43, 44 | 89.46 ± 0.27 | `checkpoints/deit-distilled-s16/standard/seed-*` |
+| Distilled DeiT-S/16 | Hard KD | 42, 43, 44 | **89.72 ± 0.11** | `checkpoints/deit-distilled-s16/hard-kd/seed-*` |
+
+The `±` value is the sample standard deviation across seeds. Single-seed rows do not have an
+uncertainty estimate.
+
+## Exact upstream model identifiers
+
+| Export folder | `timm` model identifier |
+| --- | --- |
+| `convnext-t` | `convnext_tiny.fb_in1k` |
+| `vit-s16` | `vit_small_patch16_224.augreg_in1k` |
+| `swin-t` | `swin_tiny_patch4_window7_224.ms_in1k` |
+| `efficientformer-l1` | `efficientformer_l1.snap_dist_in1k` |
+| `deit-s16` | `deit_small_patch16_224.fb_in1k` |
+| `deit-distilled-s16` | `deit_small_distilled_patch16_224.fb_in1k` |
+
+All exported models have a 100-class task head. Instantiate with `pretrained=False`; loading the
+published state replaces every parameter.
+
+## Download and load one model
+
+Install the runtime packages:
+
+```bash
+python -m pip install huggingface_hub safetensors timm torch
+```
+
+Then download and restore the ViT seed-42 model:
+
+```python
+from huggingface_hub import hf_hub_download
+from safetensors.torch import load_file
+import timm
+
+repo_id = (
+    "Mayank022/"
+    "Transformer-Vision-From-Image-Video-to-World-Models-and-Robotics-Book"
+)
+filename = (
+    "chapter-04-efficient-and-scalable-vision-transformers/"
+    "checkpoints/vit-s16/standard/seed-42/model.safetensors"
+)
+
+weights_path = hf_hub_download(repo_id=repo_id, filename=filename)
+state_dict = load_file(weights_path, device="cpu")
+
+model = timm.create_model(
+    "vit_small_patch16_224.augreg_in1k",
+    pretrained=False,
+    num_classes=100,
+)
+model.load_state_dict(state_dict, strict=True)
+model.eval()
+```
+
+For Distilled DeiT evaluation, disable training-time tuple output when supported:
+
+```python
+if hasattr(model, "set_distilled_training"):
+    model.set_distilled_training(False)
+```
+
+## Preprocessing
+
+Use checkpoint-native `timm` preprocessing. For a PIL image:
+
+```python
+from timm.data import create_transform, resolve_model_data_config
+
+data_config = resolve_model_data_config(model)
+transform = create_transform(**data_config, is_training=False)
+input_tensor = transform(pil_image).unsqueeze(0)
+```
+
+The classifier outputs CIFAR-100 class indices. Class labels and the fixed split are defined by the
+chapter project; do not interpret the output as the original ImageNet-1K label space.
+
+## Hard-distillation result
+
+Hard KD held the Distilled DeiT architecture and seed fixed while adding a frozen, fine-tuned
+ConvNeXt teacher during training. Test top-1 changed by:
+
+| Seed | Standard | Hard KD | Gain |
+| ---: | ---: | ---: | ---: |
+| 42 | 89.74 | 89.83 | +0.09 points |
+| 43 | 89.21 | 89.61 | +0.40 points |
+| 44 | 89.42 | 89.72 | +0.30 points |
+
+The mean paired gain was +0.26 percentage points. The teacher is not required when loading or
+deploying a hard-KD student.
+
+## Artifact map
+
+```text
+chapter-04-efficient-and-scalable-vision-transformers/
+├── README.md
+├── EXPERIMENT_REPORT.md
+├── manifest.json
+├── configs/
+│   └── full.yaml
+├── checkpoints/
+│   ├── SHA256SUMS
+│   ├── convnext-t/standard/seed-42/model.safetensors
+│   ├── vit-s16/standard/seed-42/model.safetensors
+│   ├── swin-t/standard/seed-42/model.safetensors
+│   ├── efficientformer-l1/standard/seed-42/model.safetensors
+│   ├── deit-s16/standard/seed-{42,43,44}/model.safetensors
+│   └── deit-distilled-s16/{standard,hard-kd}/seed-{42,43,44}/model.safetensors
+└── results/
+    ├── benchmark.json
+    ├── aggregate_results.csv
+    ├── run_results.csv
+    ├── inference_benchmark.csv
+    └── figures/
+```
+
+`manifest.json` is the authoritative model-to-file mapping and includes per-run metrics and both
+source and exported checksums.
+
+## Verify downloaded weights
+
+From the chapter folder:
+
+```bash
+cd checkpoints
+shasum -a 256 -c SHA256SUMS
+```
+
+Every line should end in `OK`.
+
+## Interpretation and limitations
+
+The main finding is that theoretical efficiency and measured runtime are different quantities.
+EfficientFormer used the fewest parameters and MACs, while ViT/DeiT achieved the highest FP16
+batch-64 throughput on the L4. Swin was competitive in accuracy but slower in this implementation.
+Hard distillation improved all three paired seeds at extra training cost and no additional inference
+cost.
+
+These results are not a universal leaderboard. Most broad architecture rows use one seed; upstream
+pretraining recipes differ; CIFAR-100 images are enlarged from 32×32 to 224×224; inference excludes
+data loading; and the timing ranking is specific to the recorded hardware and software stack. See
+`EXPERIMENT_REPORT.md` for the complete analysis.
+
+## Upstream references and terms
+
+The artifacts are fine-tuned derivatives of the exact `timm` checkpoints listed above. Review their
+upstream model cards and terms, along with CIFAR-100's terms, before redistribution or commercial
+use. Publishing a derived weight file does not remove upstream attribution or license obligations.
+

chapter-04-efficient-and-scalable-vision-transformers/checkpoints/SHA256SUMS

@@ -0,0 +1,13 @@
+1cf9f71e4a426b11d2608aef603ac36614d51cf21f63d602ebaba3844f55d1af  deit-distilled-s16/hard-kd/seed-44/model.safetensors
+4365b9b0c25a3aa13be8574df0e76c3b01cbe3cd106bf96f956c4434e2f3990f  swin-t/standard/seed-42/model.safetensors
+466722cd32fcba95055285d9d6eeb28fb18945f83dea71af1a25633a6a7217d6  deit-s16/standard/seed-43/model.safetensors
+4cb13bd3f8f0f64eb48e083dd9fc91b9bdccc5ae53861f943632f746cb5c8c39  deit-distilled-s16/standard/seed-44/model.safetensors
+4e6cc980bc70a8e108bcc1e8985a2034c14fb296a7b506e2b06b05b6524b7660  deit-s16/standard/seed-44/model.safetensors
+5773caa4abdaa282f6c7ab22a8385458aa91cf44b7b78e0137e2099c30860449  deit-distilled-s16/hard-kd/seed-43/model.safetensors
+8e3142f30a411bf73c94e72ff390be01b80d1aa3de999ae44099007306292ae5  vit-s16/standard/seed-42/model.safetensors
+91e245848a91b2775f9107713dab970cb5b316c3adeb6316e7d771940ec0ee6b  deit-s16/standard/seed-42/model.safetensors
+95bc596579deb6d376175bb08580ca3fb1492f959d6b45f50ae73eed5c026c2e  deit-distilled-s16/standard/seed-43/model.safetensors
+b133bd3de04743660ee7fb349252e145b0076d9808ee96471a93885f6f790a34  deit-distilled-s16/standard/seed-42/model.safetensors
+c9afe1fab4646cfa404c017bac68b9fe322e38a60838c01211015de8d45f2b32  efficientformer-l1/standard/seed-42/model.safetensors
+d2c1110fa7b6cd4c225cc27bf23fdeeeeaec120ee451a9bf511dcdb716a63b8e  deit-distilled-s16/hard-kd/seed-42/model.safetensors
+d61158f6d560081ee2f12d47de5dbd2e9d4efbf0deffcc4eab653c93d4833180  convnext-t/standard/seed-42/model.safetensors

chapter-04-efficient-and-scalable-vision-transformers/checkpoints/convnext-t/standard/seed-42/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d61158f6d560081ee2f12d47de5dbd2e9d4efbf0deffcc4eab653c93d4833180
+size 111606200

chapter-04-efficient-and-scalable-vision-transformers/checkpoints/deit-distilled-s16/hard-kd/seed-42/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d2c1110fa7b6cd4c225cc27bf23fdeeeeaec120ee451a9bf511dcdb716a63b8e
+size 86988248

chapter-04-efficient-and-scalable-vision-transformers/checkpoints/deit-distilled-s16/hard-kd/seed-43/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5773caa4abdaa282f6c7ab22a8385458aa91cf44b7b78e0137e2099c30860449
+size 86988248

chapter-04-efficient-and-scalable-vision-transformers/checkpoints/deit-distilled-s16/hard-kd/seed-44/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1cf9f71e4a426b11d2608aef603ac36614d51cf21f63d602ebaba3844f55d1af
+size 86988248

chapter-04-efficient-and-scalable-vision-transformers/checkpoints/deit-distilled-s16/standard/seed-42/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b133bd3de04743660ee7fb349252e145b0076d9808ee96471a93885f6f790a34
+size 86988248

chapter-04-efficient-and-scalable-vision-transformers/checkpoints/deit-distilled-s16/standard/seed-43/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:95bc596579deb6d376175bb08580ca3fb1492f959d6b45f50ae73eed5c026c2e
+size 86988248

chapter-04-efficient-and-scalable-vision-transformers/checkpoints/deit-distilled-s16/standard/seed-44/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4cb13bd3f8f0f64eb48e083dd9fc91b9bdccc5ae53861f943632f746cb5c8c39
+size 86988248

chapter-04-efficient-and-scalable-vision-transformers/checkpoints/deit-s16/standard/seed-42/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:91e245848a91b2775f9107713dab970cb5b316c3adeb6316e7d771940ec0ee6b
+size 86830888

chapter-04-efficient-and-scalable-vision-transformers/checkpoints/deit-s16/standard/seed-43/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:466722cd32fcba95055285d9d6eeb28fb18945f83dea71af1a25633a6a7217d6
+size 86830888

chapter-04-efficient-and-scalable-vision-transformers/checkpoints/deit-s16/standard/seed-44/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4e6cc980bc70a8e108bcc1e8985a2034c14fb296a7b506e2b06b05b6524b7660
+size 86830888

chapter-04-efficient-and-scalable-vision-transformers/checkpoints/efficientformer-l1/standard/seed-42/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c9afe1fab4646cfa404c017bac68b9fe322e38a60838c01211015de8d45f2b32
+size 46102432

chapter-04-efficient-and-scalable-vision-transformers/checkpoints/swin-t/standard/seed-42/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4365b9b0c25a3aa13be8574df0e76c3b01cbe3cd106bf96f956c4434e2f3990f
+size 110402760

chapter-04-efficient-and-scalable-vision-transformers/checkpoints/vit-s16/standard/seed-42/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8e3142f30a411bf73c94e72ff390be01b80d1aa3de999ae44099007306292ae5
+size 86830888

chapter-04-efficient-and-scalable-vision-transformers/configs/full.yaml

@@ -0,0 +1,53 @@
+name: full
+tutorial_only: false
+description: Complete chapter benchmark used for reportable results.
+pretrained: true
+data:
+  root: data
+  split_seed: 2027
+  train_per_class: 450
+  val_per_class: 50
+  test_limit: null
+  input_size: 224
+  num_workers: 8
+training:
+  epochs: 20
+  batch_size: 64
+  backbone_lr: 0.00005
+  head_lr: 0.0005
+  weight_decay: 0.05
+  warmup_epochs: 2
+  min_lr: 0.000001
+  label_smoothing: 0.1
+  gradient_clip: 1.0
+  amp: true
+  random_erasing: 0.25
+benchmark:
+  precisions: [fp32, fp16]
+  batch_sizes: [1, 64]
+  warmup_iterations: 50
+  timed_iterations: 200
+  repeats: 5
+runs:
+  - model: convnext
+    mode: standard
+    seeds: [42]
+  - model: vit
+    mode: standard
+    seeds: [42]
+  - model: swin
+    mode: standard
+    seeds: [42]
+  - model: efficientformer
+    mode: standard
+    seeds: [42]
+  - model: deit
+    mode: standard
+    seeds: [42, 43, 44]
+  - model: deit_distilled
+    mode: standard
+    seeds: [42, 43, 44]
+  - model: deit_distilled
+    mode: hard_kd
+    seeds: [42, 43, 44]
+

chapter-04-efficient-and-scalable-vision-transformers/manifest.json

@@ -0,0 +1,330 @@
+{
+  "book_title": "Transformer Vision: From Image, Video to World Models & Robotics",
+  "chapter": 4,
+  "chapter_directory": "chapter-04-efficient-and-scalable-vision-transformers",
+  "chapter_title": "Efficient and scalable vision transformers",
+  "checkpoint_count": 13,
+  "config_fingerprint": "7b87d57ae5546600c460245a81431288db34c3e40a44c2e83da01acc4160d731",
+  "created_at_utc": "2026-08-16T13:43:34.633218+00:00",
+  "dataset": "CIFAR-100",
+  "format": "model-only Safetensors state dictionaries",
+  "models": [
+    {
+      "best_epoch": 17,
+      "best_validation_top1": 89.86,
+      "display_name": "ConvNeXt-T",
+      "exported_bytes": 111606200,
+      "exported_sha256": "d61158f6d560081ee2f12d47de5dbd2e9d4efbf0deffcc4eab653c93d4833180",
+      "family": "modern convolutional network",
+      "input_size": 224,
+      "model": "convnext",
+      "model_file": "chapter-04-efficient-and-scalable-vision-transformers/checkpoints/convnext-t/standard/seed-42/model.safetensors",
+      "num_classes": 100,
+      "parameters": 27897028,
+      "run_key": "convnext-standard-seed42",
+      "seed": 42,
+      "source_checkpoint": "convnext-standard-seed42-7b87d57ae5/best.pt",
+      "source_checkpoint_bytes": 335007241,
+      "source_checkpoint_sha256": "ab2e8c6b9fd0fd8108c59c1f10996a2dd2107b5a2276fafa0f375275207553a0",
+      "test_top1": 89.78,
+      "test_top5": 98.86,
+      "timm_checkpoint": "convnext_tiny.fb_in1k",
+      "training_mode": "standard",
+      "training_peak_memory_mb": 4245.66796875,
+      "training_seconds": 2789.4574548180008
+    },
+    {
+      "best_epoch": 20,
+      "best_validation_top1": 89.6,
+      "display_name": "ViT-S/16",
+      "exported_bytes": 86830888,
+      "exported_sha256": "8e3142f30a411bf73c94e72ff390be01b80d1aa3de999ae44099007306292ae5",
+      "family": "plain global-attention transformer",
+      "input_size": 224,
+      "model": "vit",
+      "model_file": "chapter-04-efficient-and-scalable-vision-transformers/checkpoints/vit-s16/standard/seed-42/model.safetensors",
+      "num_classes": 100,
+      "parameters": 21704164,
+      "run_key": "vit-standard-seed42",
+      "seed": 42,
+      "source_checkpoint": "vit-standard-seed42-7b87d57ae5/best.pt",
+      "source_checkpoint_bytes": 260658233,
+      "source_checkpoint_sha256": "b174a80bd73b1947aa9b64e9561285ed90e934f0960eedb8460c85b69546df26",
+      "test_top1": 89.22,
+      "test_top5": 98.32,
+      "timm_checkpoint": "vit_small_patch16_224.augreg_in1k",
+      "training_mode": "standard",
+      "training_peak_memory_mb": 2563.0244140625,
+      "training_seconds": 1673.357545405
+    },
+    {
+      "best_epoch": 20,
+      "best_validation_top1": 89.54,
+      "display_name": "Swin-T",
+      "exported_bytes": 110402760,
+      "exported_sha256": "4365b9b0c25a3aa13be8574df0e76c3b01cbe3cd106bf96f956c4434e2f3990f",
+      "family": "hierarchical shifted-window transformer",
+      "input_size": 224,
+      "model": "swin",
+      "model_file": "chapter-04-efficient-and-scalable-vision-transformers/checkpoints/swin-t/standard/seed-42/model.safetensors",
+      "num_classes": 100,
+      "parameters": 27596254,
+      "run_key": "swin-standard-seed42",
+      "seed": 42,
+      "source_checkpoint": "swin-standard-seed42-7b87d57ae5/best.pt",
+      "source_checkpoint_bytes": 331388497,
+      "source_checkpoint_sha256": "102610136420e467186b860b34ae68a420598e417da1dc4049ec8e5ced8ba76b",
+      "test_top1": 89.54,
+      "test_top5": 98.65,
+      "timm_checkpoint": "swin_tiny_patch4_window7_224.ms_in1k",
+      "training_mode": "standard",
+      "training_peak_memory_mb": 4843.58154296875,
+      "training_seconds": 3631.1184958180006
+    },
+    {
+      "best_epoch": 20,
+      "best_validation_top1": 86.86,
+      "display_name": "EfficientFormer-L1",
+      "exported_bytes": 46102432,
+      "exported_sha256": "c9afe1fab4646cfa404c017bac68b9fe322e38a60838c01211015de8d45f2b32",
+      "family": "latency-oriented compact transformer",
+      "input_size": 224,
+      "model": "efficientformer",
+      "model_file": "chapter-04-efficient-and-scalable-vision-transformers/checkpoints/efficientformer-l1/standard/seed-42/model.safetensors",
+      "num_classes": 100,
+      "parameters": 11481728,
+      "run_key": "efficientformer-standard-seed42",
+      "seed": 42,
+      "source_checkpoint": "efficientformer-standard-seed42-7b87d57ae5/best.pt",
+      "source_checkpoint_bytes": 138203465,
+      "source_checkpoint_sha256": "b92f254262b5f1698330ac93cfa179a1174fc2d7f7faf1fc8c110f899ca2b4fb",
+      "test_top1": 86.83,
+      "test_top5": 98.26,
+      "timm_checkpoint": "efficientformer_l1.snap_dist_in1k",
+      "training_mode": "standard",
+      "training_peak_memory_mb": 2633.85888671875,
+      "training_seconds": 1701.0025390570008
+    },
+    {
+      "best_epoch": 20,
+      "best_validation_top1": 89.8,
+      "display_name": "DeiT-S/16",
+      "exported_bytes": 86830888,
+      "exported_sha256": "91e245848a91b2775f9107713dab970cb5b316c3adeb6316e7d771940ec0ee6b",
+      "family": "data-efficient ViT recipe",
+      "input_size": 224,
+      "model": "deit",
+      "model_file": "chapter-04-efficient-and-scalable-vision-transformers/checkpoints/deit-s16/standard/seed-42/model.safetensors",
+      "num_classes": 100,
+      "parameters": 21704164,
+      "run_key": "deit-standard-seed42",
+      "seed": 42,
+      "source_checkpoint": "deit-standard-seed42-7b87d57ae5/best.pt",
+      "source_checkpoint_bytes": 260658233,
+      "source_checkpoint_sha256": "9f4858d4f748ae7c7f4716f51dbeae3f0b7b20d39510c24c0eb63f1f2f25619f",
+      "test_top1": 88.89,
+      "test_top5": 98.09,
+      "timm_checkpoint": "deit_small_patch16_224.fb_in1k",
+      "training_mode": "standard",
+      "training_peak_memory_mb": 2563.0244140625,
+      "training_seconds": 1639.156589266
+    },
+    {
+      "best_epoch": 20,
+      "best_validation_top1": 89.7,
+      "display_name": "DeiT-S/16",
+      "exported_bytes": 86830888,
+      "exported_sha256": "466722cd32fcba95055285d9d6eeb28fb18945f83dea71af1a25633a6a7217d6",
+      "family": "data-efficient ViT recipe",
+      "input_size": 224,
+      "model": "deit",
+      "model_file": "chapter-04-efficient-and-scalable-vision-transformers/checkpoints/deit-s16/standard/seed-43/model.safetensors",
+      "num_classes": 100,
+      "parameters": 21704164,
+      "run_key": "deit-standard-seed43",
+      "seed": 43,
+      "source_checkpoint": "deit-standard-seed43-7b87d57ae5/best.pt",
+      "source_checkpoint_bytes": 260658233,
+      "source_checkpoint_sha256": "6773ec182869a69c0e4c9508d6630442376eb5541a6c2a052912d6307a24caf5",
+      "test_top1": 88.86,
+      "test_top5": 98.22,
+      "timm_checkpoint": "deit_small_patch16_224.fb_in1k",
+      "training_mode": "standard",
+      "training_peak_memory_mb": 2563.0244140625,
+      "training_seconds": 1688.338380226
+    },
+    {
+      "best_epoch": 18,
+      "best_validation_top1": 89.66,
+      "display_name": "DeiT-S/16",
+      "exported_bytes": 86830888,
+      "exported_sha256": "4e6cc980bc70a8e108bcc1e8985a2034c14fb296a7b506e2b06b05b6524b7660",
+      "family": "data-efficient ViT recipe",
+      "input_size": 224,
+      "model": "deit",
+      "model_file": "chapter-04-efficient-and-scalable-vision-transformers/checkpoints/deit-s16/standard/seed-44/model.safetensors",
+      "num_classes": 100,
+      "parameters": 21704164,
+      "run_key": "deit-standard-seed44",
+      "seed": 44,
+      "source_checkpoint": "deit-standard-seed44-7b87d57ae5/best.pt",
+      "source_checkpoint_bytes": 260658233,
+      "source_checkpoint_sha256": "d00aeae136f35b4d8906cace4bea2a24c3c92aa2b53dbff348cdb757377226c2",
+      "test_top1": 88.83,
+      "test_top5": 98.28,
+      "timm_checkpoint": "deit_small_patch16_224.fb_in1k",
+      "training_mode": "standard",
+      "training_peak_memory_mb": 2563.0244140625,
+      "training_seconds": 1702.554159402
+    },
+    {
+      "best_epoch": 19,
+      "best_validation_top1": 90.04,
+      "display_name": "Distilled DeiT-S/16",
+      "exported_bytes": 86988248,
+      "exported_sha256": "b133bd3de04743660ee7fb349252e145b0076d9808ee96471a93885f6f790a34",
+      "family": "DeiT with class and distillation tokens",
+      "input_size": 224,
+      "model": "deit_distilled",
+      "model_file": "chapter-04-efficient-and-scalable-vision-transformers/checkpoints/deit-distilled-s16/standard/seed-42/model.safetensors",
+      "num_classes": 100,
+      "parameters": 21743432,
+      "run_key": "deit_distilled-standard-seed42",
+      "seed": 42,
+      "source_checkpoint": "deit_distilled-standard-seed42-7b87d57ae5/best.pt",
+      "source_checkpoint_bytes": 261132897,
+      "source_checkpoint_sha256": "0b428a07b3d3ba7f5cc351434045a304c4809654f81521b615f7ca2887f74fb4",
+      "test_top1": 89.74,
+      "test_top5": 98.25,
+      "timm_checkpoint": "deit_small_distilled_patch16_224.fb_in1k",
+      "training_mode": "standard",
+      "training_peak_memory_mb": 2594.0869140625,
+      "training_seconds": 1680.1939577700005
+    },
+    {
+      "best_epoch": 20,
+      "best_validation_top1": 90.16,
+      "display_name": "Distilled DeiT-S/16",
+      "exported_bytes": 86988248,
+      "exported_sha256": "95bc596579deb6d376175bb08580ca3fb1492f959d6b45f50ae73eed5c026c2e",
+      "family": "DeiT with class and distillation tokens",
+      "input_size": 224,
+      "model": "deit_distilled",
+      "model_file": "chapter-04-efficient-and-scalable-vision-transformers/checkpoints/deit-distilled-s16/standard/seed-43/model.safetensors",
+      "num_classes": 100,
+      "parameters": 21743432,
+      "run_key": "deit_distilled-standard-seed43",
+      "seed": 43,
+      "source_checkpoint": "deit_distilled-standard-seed43-7b87d57ae5/best.pt",
+      "source_checkpoint_bytes": 261132961,
+      "source_checkpoint_sha256": "87730512bd0d2a8fd29fa6c5bf166d9d0198486dee44e66cbfacd6db51a05fac",
+      "test_top1": 89.21,
+      "test_top5": 98.24,
+      "timm_checkpoint": "deit_small_distilled_patch16_224.fb_in1k",
+      "training_mode": "standard",
+      "training_peak_memory_mb": 2594.0869140625,
+      "training_seconds": 1683.276820501
+    },
+    {
+      "best_epoch": 19,
+      "best_validation_top1": 90.3,
+      "display_name": "Distilled DeiT-S/16",
+      "exported_bytes": 86988248,
+      "exported_sha256": "4cb13bd3f8f0f64eb48e083dd9fc91b9bdccc5ae53861f943632f746cb5c8c39",
+      "family": "DeiT with class and distillation tokens",
+      "input_size": 224,
+      "model": "deit_distilled",
+      "model_file": "chapter-04-efficient-and-scalable-vision-transformers/checkpoints/deit-distilled-s16/standard/seed-44/model.safetensors",
+      "num_classes": 100,
+      "parameters": 21743432,
+      "run_key": "deit_distilled-standard-seed44",
+      "seed": 44,
+      "source_checkpoint": "deit_distilled-standard-seed44-7b87d57ae5/best.pt",
+      "source_checkpoint_bytes": 261131233,
+      "source_checkpoint_sha256": "377f506858be6d24e238941b0bafb171bb0f38508ff10042f2f5aac58ecdccdc",
+      "test_top1": 89.42,
+      "test_top5": 98.31,
+      "timm_checkpoint": "deit_small_distilled_patch16_224.fb_in1k",
+      "training_mode": "standard",
+      "training_peak_memory_mb": 2512.09765625,
+      "training_seconds": 1741.7318086770006
+    },
+    {
+      "best_epoch": 20,
+      "best_validation_top1": 90.46,
+      "display_name": "Distilled DeiT-S/16",
+      "exported_bytes": 86988248,
+      "exported_sha256": "d2c1110fa7b6cd4c225cc27bf23fdeeeeaec120ee451a9bf511dcdb716a63b8e",
+      "family": "DeiT with class and distillation tokens",
+      "input_size": 224,
+      "model": "deit_distilled",
+      "model_file": "chapter-04-efficient-and-scalable-vision-transformers/checkpoints/deit-distilled-s16/hard-kd/seed-42/model.safetensors",
+      "num_classes": 100,
+      "parameters": 21743432,
+      "run_key": "deit_distilled-hard_kd-seed42",
+      "seed": 42,
+      "source_checkpoint": "deit_distilled-hard_kd-seed42-7b87d57ae5/best.pt",
+      "source_checkpoint_bytes": 261131169,
+      "source_checkpoint_sha256": "7da1b2f363c0eb39de39c8c2d3fbb13b1286dbe9565267f50b79a1cccc52c925",
+      "test_top1": 89.83,
+      "test_top5": 98.88,
+      "timm_checkpoint": "deit_small_distilled_patch16_224.fb_in1k",
+      "training_mode": "hard_kd",
+      "training_peak_memory_mb": 3051.60888671875,
+      "training_seconds": 2489.6353192010006
+    },
+    {
+      "best_epoch": 20,
+      "best_validation_top1": 90.06,
+      "display_name": "Distilled DeiT-S/16",
+      "exported_bytes": 86988248,
+      "exported_sha256": "5773caa4abdaa282f6c7ab22a8385458aa91cf44b7b78e0137e2099c30860449",
+      "family": "DeiT with class and distillation tokens",
+      "input_size": 224,
+      "model": "deit_distilled",
+      "model_file": "chapter-04-efficient-and-scalable-vision-transformers/checkpoints/deit-distilled-s16/hard-kd/seed-43/model.safetensors",
+      "num_classes": 100,
+      "parameters": 21743432,
+      "run_key": "deit_distilled-hard_kd-seed43",
+      "seed": 43,
+      "source_checkpoint": "deit_distilled-hard_kd-seed43-7b87d57ae5/best.pt",
+      "source_checkpoint_bytes": 261131233,
+      "source_checkpoint_sha256": "62f5fe35ee2926ff1e6d69cc588f4483605ead10ad96b231f8a4cf21e9de157b",
+      "test_top1": 89.61,
+      "test_top5": 98.94,
+      "timm_checkpoint": "deit_small_distilled_patch16_224.fb_in1k",
+      "training_mode": "hard_kd",
+      "training_peak_memory_mb": 3051.60888671875,
+      "training_seconds": 2475.6306675059996
+    },
+    {
+      "best_epoch": 19,
+      "best_validation_top1": 90.28,
+      "display_name": "Distilled DeiT-S/16",
+      "exported_bytes": 86988248,
+      "exported_sha256": "1cf9f71e4a426b11d2608aef603ac36614d51cf21f63d602ebaba3844f55d1af",
+      "family": "DeiT with class and distillation tokens",
+      "input_size": 224,
+      "model": "deit_distilled",
+      "model_file": "chapter-04-efficient-and-scalable-vision-transformers/checkpoints/deit-distilled-s16/hard-kd/seed-44/model.safetensors",
+      "num_classes": 100,
+      "parameters": 21743432,
+      "run_key": "deit_distilled-hard_kd-seed44",
+      "seed": 44,
+      "source_checkpoint": "deit_distilled-hard_kd-seed44-7b87d57ae5/best.pt",
+      "source_checkpoint_bytes": 261131233,
+      "source_checkpoint_sha256": "3fc93e36eb684ae97d74159987a2d204ae148b88973a0765af38f71268c2d7f6",
+      "test_top1": 89.72,
+      "test_top5": 98.85,
+      "timm_checkpoint": "deit_small_distilled_patch16_224.fb_in1k",
+      "training_mode": "hard_kd",
+      "training_peak_memory_mb": 3051.60888671875,
+      "training_seconds": 2467.4307780010004
+    }
+  ],
+  "repository_id": "Mayank022/Transformer-Vision-From-Image-Video-to-World-Models-and-Robotics-Book",
+  "safetensors_version": "0.8.0",
+  "schema_version": 1,
+  "torch_version": "2.9.1",
+  "total_exported_bytes": 1137364432
+}

chapter-04-efficient-and-scalable-vision-transformers/results/aggregate_results.csv

@@ -0,0 +1,8 @@
+model,display_name,mode,seeds,test_top1_mean,test_top1_std,test_top5_mean,best_validation_top1_mean,parameters_m,training_seconds_mean,training_peak_memory_mb
+convnext,ConvNeXt-T,standard,1,89.78,,98.86,89.86,27.897028,2789.4574548180008,4245.66796875
+vit,ViT-S/16,standard,1,89.22,,98.32,89.6,21.704164,1673.357545405,2563.0244140625
+swin,Swin-T,standard,1,89.54,,98.65,89.54,27.596254,3631.1184958180006,4843.58154296875
+efficientformer,EfficientFormer-L1,standard,1,86.83,,98.26,86.86,11.481728,1701.0025390570008,2633.85888671875
+deit,DeiT-S/16,standard,3,88.86,0.030000000000001137,98.19666666666667,89.71999999999998,21.704164,1676.6830429646668,2563.0244140625
+deit_distilled,Distilled DeiT-S/16,standard,3,89.45666666666666,0.2668957349478132,98.26666666666667,90.16666666666667,21.743432,1701.7341956493337,2566.7571614583335
+deit_distilled,Distilled DeiT-S/16,hard_kd,3,89.71999999999998,0.10999999999999943,98.88999999999999,90.26666666666667,21.743432,2477.565588236,3051.60888671875

chapter-04-efficient-and-scalable-vision-transformers/results/benchmark.json

@@ -0,0 +1,529 @@
+{
+  "config_fingerprint": "7b87d57ae5546600c460245a81431288db34c3e40a44c2e83da01acc4160d731",
+  "environment": {
+    "cuda": "12.8",
+    "cudnn": 91002,
+    "device_memory_mb": 22563.125,
+    "device_name": "NVIDIA L4",
+    "device_type": "cuda",
+    "machine": "x86_64",
+    "pid": 2,
+    "platform": "Linux-4.19.0-gvisor-x86_64-with-glibc2.36",
+    "python": "3.12.10",
+    "timm": "1.0.28",
+    "torch": "2.9.1+cu128",
+    "torchvision": "0.24.1+cu128"
+  },
+  "measurements": [
+    {
+      "batch_size": 1,
+      "latency_mean_ms": 4.777800642490387,
+      "latency_median_ms": 4.769615888595581,
+      "latency_p90_ms": 4.826025533676148,
+      "latency_std_ms": 0.09854292782443023,
+      "model": "convnext",
+      "peak_memory_mb": 132.1884765625,
+      "precision": "fp32",
+      "repeats": 5,
+      "samples": 1000,
+      "status": "complete",
+      "throughput_images_per_second": 209.3013239411259,
+      "timed_iterations": 200,
+      "warmup_iterations": 50
+    },
+    {
+      "batch_size": 64,
+      "latency_mean_ms": 127.90953128051758,
+      "latency_median_ms": 128.02047729492188,
+      "latency_p90_ms": 132.2467269897461,
+      "latency_std_ms": 3.4320032409674415,
+      "model": "convnext",
+      "peak_memory_mb": 962.6767578125,
+      "precision": "fp32",
+      "repeats": 5,
+      "samples": 1000,
+      "status": "complete",
+      "throughput_images_per_second": 500.3536433859804,
+      "timed_iterations": 200,
+      "warmup_iterations": 50
+    },
+    {
+      "batch_size": 1,
+      "latency_mean_ms": 5.376872823238373,
+      "latency_median_ms": 5.378112077713013,
+      "latency_p90_ms": 5.495808124542236,
+      "latency_std_ms": 0.11512835348717641,
+      "model": "convnext",
+      "peak_memory_mb": 183.4208984375,
+      "precision": "fp16",
+      "repeats": 5,
+      "samples": 1000,
+      "status": "complete",
+      "throughput_images_per_second": 185.98170960601627,
+      "timed_iterations": 200,
+      "warmup_iterations": 50
+    },
+    {
+      "batch_size": 64,
+      "latency_mean_ms": 57.763763633728026,
+      "latency_median_ms": 57.5544319152832,
+      "latency_p90_ms": 59.53341598510742,
+      "latency_std_ms": 1.2430091115071689,
+      "model": "convnext",
+      "peak_memory_mb": 728.7607421875,
+      "precision": "fp16",
+      "repeats": 5,
+      "samples": 1000,
+      "status": "complete",
+      "throughput_images_per_second": 1107.9610464064474,
+      "timed_iterations": 200,
+      "warmup_iterations": 50
+    },
+    {
+      "batch_size": 1,
+      "latency_mean_ms": 4.280692895412445,
+      "latency_median_ms": 4.261360168457031,
+      "latency_p90_ms": 4.343808174133301,
+      "latency_std_ms": 0.16694712271134712,
+      "model": "vit",
+      "peak_memory_mb": 97.1484375,
+      "precision": "fp32",
+      "repeats": 5,
+      "samples": 1000,
+      "status": "complete",
+      "throughput_images_per_second": 233.60704083016213,
+      "timed_iterations": 200,
+      "warmup_iterations": 50
+    },
+    {
+      "batch_size": 64,
+      "latency_mean_ms": 106.9532296295166,
+      "latency_median_ms": 106.99007797241211,
+      "latency_p90_ms": 108.48696365356446,
+      "latency_std_ms": 1.1434555353430906,
+      "model": "vit",
+      "peak_memory_mb": 351.763671875,
+      "precision": "fp32",
+      "repeats": 5,
+      "samples": 1000,
+      "status": "complete",
+      "throughput_images_per_second": 598.3924021901391,
+      "timed_iterations": 200,
+      "warmup_iterations": 50
+    },
+    {
+      "batch_size": 1,
+      "latency_mean_ms": 4.69389853143692,
+      "latency_median_ms": 4.659152030944824,
+      "latency_p90_ms": 4.780019092559815,
+      "latency_std_ms": 0.16487908037721768,
+      "model": "vit",
+      "peak_memory_mb": 137.90283203125,
+      "precision": "fp16",
+      "repeats": 5,
+      "samples": 1000,
+      "status": "complete",
+      "throughput_images_per_second": 213.04252601597565,
+      "timed_iterations": 200,
+      "warmup_iterations": 50
+    },
+    {
+      "batch_size": 64,
+      "latency_mean_ms": 32.072251497268674,
+      "latency_median_ms": 32.125953674316406,
+      "latency_p90_ms": 33.052775573730464,
+      "latency_std_ms": 0.8095062193205872,
+      "model": "vit",
+      "peak_memory_mb": 320.05126953125,
+      "precision": "fp16",
+      "repeats": 5,
+      "samples": 1000,
+      "status": "complete",
+      "throughput_images_per_second": 1995.4944543089014,
+      "timed_iterations": 200,
+      "warmup_iterations": 50
+    },
+    {
+      "batch_size": 1,
+      "latency_mean_ms": 8.11762380886078,
+      "latency_median_ms": 8.089599609375,
+      "latency_p90_ms": 8.21555233001709,
+      "latency_std_ms": 0.07401799461648069,
+      "model": "swin",
+      "peak_memory_mb": 134.0771484375,
+      "precision": "fp32",
+      "repeats": 5,
+      "samples": 1000,
+      "status": "complete",
+      "throughput_images_per_second": 123.18875862520896,
+      "timed_iterations": 200,
+      "warmup_iterations": 50
+    },
+    {
+      "batch_size": 64,
+      "latency_mean_ms": 178.14805532836914,
+      "latency_median_ms": 178.41305541992188,
+      "latency_p90_ms": 180.40453033447267,
+      "latency_std_ms": 1.9795728141419107,
+      "model": "swin",
+      "peak_memory_mb": 1077.705078125,
+      "precision": "fp32",
+      "repeats": 5,
+      "samples": 1000,
+      "status": "complete",
+      "throughput_images_per_second": 359.251746430983,
+      "timed_iterations": 200,
+      "warmup_iterations": 50
+    },
+    {
+      "batch_size": 1,
+      "latency_mean_ms": 9.13337626171112,
+      "latency_median_ms": 9.041919708251953,
+      "latency_p90_ms": 9.377792358398438,
+      "latency_std_ms": 0.29909913260705207,
+      "model": "swin",
+      "peak_memory_mb": 185.33349609375,
+      "precision": "fp16",
+      "repeats": 5,
+      "samples": 1000,
+      "status": "complete",
+      "throughput_images_per_second": 109.4885364782565,
+      "timed_iterations": 200,
+      "warmup_iterations": 50
+    },
+    {
+      "batch_size": 64,
+      "latency_mean_ms": 84.70931391143799,
+      "latency_median_ms": 84.73241424560547,
+      "latency_p90_ms": 86.2045181274414,
+      "latency_std_ms": 1.2024556603816736,
+      "model": "swin",
+      "peak_memory_mb": 970.5625,
+      "precision": "fp16",
+      "repeats": 5,
+      "samples": 1000,
+      "status": "complete",
+      "throughput_images_per_second": 755.5249481410133,
+      "timed_iterations": 200,
+      "warmup_iterations": 50
+    },
+    {
+      "batch_size": 1,
+      "latency_mean_ms": 5.772215712070465,
+      "latency_median_ms": 5.725183963775635,
+      "latency_p90_ms": 5.933670282363892,
+      "latency_std_ms": 0.18675654583815726,
+      "model": "efficientformer",
+      "peak_memory_mb": 60.97119140625,
+      "precision": "fp32",
+      "repeats": 5,
+      "samples": 1000,
+      "status": "complete",
+      "throughput_images_per_second": 173.2436987600564,
+      "timed_iterations": 200,
+      "warmup_iterations": 50
+    },
+    {
+      "batch_size": 64,
+      "latency_mean_ms": 56.72956251525879,
+      "latency_median_ms": 56.71116638183594,
+      "latency_p90_ms": 57.0277889251709,
+      "latency_std_ms": 0.22907337112652218,
+      "model": "efficientformer",
+      "peak_memory_mb": 497.74072265625,
+      "precision": "fp32",
+      "repeats": 5,
+      "samples": 1000,
+      "status": "complete",
+      "throughput_images_per_second": 1128.1595902098777,
+      "timed_iterations": 200,
+      "warmup_iterations": 50
+    },
+    {
+      "batch_size": 1,
+      "latency_mean_ms": 6.20901661491394,
+      "latency_median_ms": 6.112767934799194,
+      "latency_p90_ms": 6.5593345165252686,
+      "latency_std_ms": 0.30308211309436084,
+      "model": "efficientformer",
+      "peak_memory_mb": 80.21435546875,
+      "precision": "fp16",
+      "repeats": 5,
+      "samples": 1000,
+      "status": "complete",
+      "throughput_images_per_second": 161.05609986580146,
+      "timed_iterations": 200,
+      "warmup_iterations": 50
+    },
+    {
+      "batch_size": 64,
+      "latency_mean_ms": 32.634319339752196,
+      "latency_median_ms": 32.62673568725586,
+      "latency_p90_ms": 32.93388748168945,
+      "latency_std_ms": 0.21541325392113395,
+      "model": "efficientformer",
+      "peak_memory_mb": 354.11083984375,
+      "precision": "fp16",
+      "repeats": 5,
+      "samples": 1000,
+      "status": "complete",
+      "throughput_images_per_second": 1961.125627708158,
+      "timed_iterations": 200,
+      "warmup_iterations": 50
+    },
+    {
+      "batch_size": 1,
+      "latency_mean_ms": 4.368596731185913,
+      "latency_median_ms": 4.306943893432617,
+      "latency_p90_ms": 4.604022598266601,
+      "latency_std_ms": 0.16726820626053793,
+      "model": "deit",
+      "peak_memory_mb": 97.1484375,
+      "precision": "fp32",
+      "repeats": 5,
+      "samples": 1000,
+      "status": "complete",
+      "throughput_images_per_second": 228.9064570463424,
+      "timed_iterations": 200,
+      "warmup_iterations": 50
+    },
+    {
+      "batch_size": 64,
+      "latency_mean_ms": 106.82791548156739,
+      "latency_median_ms": 106.8067855834961,
+      "latency_p90_ms": 107.79443359375,
+      "latency_std_ms": 0.8205535043948153,
+      "model": "deit",
+      "peak_memory_mb": 351.763671875,
+      "precision": "fp32",
+      "repeats": 5,
+      "samples": 1000,
+      "status": "complete",
+      "throughput_images_per_second": 599.094344502518,
+      "timed_iterations": 200,
+      "warmup_iterations": 50
+    },
+    {
+      "batch_size": 1,
+      "latency_mean_ms": 4.722966948986054,
+      "latency_median_ms": 4.696063995361328,
+      "latency_p90_ms": 4.82521595954895,
+      "latency_std_ms": 0.09960610584523705,
+      "model": "deit",
+      "peak_memory_mb": 137.90283203125,
+      "precision": "fp16",
+      "repeats": 5,
+      "samples": 1000,
+      "status": "complete",
+      "throughput_images_per_second": 211.73131440495982,
+      "timed_iterations": 200,
+      "warmup_iterations": 50
+    },
+    {
+      "batch_size": 64,
+      "latency_mean_ms": 32.135718923568724,
+      "latency_median_ms": 32.168447494506836,
+      "latency_p90_ms": 33.121176147460936,
+      "latency_std_ms": 0.8138286399443037,
+      "model": "deit",
+      "peak_memory_mb": 320.05126953125,
+      "precision": "fp16",
+      "repeats": 5,
+      "samples": 1000,
+      "status": "complete",
+      "throughput_images_per_second": 1991.5533911725133,
+      "timed_iterations": 200,
+      "warmup_iterations": 50
+    },
+    {
+      "batch_size": 1,
+      "latency_mean_ms": 4.383408827781677,
+      "latency_median_ms": 4.355168104171753,
+      "latency_p90_ms": 4.442111968994141,
+      "latency_std_ms": 0.20651985327066089,
+      "model": "deit_distilled",
+      "peak_memory_mb": 97.30419921875,
+      "precision": "fp32",
+      "repeats": 5,
+      "samples": 1000,
+      "status": "complete",
+      "throughput_images_per_second": 228.13295297990092,
+      "timed_iterations": 200,
+      "warmup_iterations": 50
+    },
+    {
+      "batch_size": 64,
+      "latency_mean_ms": 105.38310792541503,
+      "latency_median_ms": 105.41926574707031,
+      "latency_p90_ms": 106.30379638671876,
+      "latency_std_ms": 0.7789229665081872,
+      "model": "deit_distilled",
+      "peak_memory_mb": 352.50732421875,
+      "precision": "fp32",
+      "repeats": 5,
+      "samples": 1000,
+      "status": "complete",
+      "throughput_images_per_second": 607.3079572230498,
+      "timed_iterations": 200,
+      "warmup_iterations": 50
+    },
+    {
+      "batch_size": 1,
+      "latency_mean_ms": 4.723058529853821,
+      "latency_median_ms": 4.704256057739258,
+      "latency_p90_ms": 4.769792079925537,
+      "latency_std_ms": 0.15652674115081888,
+      "model": "deit_distilled",
+      "peak_memory_mb": 138.13818359375,
+      "precision": "fp16",
+      "repeats": 5,
+      "samples": 1000,
+      "status": "complete",
+      "throughput_images_per_second": 211.72720890057442,
+      "timed_iterations": 200,
+      "warmup_iterations": 50
+    },
+    {
+      "batch_size": 64,
+      "latency_mean_ms": 32.341811212539675,
+      "latency_median_ms": 32.4213752746582,
+      "latency_p90_ms": 33.34256629943848,
+      "latency_std_ms": 0.8330030872120695,
+      "model": "deit_distilled",
+      "peak_memory_mb": 321.55615234375,
+      "precision": "fp16",
+      "repeats": 5,
+      "samples": 1000,
+      "status": "complete",
+      "throughput_images_per_second": 1978.8625806827326,
+      "timed_iterations": 200,
+      "warmup_iterations": 50
+    }
+  ],
+  "preset": "full",
+  "profiles": [
+    {
+      "checkpoint": "convnext_tiny.fb_in1k",
+      "display_name": "ConvNeXt-T",
+      "error": null,
+      "expected_gmacs_from_model_card": 4.5,
+      "expected_parameters_m": 28.6,
+      "measured_gmacs": 4.469746176,
+      "measured_macs": 4469746176,
+      "model": "convnext",
+      "parameters": 27897028,
+      "representative_run": "convnext-standard-seed42",
+      "unsupported_operators": {
+        "aten::add": 18,
+        "aten::gelu": 18,
+        "aten::mul": 18
+      }
+    },
+    {
+      "checkpoint": "vit_small_patch16_224.augreg_in1k",
+      "display_name": "ViT-S/16",
+      "error": null,
+      "expected_gmacs_from_model_card": 4.3,
+      "expected_parameters_m": 22.1,
+      "measured_gmacs": 4.25032896,
+      "measured_macs": 4250328960,
+      "model": "vit",
+      "parameters": 21704164,
+      "representative_run": "vit-standard-seed42",
+      "unsupported_operators": {
+        "aten::add": 25,
+        "aten::gelu": 12,
+        "aten::scaled_dot_product_attention": 12
+      }
+    },
+    {
+      "checkpoint": "swin_tiny_patch4_window7_224.ms_in1k",
+      "display_name": "Swin-T",
+      "error": null,
+      "expected_gmacs_from_model_card": 4.5,
+      "expected_parameters_m": 28.3,
+      "measured_gmacs": 4.508503296,
+      "measured_macs": 4508503296,
+      "model": "swin",
+      "parameters": 27596254,
+      "representative_run": "swin-standard-seed42",
+      "unsupported_operators": {
+        "aten::add": 41,
+        "aten::bernoulli_": 22,
+        "aten::div_": 22,
+        "aten::gelu": 12,
+        "aten::mean": 1,
+        "aten::mul": 34,
+        "aten::pad": 15,
+        "aten::rsub": 24,
+        "aten::softmax": 12
+      }
+    },
+    {
+      "checkpoint": "efficientformer_l1.snap_dist_in1k",
+      "display_name": "EfficientFormer-L1",
+      "error": null,
+      "expected_gmacs_from_model_card": 1.3,
+      "expected_parameters_m": 12.3,
+      "measured_gmacs": 1.317055936,
+      "measured_macs": 1317055936,
+      "model": "efficientformer",
+      "parameters": 11481728,
+      "representative_run": "efficientformer-standard-seed42",
+      "unsupported_operators": {
+        "aten::add": 32,
+        "aten::add_": 33,
+        "aten::avg_pool2d": 14,
+        "aten::div": 1,
+        "aten::gelu": 15,
+        "aten::mean": 1,
+        "aten::mul": 31,
+        "aten::softmax": 1,
+        "aten::sub": 14
+      }
+    },
+    {
+      "checkpoint": "deit_small_patch16_224.fb_in1k",
+      "display_name": "DeiT-S/16",
+      "error": null,
+      "expected_gmacs_from_model_card": 4.6,
+      "expected_parameters_m": 22.1,
+      "measured_gmacs": 4.25032896,
+      "measured_macs": 4250328960,
+      "model": "deit",
+      "parameters": 21704164,
+      "representative_run": "deit-standard-seed42",
+      "unsupported_operators": {
+        "aten::add": 25,
+        "aten::gelu": 12,
+        "aten::scaled_dot_product_attention": 12
+      }
+    },
+    {
+      "checkpoint": "deit_small_distilled_patch16_224.fb_in1k",
+      "display_name": "Distilled DeiT-S/16",
+      "error": null,
+      "expected_gmacs_from_model_card": 4.6,
+      "expected_parameters_m": 22.4,
+      "measured_gmacs": 4.271649024,
+      "measured_macs": 4271649024,
+      "model": "deit_distilled",
+      "parameters": 21743432,
+      "representative_run": "deit_distilled-standard-seed42",
+      "unsupported_operators": {
+        "aten::add": 26,
+        "aten::div": 1,
+        "aten::gelu": 12,
+        "aten::scaled_dot_product_attention": 12
+      }
+    }
+  ],
+  "protocol": {
+    "includes_data_loading": false,
+    "input_size": 224,
+    "synchronization": "CUDA events per iteration; synchronized wall clock otherwise",
+    "weights": "best validation checkpoint; one representative per architecture"
+  },
+  "schema_version": 1,
+  "tutorial_only": false
+}

chapter-04-efficient-and-scalable-vision-transformers/results/distillation_gains.csv

@@ -0,0 +1,4 @@
+seed,standard_top1,hard_kd_top1,gain_percentage_points
+42,89.74,89.83,0.09000000000000341
+43,89.21,89.61,0.4000000000000057
+44,89.42,89.72,0.29999999999999716

chapter-04-efficient-and-scalable-vision-transformers/results/epoch_metrics.csv

@@ -0,0 +1,261 @@
+run_key,model,display_name,mode,seed,epoch,train_loss,train_top1,validation_loss,validation_top1,cumulative_seconds,classification_loss,distillation_loss
+convnext-standard-seed42,convnext,ConvNeXt-T,standard,42,1,3.7667001406351726,31.56888888888889,1.8161395473480224,70.0,159.24966592000033,,
+convnext-standard-seed42,convnext,ConvNeXt-T,standard,42,2,1.8095489675309924,70.46,0.7216742023468018,83.18,296.13400899700014,,
+convnext-standard-seed42,convnext,ConvNeXt-T,standard,42,3,1.4038030299504598,79.46666666666667,0.5683593006610871,86.54,434.103854834,,
+convnext-standard-seed42,convnext,ConvNeXt-T,standard,42,4,1.2779912848154704,83.35555555555555,0.5082297623157501,87.32,572.8992680669996,,
+convnext-standard-seed42,convnext,ConvNeXt-T,standard,42,5,1.197589386929406,85.94222222222223,0.4819089977264404,87.86,711.395580007,,
+convnext-standard-seed42,convnext,ConvNeXt-T,standard,42,6,1.139879513253106,88.0,0.4684164373397827,88.32,849.9560964589998,,
+convnext-standard-seed42,convnext,ConvNeXt-T,standard,42,7,1.0999575252744886,89.35777777777778,0.4619133004665375,88.76,988.9534960309995,,
+convnext-standard-seed42,convnext,ConvNeXt-T,standard,42,8,1.0620388389693367,90.66,0.4679763640880585,88.36,1127.306333117,,
+convnext-standard-seed42,convnext,ConvNeXt-T,standard,42,9,1.0302280043496026,91.79777777777778,0.45847382922172547,89.2,1265.6016385320002,,
+convnext-standard-seed42,convnext,ConvNeXt-T,standard,42,10,1.0055605604489644,92.85111111111111,0.44975349364280703,89.12,1404.8528756820003,,
+convnext-standard-seed42,convnext,ConvNeXt-T,standard,42,11,0.9827514073371887,93.58666666666667,0.45226348009109496,88.88,1542.4162078940005,,
+convnext-standard-seed42,convnext,ConvNeXt-T,standard,42,12,0.9706555672645569,94.02888888888889,0.4561493213176727,89.16,1681.0910127630007,,
+convnext-standard-seed42,convnext,ConvNeXt-T,standard,42,13,0.9541406662835015,94.58666666666667,0.45763017377853393,89.24,1819.9787216280004,,
+convnext-standard-seed42,convnext,ConvNeXt-T,standard,42,14,0.9400494391123454,95.12666666666667,0.4576495222568512,89.08,1958.8628607710004,,
+convnext-standard-seed42,convnext,ConvNeXt-T,standard,42,15,0.9326667542987399,95.41555555555556,0.45384993853569033,89.6,2098.1818932940005,,
+convnext-standard-seed42,convnext,ConvNeXt-T,standard,42,16,0.9284473850038316,95.42888888888889,0.45101784572601317,89.6,2237.0839156250004,,
+convnext-standard-seed42,convnext,ConvNeXt-T,standard,42,17,0.9209549523247613,95.80666666666667,0.4518927227973938,89.86,2376.160370025,,
+convnext-standard-seed42,convnext,ConvNeXt-T,standard,42,18,0.9181310733053419,95.89111111111112,0.4524921130180359,89.66,2514.854323261,,
+convnext-standard-seed42,convnext,ConvNeXt-T,standard,42,19,0.9160427381409539,95.92222222222222,0.4524287845611572,89.68,2651.8460178230007,,
+convnext-standard-seed42,convnext,ConvNeXt-T,standard,42,20,0.9136880412737528,96.03333333333333,0.45195455298423765,89.7,2789.4574548180008,,
+vit-standard-seed42,vit,ViT-S/16,standard,42,1,2.902411583709717,42.67333333333333,0.7650930633544922,80.46,83.11916694800001,,
+vit-standard-seed42,vit,ViT-S/16,standard,42,2,1.5302659155527751,75.52666666666667,0.649686996459961,83.54,164.81945953899998,,
+vit-standard-seed42,vit,ViT-S/16,standard,42,3,1.3980587674034965,79.51555555555555,0.5959713744163513,84.52,246.867734873,,
+vit-standard-seed42,vit,ViT-S/16,standard,42,4,1.2869545261594983,83.11777777777777,0.573504077720642,85.8,331.16401449499995,,
+vit-standard-seed42,vit,ViT-S/16,standard,42,5,1.2156278246985541,85.6288888888889,0.5497735964775086,86.28,415.21942545599995,,
+vit-standard-seed42,vit,ViT-S/16,standard,42,6,1.1534308321634927,87.72666666666667,0.5519826049804688,86.18,499.18630522800004,,
+vit-standard-seed42,vit,ViT-S/16,standard,42,7,1.1043820426834954,89.32,0.5393952094078064,87.08,583.2546063570001,,
+vit-standard-seed42,vit,ViT-S/16,standard,42,8,1.0637690071105956,90.77777777777777,0.5385724670410156,86.7,667.412148083,,
+vit-standard-seed42,vit,ViT-S/16,standard,42,9,1.0268184286435444,92.14222222222222,0.5259280432701111,87.26,751.691665754,,
+vit-standard-seed42,vit,ViT-S/16,standard,42,10,0.999160442966885,93.18888888888888,0.5202932633399964,87.9,835.836512563,,
+vit-standard-seed42,vit,ViT-S/16,standard,42,11,0.9693983827379015,94.13555555555556,0.514621384191513,88.42,920.0706118429999,,
+vit-standard-seed42,vit,ViT-S/16,standard,42,12,0.9511773754755656,94.86666666666666,0.5179296373844147,87.9,1004.0796335339998,,
+vit-standard-seed42,vit,ViT-S/16,standard,42,13,0.9272429713990953,95.67111111111112,0.5121905665397644,88.8,1086.6519678279997,,
+vit-standard-seed42,vit,ViT-S/16,standard,42,14,0.910502517996894,96.21333333333334,0.5084908224105835,88.6,1168.9551837609997,,
+vit-standard-seed42,vit,ViT-S/16,standard,42,15,0.8966341973834567,96.69555555555556,0.5138636219501496,88.86,1252.1163708319998,,
+vit-standard-seed42,vit,ViT-S/16,standard,42,16,0.8879092311435276,97.07111111111111,0.5049399044036865,89.06,1336.772580855,,
+vit-standard-seed42,vit,ViT-S/16,standard,42,17,0.8786597423129612,97.32,0.5035600093841552,89.32,1421.238053443,,
+vit-standard-seed42,vit,ViT-S/16,standard,42,18,0.8743622001012167,97.47777777777777,0.4988097204208374,89.2,1505.2902451799998,,
+vit-standard-seed42,vit,ViT-S/16,standard,42,19,0.8682376603338453,97.64888888888889,0.4981942099571228,89.18,1589.188959569,,
+vit-standard-seed42,vit,ViT-S/16,standard,42,20,0.8671988519350687,97.64888888888889,0.49440441904067994,89.6,1673.357545405,,
+swin-standard-seed42,swin,Swin-T,standard,42,1,3.1138392703586155,36.37777777777778,0.8303258764266968,78.2,182.143709855,,
+swin-standard-seed42,swin,Swin-T,standard,42,2,1.6815805814531115,71.27555555555556,0.6566583446502685,82.28,362.3566214130001,,
+swin-standard-seed42,swin,Swin-T,standard,42,3,1.4958824817233616,76.2688888888889,0.5993878432750702,84.08,542.215195601,,
+swin-standard-seed42,swin,Swin-T,standard,42,4,1.3791291867574056,80.05111111111111,0.5440349873542786,85.46,722.4046990390001,,
+swin-standard-seed42,swin,Swin-T,standard,42,5,1.2997279549280802,82.76222222222222,0.5114192454338073,86.52,902.6733066970002,,
+swin-standard-seed42,swin,Swin-T,standard,42,6,1.2405421981493632,84.82222222222222,0.5049465229034423,87.18,1083.1310168,,
+swin-standard-seed42,swin,Swin-T,standard,42,7,1.1918076894548204,86.44444444444444,0.49310361399650576,87.02,1264.15614292,,
+swin-standard-seed42,swin,Swin-T,standard,42,8,1.1489616010453967,88.01777777777778,0.4983553493976593,87.32,1445.669742795,,
+swin-standard-seed42,swin,Swin-T,standard,42,9,1.1118056599511041,89.42666666666666,0.47800114431381224,87.84,1627.8956264800001,,
+swin-standard-seed42,swin,Swin-T,standard,42,10,1.0824293551127115,90.58888888888889,0.48054675245285033,87.88,1809.7915364240002,,
+swin-standard-seed42,swin,Swin-T,standard,42,11,1.0564987625969782,91.44222222222223,0.4688482060432434,88.04,1992.3521930600004,,
+swin-standard-seed42,swin,Swin-T,standard,42,12,1.0332676078372531,92.25555555555556,0.4628698630809784,89.16,2174.7467130040004,,
+swin-standard-seed42,swin,Swin-T,standard,42,13,1.0080077913920085,93.19777777777777,0.4688817408561707,88.56,2357.198300522,,
+swin-standard-seed42,swin,Swin-T,standard,42,14,0.9919483540429009,93.75333333333333,0.4652939272403717,88.94,2538.5730069070005,,
+swin-standard-seed42,swin,Swin-T,standard,42,15,0.9800122749010722,94.24,0.4553578992366791,89.3,2720.9659736020003,,
+swin-standard-seed42,swin,Swin-T,standard,42,16,0.9694040551715427,94.65333333333334,0.4507915984630585,89.48,2903.200041328,,
+swin-standard-seed42,swin,Swin-T,standard,42,17,0.9622505562676323,94.91555555555556,0.4503328215122223,89.5,3085.8433564590005,,
+swin-standard-seed42,swin,Swin-T,standard,42,18,0.9549930867618984,95.14666666666666,0.4524413750648498,89.52,3267.932922547,,
+swin-standard-seed42,swin,Swin-T,standard,42,19,0.9485410739368862,95.36444444444444,0.4528714479923248,89.54,3450.0135159150004,,
+swin-standard-seed42,swin,Swin-T,standard,42,20,0.9490107592052883,95.34222222222222,0.45156445999145506,89.54,3631.1184958180006,,
+efficientformer-standard-seed42,efficientformer,EfficientFormer-L1,standard,42,1,3.86097444513109,20.94,1.5167168102264403,61.48,87.81622463100001,,
+efficientformer-standard-seed42,efficientformer,EfficientFormer-L1,standard,42,2,1.963156677500407,62.82222222222222,0.8302864381790162,78.68,171.946322549,,
+efficientformer-standard-seed42,efficientformer,EfficientFormer-L1,standard,42,3,1.6159678107791478,73.06888888888889,0.7159340080261231,82.66,256.353993296,,
+efficientformer-standard-seed42,efficientformer,EfficientFormer-L1,standard,42,4,1.4824932571411134,77.32,0.6627669515609741,83.4,340.65199616900003,,
+efficientformer-standard-seed42,efficientformer,EfficientFormer-L1,standard,42,5,1.386941508441501,80.6,0.6122228536605835,84.48,425.15089323200004,,
+efficientformer-standard-seed42,efficientformer,EfficientFormer-L1,standard,42,6,1.3214185864342582,82.75555555555556,0.6030615650177001,84.94,509.56850352800006,,
+efficientformer-standard-seed42,efficientformer,EfficientFormer-L1,standard,42,7,1.2774883761088054,84.15555555555555,0.5855043698310852,85.2,593.7636439120001,,
+efficientformer-standard-seed42,efficientformer,EfficientFormer-L1,standard,42,8,1.2294677474127875,85.8711111111111,0.5810406120300293,85.48,678.2063882370003,,
+efficientformer-standard-seed42,efficientformer,EfficientFormer-L1,standard,42,9,1.1969785375383166,87.09333333333333,0.5696206838607788,85.52,762.6665857520002,,
+efficientformer-standard-seed42,efficientformer,EfficientFormer-L1,standard,42,10,1.1670788527382745,88.24222222222222,0.5626538040161133,86.16,847.2176097350002,,
+efficientformer-standard-seed42,efficientformer,EfficientFormer-L1,standard,42,11,1.1434139019860161,89.11111111111111,0.555591562795639,86.18,931.4248162340004,,
+efficientformer-standard-seed42,efficientformer,EfficientFormer-L1,standard,42,12,1.1194964197794597,89.94,0.5510715413093566,86.28,1015.5840988140005,,
+efficientformer-standard-seed42,efficientformer,EfficientFormer-L1,standard,42,13,1.101558770688375,90.42444444444445,0.5498418887138367,86.48,1102.2390305400006,,
+efficientformer-standard-seed42,efficientformer,EfficientFormer-L1,standard,42,14,1.0842033411237928,91.24,0.5494314468383789,86.3,1186.8125441570005,,
+efficientformer-standard-seed42,efficientformer,EfficientFormer-L1,standard,42,15,1.0785100873947144,91.50444444444445,0.5476529896259308,86.46,1272.3136917130005,,
+efficientformer-standard-seed42,efficientformer,EfficientFormer-L1,standard,42,16,1.0688635341538324,91.85111111111111,0.5373651230812073,86.66,1357.7778178120006,,
+efficientformer-standard-seed42,efficientformer,EfficientFormer-L1,standard,42,17,1.0617455692503188,92.08222222222223,0.5425855985641479,86.82,1443.1470524550007,,
+efficientformer-standard-seed42,efficientformer,EfficientFormer-L1,standard,42,18,1.0559050056669448,92.26,0.5469162731170655,86.64,1529.4091974040007,,
+efficientformer-standard-seed42,efficientformer,EfficientFormer-L1,standard,42,19,1.056951585345798,92.27333333333333,0.5501786388397217,86.62,1615.1883237940008,,
+efficientformer-standard-seed42,efficientformer,EfficientFormer-L1,standard,42,20,1.0506543249236213,92.40444444444445,0.5462933820724487,86.86,1701.0025390570008,,
+deit-standard-seed42,deit,DeiT-S/16,standard,42,1,3.194199195543925,37.67111111111111,0.9571177589416504,77.64,82.860228159,,
+deit-standard-seed42,deit,DeiT-S/16,standard,42,2,1.5704157097286648,74.24444444444444,0.6539145792961121,83.72,163.526656124,,
+deit-standard-seed42,deit,DeiT-S/16,standard,42,3,1.3949972147199843,79.32666666666667,0.5740103552818299,85.5,244.49539434799996,,
+deit-standard-seed42,deit,DeiT-S/16,standard,42,4,1.2877895130793253,82.95555555555555,0.5353437189102173,86.28,326.0574370830001,,
+deit-standard-seed42,deit,DeiT-S/16,standard,42,5,1.2159836768468222,85.20666666666666,0.5291574817180633,86.56,407.06445852300004,,
+deit-standard-seed42,deit,DeiT-S/16,standard,42,6,1.154969121668074,87.49111111111111,0.5094177061080932,88.12,487.88356403500006,,
+deit-standard-seed42,deit,DeiT-S/16,standard,42,7,1.110998962349362,88.97111111111111,0.5132681596279144,87.16,568.3669005710001,,
+deit-standard-seed42,deit,DeiT-S/16,standard,42,8,1.066391011344062,90.48222222222222,0.4960452775001526,88.08,648.94724951,,
+deit-standard-seed42,deit,DeiT-S/16,standard,42,9,1.0294513582865397,91.91111111111111,0.49717512345314024,88.4,730.5383563429999,,
+deit-standard-seed42,deit,DeiT-S/16,standard,42,10,1.0010287435213725,92.9888888888889,0.49309459257125854,88.36,811.1509497509999,,
+deit-standard-seed42,deit,DeiT-S/16,standard,42,11,0.9738257635116577,93.88444444444444,0.5020302115440368,88.28,892.7442546759999,,
+deit-standard-seed42,deit,DeiT-S/16,standard,42,12,0.952776749420166,94.6,0.4851966609477997,88.68,973.4491799649999,,
+deit-standard-seed42,deit,DeiT-S/16,standard,42,13,0.9325482057147556,95.30888888888889,0.4961971499443054,89.02,1054.4753422049998,,
+deit-standard-seed42,deit,DeiT-S/16,standard,42,14,0.9174271399603949,95.89111111111112,0.48942312593460086,89.32,1136.504434496,,
+deit-standard-seed42,deit,DeiT-S/16,standard,42,15,0.9073710475073921,96.11777777777777,0.4874080156326294,89.16,1219.5248612139999,,
+deit-standard-seed42,deit,DeiT-S/16,standard,42,16,0.8973959156354269,96.44888888888889,0.4900791407585144,89.44,1302.509800286,,
+deit-standard-seed42,deit,DeiT-S/16,standard,42,17,0.8886431807623969,96.7088888888889,0.48342578125,89.74,1386.602007661,,
+deit-standard-seed42,deit,DeiT-S/16,standard,42,18,0.8851760086165534,96.89333333333333,0.4842074983596802,89.58,1470.9288105599999,,
+deit-standard-seed42,deit,DeiT-S/16,standard,42,19,0.8782249345779419,97.1511111111111,0.48175542941093447,89.7,1555.347857367,,
+deit-standard-seed42,deit,DeiT-S/16,standard,42,20,0.8758682016266717,97.21111111111111,0.48031524109840396,89.8,1639.156589266,,
+deit-standard-seed43,deit,DeiT-S/16,standard,43,1,3.1554136654324,38.24444444444445,0.9740959652900696,77.38,84.901784078,,
+deit-standard-seed43,deit,DeiT-S/16,standard,43,2,1.5730218242645264,74.08444444444444,0.6653622232437134,82.8,168.07388327900003,,
+deit-standard-seed43,deit,DeiT-S/16,standard,43,3,1.4070982536527845,79.16666666666667,0.5856769444465637,85.18,253.31197407000002,,
+deit-standard-seed43,deit,DeiT-S/16,standard,43,4,1.2846227007971869,82.9,0.5376330299377441,86.32,337.77078011800006,,
+deit-standard-seed43,deit,DeiT-S/16,standard,43,5,1.216858652750651,85.30888888888889,0.5463018033981323,86.14,421.9236422840001,,
+deit-standard-seed43,deit,DeiT-S/16,standard,43,6,1.1511801248338487,87.44222222222223,0.509611957025528,87.22,507.1537427510001,,
+deit-standard-seed43,deit,DeiT-S/16,standard,43,7,1.1035662026723225,89.2688888888889,0.49713537378311157,87.76,590.0196407740001,,
+deit-standard-seed43,deit,DeiT-S/16,standard,43,8,1.0636458375400968,90.48,0.48970663332939146,88.16,674.4691078180001,,
+deit-standard-seed43,deit,DeiT-S/16,standard,43,9,1.0285717680401272,91.92444444444445,0.48187320470809936,88.54,759.7265785280001,,
+deit-standard-seed43,deit,DeiT-S/16,standard,43,10,1.0023919584910075,92.85111111111111,0.48758344564437867,88.52,845.157430746,,
+deit-standard-seed43,deit,DeiT-S/16,standard,43,11,0.9762912966516283,93.6,0.4860206575870514,88.9,929.443957199,,
+deit-standard-seed43,deit,DeiT-S/16,standard,43,12,0.9509553065723843,94.59333333333333,0.4846278016090393,89.0,1014.9636868690001,,
+deit-standard-seed43,deit,DeiT-S/16,standard,43,13,0.933472623931037,95.24666666666667,0.48037488231658937,88.78,1097.9972289520001,,
+deit-standard-seed43,deit,DeiT-S/16,standard,43,14,0.9183323530515035,95.80666666666667,0.4773489830970764,89.34,1181.646935469,,
+deit-standard-seed43,deit,DeiT-S/16,standard,43,15,0.9042640240881178,96.16444444444444,0.4777989665031433,89.24,1266.802762635,,
+deit-standard-seed43,deit,DeiT-S/16,standard,43,16,0.9012931256082323,96.37555555555555,0.4747199629306793,89.44,1349.042777788,,
+deit-standard-seed43,deit,DeiT-S/16,standard,43,17,0.8854330974155002,96.88444444444444,0.46981862335205077,89.56,1432.589390092,,
+deit-standard-seed43,deit,DeiT-S/16,standard,43,18,0.8824417904429965,97.02,0.4704814514160156,89.6,1518.1876454740002,,
+deit-standard-seed43,deit,DeiT-S/16,standard,43,19,0.8763937163247002,97.18888888888888,0.4697572201251984,89.68,1604.0159294580003,,
+deit-standard-seed43,deit,DeiT-S/16,standard,43,20,0.8764924024793836,97.24444444444444,0.46827153100967406,89.7,1688.338380226,,
+deit-standard-seed44,deit,DeiT-S/16,standard,44,1,3.196826624806722,37.33777777777778,1.0013699850082398,76.7,84.213616519,,
+deit-standard-seed44,deit,DeiT-S/16,standard,44,2,1.571290406841702,74.00222222222222,0.6547010645389557,83.54,168.12630540600003,,
+deit-standard-seed44,deit,DeiT-S/16,standard,44,3,1.3961302981058756,79.30666666666667,0.5598467971801758,85.62,253.62356726400006,,
+deit-standard-seed44,deit,DeiT-S/16,standard,44,4,1.2877197409311931,82.85333333333334,0.5460019410133362,85.68,338.49668671300003,,
+deit-standard-seed44,deit,DeiT-S/16,standard,44,5,1.209022402911716,85.53555555555556,0.5317278738021851,86.64,423.599549614,,
+deit-standard-seed44,deit,DeiT-S/16,standard,44,6,1.1572605774349636,87.45333333333333,0.5061660194873809,87.44,511.098877663,,
+deit-standard-seed44,deit,DeiT-S/16,standard,44,7,1.1038765400038826,89.26444444444445,0.5105860779762268,87.18,596.784319558,,
+deit-standard-seed44,deit,DeiT-S/16,standard,44,8,1.0689265658272638,90.45111111111112,0.491797119808197,88.16,681.875149594,,
+deit-standard-seed44,deit,DeiT-S/16,standard,44,9,1.0335834069781833,91.81777777777778,0.4941571761608124,88.18,767.032548826,,
+deit-standard-seed44,deit,DeiT-S/16,standard,44,10,1.0024565981970892,92.84666666666666,0.48806377296447756,88.56,852.112595989,,
+deit-standard-seed44,deit,DeiT-S/16,standard,44,11,0.9778886142412822,93.74444444444444,0.4922834037303925,88.36,937.2038073390002,,
+deit-standard-seed44,deit,DeiT-S/16,standard,44,12,0.9564637901730008,94.55111111111111,0.4861561080932617,88.52,1022.2376097670001,,
+deit-standard-seed44,deit,DeiT-S/16,standard,44,13,0.9358821586290995,95.18666666666667,0.48182631273269655,88.72,1107.0871700500002,,
+deit-standard-seed44,deit,DeiT-S/16,standard,44,14,0.9205817525015937,95.71333333333334,0.4810323775291443,89.02,1192.059168028,,
+deit-standard-seed44,deit,DeiT-S/16,standard,44,15,0.9094649105919732,96.12222222222222,0.4806832550048828,89.3,1277.24111453,,
+deit-standard-seed44,deit,DeiT-S/16,standard,44,16,0.8969224764929877,96.5111111111111,0.48068887367248536,89.36,1362.426087027,,
+deit-standard-seed44,deit,DeiT-S/16,standard,44,17,0.8901528752538893,96.70666666666666,0.47879109582901,89.62,1447.373477251,,
+deit-standard-seed44,deit,DeiT-S/16,standard,44,18,0.881282042588128,97.01555555555555,0.4804483486175537,89.66,1532.604494716,,
+deit-standard-seed44,deit,DeiT-S/16,standard,44,19,0.8807935371186998,97.10444444444444,0.47964754467010495,89.6,1617.6611158340002,,
+deit-standard-seed44,deit,DeiT-S/16,standard,44,20,0.8797675667126974,97.11111111111111,0.47917526655197146,89.58,1702.554159402,,
+deit_distilled-standard-seed42,deit_distilled,Distilled DeiT-S/16,standard,42,1,2.971982460445828,43.931111111111115,0.8180987700462341,80.22,83.843518139,,
+deit_distilled-standard-seed42,deit_distilled,Distilled DeiT-S/16,standard,42,2,1.5167006523132325,75.96,0.6119530912399292,84.4,167.168673744,,
+deit_distilled-standard-seed42,deit_distilled,Distilled DeiT-S/16,standard,42,3,1.3666032607608372,80.22444444444444,0.569834406042099,85.36,250.33327016199993,,
+deit_distilled-standard-seed42,deit_distilled,Distilled DeiT-S/16,standard,42,4,1.263768236372206,83.72,0.5298959607124328,86.88,333.486853559,,
+deit_distilled-standard-seed42,deit_distilled,Distilled DeiT-S/16,standard,42,5,1.19601071272956,86.19555555555556,0.517924436044693,87.26,416.604916755,,
+deit_distilled-standard-seed42,deit_distilled,Distilled DeiT-S/16,standard,42,6,1.1366201146337722,88.34444444444445,0.4964476699829102,87.62,500.433297622,,
+deit_distilled-standard-seed42,deit_distilled,Distilled DeiT-S/16,standard,42,7,1.0915187283833823,89.85777777777778,0.49391090426445006,87.94,583.6337957989999,,
+deit_distilled-standard-seed42,deit_distilled,Distilled DeiT-S/16,standard,42,8,1.0519614795684815,91.31111111111112,0.4845743315219879,88.52,667.746342061,,
+deit_distilled-standard-seed42,deit_distilled,Distilled DeiT-S/16,standard,42,9,1.0190510958777534,92.41111111111111,0.4960920108795166,88.6,752.2955900499999,,
+deit_distilled-standard-seed42,deit_distilled,Distilled DeiT-S/16,standard,42,10,0.9927112843725416,93.47777777777777,0.48470961332321166,88.94,836.8128453109999,,
+deit_distilled-standard-seed42,deit_distilled,Distilled DeiT-S/16,standard,42,11,0.9697918816778395,94.2,0.4851248734474182,88.82,921.118611499,,
+deit_distilled-standard-seed42,deit_distilled,Distilled DeiT-S/16,standard,42,12,0.9485911116917928,94.96,0.48808993263244627,89.18,1007.4213817830001,,
+deit_distilled-standard-seed42,deit_distilled,Distilled DeiT-S/16,standard,42,13,0.9298490361425612,95.5911111111111,0.4955332342624664,89.12,1092.0135485810001,,
+deit_distilled-standard-seed42,deit_distilled,Distilled DeiT-S/16,standard,42,14,0.9151839838027954,96.06444444444445,0.4814388228416443,89.38,1175.237132826,,
+deit_distilled-standard-seed42,deit_distilled,Distilled DeiT-S/16,standard,42,15,0.905952920850118,96.33333333333333,0.4765923174858093,89.34,1259.867307092,,
+deit_distilled-standard-seed42,deit_distilled,Distilled DeiT-S/16,standard,42,16,0.8961676205211215,96.72444444444444,0.47162820463180544,89.7,1344.531542515,,
+deit_distilled-standard-seed42,deit_distilled,Distilled DeiT-S/16,standard,42,17,0.8858082230461969,97.03777777777778,0.4764399881362915,89.64,1428.171998382,,
+deit_distilled-standard-seed42,deit_distilled,Distilled DeiT-S/16,standard,42,18,0.8820553650644091,97.22,0.474872510433197,89.94,1512.8349848000003,,
+deit_distilled-standard-seed42,deit_distilled,Distilled DeiT-S/16,standard,42,19,0.878790746720632,97.33333333333333,0.47467298197746277,90.04,1596.8932530280003,,
+deit_distilled-standard-seed42,deit_distilled,Distilled DeiT-S/16,standard,42,20,0.8764258578088548,97.31777777777778,0.47465604796409605,90.0,1680.1939577700005,,
+deit_distilled-standard-seed43,deit_distilled,Distilled DeiT-S/16,standard,43,1,3.0095239078097875,41.98,0.8497823201179504,79.96,86.148048668,,
+deit_distilled-standard-seed43,deit_distilled,Distilled DeiT-S/16,standard,43,2,1.5267975524478488,75.38666666666667,0.61035900349617,84.02,172.97497606500002,,
+deit_distilled-standard-seed43,deit_distilled,Distilled DeiT-S/16,standard,43,3,1.3784011817084418,80.10666666666667,0.5814183602333068,85.56,259.57245713,,
+deit_distilled-standard-seed43,deit_distilled,Distilled DeiT-S/16,standard,43,4,1.2618183036592272,83.78222222222222,0.5143791406154633,87.12,346.4161076959999,,
+deit_distilled-standard-seed43,deit_distilled,Distilled DeiT-S/16,standard,43,5,1.2004180563078986,86.07333333333334,0.5008319865703583,87.5,433.1932378729999,,
+deit_distilled-standard-seed43,deit_distilled,Distilled DeiT-S/16,standard,43,6,1.1408493053860134,88.00666666666666,0.5139105744361877,87.16,515.8224261859998,,
+deit_distilled-standard-seed43,deit_distilled,Distilled DeiT-S/16,standard,43,7,1.093280386077033,89.74888888888889,0.48809808349609374,87.72,597.2057122559999,,
+deit_distilled-standard-seed43,deit_distilled,Distilled DeiT-S/16,standard,43,8,1.0524332208739386,91.29555555555555,0.4791537547111511,88.68,679.6215752329998,,
+deit_distilled-standard-seed43,deit_distilled,Distilled DeiT-S/16,standard,43,9,1.0247595519913568,92.36,0.4839293314933777,88.42,761.7212217409999,,
+deit_distilled-standard-seed43,deit_distilled,Distilled DeiT-S/16,standard,43,10,0.9961849431355795,93.32666666666667,0.467258198928833,88.8,844.9681168279999,,
+deit_distilled-standard-seed43,deit_distilled,Distilled DeiT-S/16,standard,43,11,0.9716094647195604,94.1288888888889,0.48206463735103605,89.2,929.127655951,,
+deit_distilled-standard-seed43,deit_distilled,Distilled DeiT-S/16,standard,43,12,0.9504373678631253,94.85111111111111,0.47850291204452516,89.26,1012.4233676609998,,
+deit_distilled-standard-seed43,deit_distilled,Distilled DeiT-S/16,standard,43,13,0.9304480115678575,95.58222222222223,0.47780429615974424,89.1,1096.4166046849998,,
+deit_distilled-standard-seed43,deit_distilled,Distilled DeiT-S/16,standard,43,14,0.9158269773907132,96.06888888888889,0.4794639537334442,89.46,1180.3875245379998,,
+deit_distilled-standard-seed43,deit_distilled,Distilled DeiT-S/16,standard,43,15,0.9014540897687277,96.51777777777778,0.4796844746589661,89.76,1263.5969933509998,,
+deit_distilled-standard-seed43,deit_distilled,Distilled DeiT-S/16,standard,43,16,0.8968948127322727,96.67777777777778,0.47670816011428835,89.82,1347.8067791919998,,
+deit_distilled-standard-seed43,deit_distilled,Distilled DeiT-S/16,standard,43,17,0.8847612547768486,97.11777777777777,0.46957905249595644,90.0,1431.7310385709998,,
+deit_distilled-standard-seed43,deit_distilled,Distilled DeiT-S/16,standard,43,18,0.8790466987821791,97.24222222222222,0.47458293809890745,90.0,1514.7739755899997,,
+deit_distilled-standard-seed43,deit_distilled,Distilled DeiT-S/16,standard,43,19,0.8732703325483534,97.46,0.47182472505569456,90.0,1599.1957372069999,,
+deit_distilled-standard-seed43,deit_distilled,Distilled DeiT-S/16,standard,43,20,0.8731164227909513,97.54444444444445,0.47295894360542295,90.16,1683.276820501,,
+deit_distilled-standard-seed44,deit_distilled,Distilled DeiT-S/16,standard,44,1,2.9977581045362682,42.75555555555555,0.8509647571563721,79.68,84.188060908,,
+deit_distilled-standard-seed44,deit_distilled,Distilled DeiT-S/16,standard,44,2,1.5293172847323948,75.35333333333334,0.627078067445755,83.96,169.434934128,,
+deit_distilled-standard-seed44,deit_distilled,Distilled DeiT-S/16,standard,44,3,1.369779471927219,80.32222222222222,0.5474709315299988,85.8,257.82205899200005,,
+deit_distilled-standard-seed44,deit_distilled,Distilled DeiT-S/16,standard,44,4,1.2741754770067002,83.47333333333333,0.529567231798172,86.84,342.73414671300003,,
+deit_distilled-standard-seed44,deit_distilled,Distilled DeiT-S/16,standard,44,5,1.196279378551907,86.05777777777777,0.4961685337543488,88.0,430.40775736300003,,
+deit_distilled-standard-seed44,deit_distilled,Distilled DeiT-S/16,standard,44,6,1.143081200281779,87.99333333333334,0.491046804189682,88.18,516.3663920089999,,
+deit_distilled-standard-seed44,deit_distilled,Distilled DeiT-S/16,standard,44,7,1.0937492934544881,89.76666666666667,0.49740197639465333,87.96,605.995652101,,
+deit_distilled-standard-seed44,deit_distilled,Distilled DeiT-S/16,standard,44,8,1.0561246426052517,91.10444444444444,0.492019255065918,88.48,690.7955250440001,,
+deit_distilled-standard-seed44,deit_distilled,Distilled DeiT-S/16,standard,44,9,1.0237880036248102,92.31777777777778,0.47564145946502684,89.0,776.0408037910001,,
+deit_distilled-standard-seed44,deit_distilled,Distilled DeiT-S/16,standard,44,10,0.995832539876302,93.32222222222222,0.48339254236221313,89.46,864.8096337790003,,
+deit_distilled-standard-seed44,deit_distilled,Distilled DeiT-S/16,standard,44,11,0.9728447216245864,94.0111111111111,0.48064363379478453,88.82,953.6293515720001,,
+deit_distilled-standard-seed44,deit_distilled,Distilled DeiT-S/16,standard,44,12,0.9496805101182726,94.87333333333333,0.4762904680252075,89.54,1041.7340647820001,,
+deit_distilled-standard-seed44,deit_distilled,Distilled DeiT-S/16,standard,44,13,0.9323136636734008,95.46,0.48379494886398317,89.32,1133.8188962220004,,
+deit_distilled-standard-seed44,deit_distilled,Distilled DeiT-S/16,standard,44,14,0.9161827259805467,96.04444444444445,0.4832069589614868,89.44,1222.4505687970002,,
+deit_distilled-standard-seed44,deit_distilled,Distilled DeiT-S/16,standard,44,15,0.9069438054826524,96.36222222222223,0.4768580966949463,89.66,1311.3063252710003,,
+deit_distilled-standard-seed44,deit_distilled,Distilled DeiT-S/16,standard,44,16,0.8942975118849013,96.7311111111111,0.475810445690155,89.7,1396.1095069200005,,
+deit_distilled-standard-seed44,deit_distilled,Distilled DeiT-S/16,standard,44,17,0.8902478694915772,96.85333333333334,0.47872688302993777,89.88,1483.4896943250005,,
+deit_distilled-standard-seed44,deit_distilled,Distilled DeiT-S/16,standard,44,18,0.8789131625493367,97.26222222222222,0.480087397480011,89.98,1570.4980786050005,,
+deit_distilled-standard-seed44,deit_distilled,Distilled DeiT-S/16,standard,44,19,0.8805714562416077,97.26444444444445,0.47695729274749754,90.3,1657.6466858940005,,
+deit_distilled-standard-seed44,deit_distilled,Distilled DeiT-S/16,standard,44,20,0.8774085379070706,97.32444444444444,0.4765383381843567,90.28,1741.7318086770006,,
+deit_distilled-hard_kd-seed42,deit_distilled,Distilled DeiT-S/16,hard_kd,42,1,2.891303874291314,43.58444444444444,0.7823614247322083,79.72,129.67461937000002,3.0271674709743923,2.7554402774386935
+deit_distilled-hard_kd-seed42,deit_distilled,Distilled DeiT-S/16,hard_kd,42,2,1.1876610801908705,75.83555555555556,0.5414647546768189,84.16,252.34651784900004,1.5427573557959662,0.8325648040771484
+deit_distilled-hard_kd-seed42,deit_distilled,Distilled DeiT-S/16,hard_kd,42,3,1.0061806173854404,80.2088888888889,0.4768359443664551,85.9,375.692797325,1.3834636107232836,0.6288976232846578
+deit_distilled-hard_kd-seed42,deit_distilled,Distilled DeiT-S/16,hard_kd,42,4,0.8947540015962389,83.70222222222222,0.4441151406288147,86.78,500.362236125,1.2784623081419204,0.5110456932703654
+deit_distilled-hard_kd-seed42,deit_distilled,Distilled DeiT-S/16,hard_kd,42,5,0.8228176123407152,86.06222222222222,0.42421336784362795,87.16,625.7417416510001,1.2116484966066148,0.43398672875298394
+deit_distilled-hard_kd-seed42,deit_distilled,Distilled DeiT-S/16,hard_kd,42,6,0.763524847624037,88.20666666666666,0.409509094285965,87.98,751.6183524050001,1.1525930098109776,0.37445668535232546
+deit_distilled-hard_kd-seed42,deit_distilled,Distilled DeiT-S/16,hard_kd,42,7,0.7157429592556424,89.65555555555555,0.4245356678009033,87.68,877.2789197310001,1.109786534108056,0.3216993831422594
+deit_distilled-hard_kd-seed42,deit_distilled,Distilled DeiT-S/16,hard_kd,42,8,0.6763199755350748,90.95555555555555,0.4036053611040115,88.4,1000.958533601,1.071368237792121,0.28127171156141495
+deit_distilled-hard_kd-seed42,deit_distilled,Distilled DeiT-S/16,hard_kd,42,9,0.6393239784028795,92.3,0.4089566869735718,88.54,1125.552717204,1.037098964267307,0.2415489924536811
+deit_distilled-hard_kd-seed42,deit_distilled,Distilled DeiT-S/16,hard_kd,42,10,0.6139209205097622,93.15555555555555,0.4001792684316635,88.52,1249.907995705,1.0112061354955038,0.2166357067373064
+deit_distilled-hard_kd-seed42,deit_distilled,Distilled DeiT-S/16,hard_kd,42,11,0.5895874520301819,94.02222222222223,0.396558064365387,89.16,1375.8699237350002,0.9875530295477973,0.19162187317477333
+deit_distilled-hard_kd-seed42,deit_distilled,Distilled DeiT-S/16,hard_kd,42,12,0.573295332898034,94.64666666666666,0.3879859871864319,88.9,1499.8493775440002,0.9677312017228868,0.17885946276452805
+deit_distilled-hard_kd-seed42,deit_distilled,Distilled DeiT-S/16,hard_kd,42,13,0.5550679882844289,95.28,0.39951880366802217,89.2,1625.9221191220001,0.9492251159667969,0.16091086073981392
+deit_distilled-hard_kd-seed42,deit_distilled,Distilled DeiT-S/16,hard_kd,42,14,0.5398053416570028,95.70666666666666,0.3892052047252655,89.5,1751.1481698890002,0.9340377052730984,0.14557297928598192
+deit_distilled-hard_kd-seed42,deit_distilled,Distilled DeiT-S/16,hard_kd,42,15,0.5345867886543274,96.03333333333333,0.38163992581367495,89.64,1873.1241391210003,0.9239453143861559,0.145228263314565
+deit_distilled-hard_kd-seed42,deit_distilled,Distilled DeiT-S/16,hard_kd,42,16,0.5279601024097866,96.36222222222223,0.36974528813362123,90.16,1996.8540595680006,0.9125766407118904,0.14334356382422977
+deit_distilled-hard_kd-seed42,deit_distilled,Distilled DeiT-S/16,hard_kd,42,17,0.5234279460430146,96.59777777777778,0.36778093738555906,90.16,2121.2349653710007,0.9053525572141011,0.1415033326705297
+deit_distilled-hard_kd-seed42,deit_distilled,Distilled DeiT-S/16,hard_kd,42,18,0.5371519863181644,96.66888888888889,0.36090206460952756,90.32,2244.1225055700006,0.9007420391188727,0.17356193328963385
+deit_distilled-hard_kd-seed42,deit_distilled,Distilled DeiT-S/16,hard_kd,42,19,0.5340569213761224,96.74666666666667,0.359119135427475,90.38,2367.225204148001,0.8981096373028226,0.17000420770910052
+deit_distilled-hard_kd-seed42,deit_distilled,Distilled DeiT-S/16,hard_kd,42,20,0.5313887091318766,96.80444444444444,0.35858049035072326,90.46,2489.6353192010006,0.8954598051495022,0.16731761072741613
+deit_distilled-hard_kd-seed43,deit_distilled,Distilled DeiT-S/16,hard_kd,43,1,2.9287832387924193,41.54666666666667,0.8195689846992492,79.32,135.76922614600002,3.0824710364023846,2.7750954484727646
+deit_distilled-hard_kd-seed43,deit_distilled,Distilled DeiT-S/16,hard_kd,43,2,1.1976113839891223,75.24888888888889,0.5429152712345123,84.22,259.0439842480001,1.5526228285047743,0.842599940999349
+deit_distilled-hard_kd-seed43,deit_distilled,Distilled DeiT-S/16,hard_kd,43,3,1.0213200540648566,79.89111111111112,0.4906630965709686,85.7,382.164758299,1.3959846112993028,0.6466554945415921
+deit_distilled-hard_kd-seed43,deit_distilled,Distilled DeiT-S/16,hard_kd,43,4,0.8950510659959581,83.86888888888889,0.43704870057106016,86.72,505.474577456,1.2768856774224175,0.5132164569642809
+deit_distilled-hard_kd-seed43,deit_distilled,Distilled DeiT-S/16,hard_kd,43,5,0.8310969106144376,86.01333333333334,0.4212643687725067,87.84,628.704182082,1.2137438252555,0.4484499966568417
+deit_distilled-hard_kd-seed43,deit_distilled,Distilled DeiT-S/16,hard_kd,43,6,0.7742280316564771,87.83777777777777,0.43822868962287903,86.8,751.8393803299999,1.156732429631551,0.3917236333211263
+deit_distilled-hard_kd-seed43,deit_distilled,Distilled DeiT-S/16,hard_kd,43,7,0.728911703130934,89.42222222222222,0.4072139594554901,87.86,874.911157876,1.1124767728381686,0.34534663320117526
+deit_distilled-hard_kd-seed43,deit_distilled,Distilled DeiT-S/16,hard_kd,43,8,0.6893146724806891,90.94444444444444,0.4018845633506775,88.28,998.1578501330001,1.0699806736098396,0.3086486725065443
+deit_distilled-hard_kd-seed43,deit_distilled,Distilled DeiT-S/16,hard_kd,43,9,0.6664539758152432,91.9088888888889,0.4044099115848541,88.86,1121.247578128,1.0472418416129219,0.285666108724806
+deit_distilled-hard_kd-seed43,deit_distilled,Distilled DeiT-S/16,hard_kd,43,10,0.6395534198760986,92.82222222222222,0.39913368508815766,88.7,1244.5626637179998,1.0193096835242377,0.2597971566412184
+deit_distilled-hard_kd-seed43,deit_distilled,Distilled DeiT-S/16,hard_kd,43,11,0.6162675294452243,93.65777777777778,0.38752014577388766,89.04,1367.7723100969997,0.9938013131247626,0.23873374446762932
+deit_distilled-hard_kd-seed43,deit_distilled,Distilled DeiT-S/16,hard_kd,43,12,0.5965302309354147,94.24888888888889,0.38459586548805236,89.16,1490.8595400249997,0.9713725334591335,0.22168792927530076
+deit_distilled-hard_kd-seed43,deit_distilled,Distilled DeiT-S/16,hard_kd,43,13,0.5757942104657491,94.98,0.3848479796409607,89.1,1613.8521726029996,0.9526513829125298,0.19893703789446088
+deit_distilled-hard_kd-seed43,deit_distilled,Distilled DeiT-S/16,hard_kd,43,14,0.5671409094863468,95.52,0.3798333389759064,89.48,1737.0080181519995,0.9361915278858609,0.19809029173056283
+deit_distilled-hard_kd-seed43,deit_distilled,Distilled DeiT-S/16,hard_kd,43,15,0.554848009051217,95.81777777777778,0.3778343957185745,89.52,1860.1085322999993,0.9253742494795058,0.18432176601621839
+deit_distilled-hard_kd-seed43,deit_distilled,Distilled DeiT-S/16,hard_kd,43,16,0.5501120466444227,96.07555555555555,0.3691397216796875,89.62,1983.0631068049997,0.9187914066208733,0.18143268666267395
+deit_distilled-hard_kd-seed43,deit_distilled,Distilled DeiT-S/16,hard_kd,43,17,0.5379814247820113,96.32888888888888,0.3659845009803772,90.0,2106.316168004,0.9061751669036018,0.16978768435054356
+deit_distilled-hard_kd-seed43,deit_distilled,Distilled DeiT-S/16,hard_kd,43,18,0.5337994150055779,96.59777777777778,0.3606197656154633,89.7,2229.3478398089997,0.9013160623868306,0.1662827663845486
+deit_distilled-hard_kd-seed43,deit_distilled,Distilled DeiT-S/16,hard_kd,43,19,0.528296198993259,96.69333333333333,0.3612505676746368,89.9,2352.5222674789993,0.8955731566111247,0.16101924628681608
+deit_distilled-hard_kd-seed43,deit_distilled,Distilled DeiT-S/16,hard_kd,43,20,0.5296659093644884,96.8488888888889,0.361381387758255,90.06,2475.6306675059996,0.8955944094869825,0.16373740825917985
+deit_distilled-hard_kd-seed44,deit_distilled,Distilled DeiT-S/16,hard_kd,44,1,2.9172539457745024,42.32666666666667,0.8037968199729919,80.0,129.24327998899997,3.0575056590186227,2.777002233039008
+deit_distilled-hard_kd-seed44,deit_distilled,Distilled DeiT-S/16,hard_kd,44,2,1.199185688633389,75.25555555555556,0.5595191810131073,83.54,250.98709217099994,1.5534146911621094,0.8449566852993435
+deit_distilled-hard_kd-seed44,deit_distilled,Distilled DeiT-S/16,hard_kd,44,3,1.0142347576565212,80.26666666666667,0.4710086986064911,85.78,373.29275533500004,1.3892860037909613,0.6391835083431667
+deit_distilled-hard_kd-seed44,deit_distilled,Distilled DeiT-S/16,hard_kd,44,4,0.9099242265171474,83.39333333333333,0.4479987185955048,86.38,496.21334167299995,1.28951179945204,0.5303366531133652
+deit_distilled-hard_kd-seed44,deit_distilled,Distilled DeiT-S/16,hard_kd,44,5,0.8317347321404351,86.01333333333334,0.4168547761917114,87.74,619.0086425959998,1.2139732154740228,0.44949625146653915
+deit_distilled-hard_kd-seed44,deit_distilled,Distilled DeiT-S/16,hard_kd,44,6,0.7770806811332702,87.74888888888889,0.41382922625541685,88.04,741.9203825609998,1.1590111846076117,0.3951501806047228
+deit_distilled-hard_kd-seed44,deit_distilled,Distilled DeiT-S/16,hard_kd,44,7,0.7310264108763801,89.63777777777777,0.41661795749664304,87.6,864.9877230799998,1.111869426557753,0.3501833961725235
+deit_distilled-hard_kd-seed44,deit_distilled,Distilled DeiT-S/16,hard_kd,44,8,0.6927898583730062,90.92666666666666,0.4051929706573486,88.38,988.2038016439999,1.0743477234734429,0.31123199504216514
+deit_distilled-hard_kd-seed44,deit_distilled,Distilled DeiT-S/16,hard_kd,44,9,0.6638727407879299,91.84222222222222,0.37963612928390505,89.06,1111.5598942339998,1.0453598051495023,0.28238567671246
+deit_distilled-hard_kd-seed44,deit_distilled,Distilled DeiT-S/16,hard_kd,44,10,0.6362758345921834,92.82888888888888,0.39949473524093626,88.9,1234.8359054669997,1.0173541036817764,0.25519756574630736
+deit_distilled-hard_kd-seed44,deit_distilled,Distilled DeiT-S/16,hard_kd,44,11,0.6193733670075734,93.57777777777778,0.385602642250061,89.36,1357.7475060469997,0.9946745898988512,0.2440721433401108
+deit_distilled-hard_kd-seed44,deit_distilled,Distilled DeiT-S/16,hard_kd,44,12,0.5950572917408413,94.30222222222223,0.39176607789993284,89.08,1481.1199927039995,0.9713103383594089,0.21880424451298183
+deit_distilled-hard_kd-seed44,deit_distilled,Distilled DeiT-S/16,hard_kd,44,13,0.5815811908192104,94.81555555555556,0.38579091906547547,89.26,1604.1639940099997,0.9551089630974664,0.20805342066287993
+deit_distilled-hard_kd-seed44,deit_distilled,Distilled DeiT-S/16,hard_kd,44,14,0.562831224515703,95.49333333333334,0.38141796951293944,89.42,1727.3423413549997,0.9377868717829386,0.18787557439009347
+deit_distilled-hard_kd-seed44,deit_distilled,Distilled DeiT-S/16,hard_kd,44,15,0.5554835402594672,95.67111111111112,0.3635530668258667,90.18,1850.2814545489998,0.9289417793273925,0.18202530020078023
+deit_distilled-hard_kd-seed44,deit_distilled,Distilled DeiT-S/16,hard_kd,44,16,0.5463188575108846,96.04888888888888,0.356517165517807,90.16,1973.2980403569998,0.9175062625249227,0.17513145196172927
+deit_distilled-hard_kd-seed44,deit_distilled,Distilled DeiT-S/16,hard_kd,44,17,0.5405616811116536,96.2688888888889,0.3586801685333252,90.04,2096.783539895,0.9109745461569893,0.17014881643454233
+deit_distilled-hard_kd-seed44,deit_distilled,Distilled DeiT-S/16,hard_kd,44,18,0.5333575668652852,96.67333333333333,0.3591350576877594,90.08,2220.469081041,0.8992170536465115,0.16749807998869154
+deit_distilled-hard_kd-seed44,deit_distilled,Distilled DeiT-S/16,hard_kd,44,19,0.5337492761823867,96.54222222222222,0.35689546976089476,90.28,2344.1217141380007,0.9018334956169128,0.1656650566233529
+deit_distilled-hard_kd-seed44,deit_distilled,Distilled DeiT-S/16,hard_kd,44,20,0.533892258728875,96.57555555555555,0.3583702641963959,90.06,2467.4307780010004,0.8999511775864495,0.1678333395799001

chapter-04-efficient-and-scalable-vision-transformers/results/inference_benchmark.csv

@@ -0,0 +1,25 @@
+batch_size,latency_mean_ms,latency_median_ms,latency_p90_ms,latency_std_ms,model,peak_memory_mb,precision,repeats,samples,status,throughput_images_per_second,timed_iterations,warmup_iterations
+1,4.777800642490387,4.769615888595581,4.826025533676148,0.09854292782443023,convnext,132.1884765625,fp32,5,1000,complete,209.3013239411259,200,50
+64,127.90953128051758,128.02047729492188,132.2467269897461,3.4320032409674415,convnext,962.6767578125,fp32,5,1000,complete,500.3536433859804,200,50
+1,5.376872823238373,5.378112077713013,5.495808124542236,0.11512835348717641,convnext,183.4208984375,fp16,5,1000,complete,185.98170960601627,200,50
+64,57.763763633728026,57.5544319152832,59.53341598510742,1.2430091115071689,convnext,728.7607421875,fp16,5,1000,complete,1107.9610464064474,200,50
+1,4.280692895412445,4.261360168457031,4.343808174133301,0.16694712271134712,vit,97.1484375,fp32,5,1000,complete,233.60704083016213,200,50
+64,106.9532296295166,106.99007797241211,108.48696365356446,1.1434555353430906,vit,351.763671875,fp32,5,1000,complete,598.3924021901391,200,50
+1,4.69389853143692,4.659152030944824,4.780019092559815,0.16487908037721768,vit,137.90283203125,fp16,5,1000,complete,213.04252601597565,200,50
+64,32.072251497268674,32.125953674316406,33.052775573730464,0.8095062193205872,vit,320.05126953125,fp16,5,1000,complete,1995.4944543089014,200,50
+1,8.11762380886078,8.089599609375,8.21555233001709,0.07401799461648069,swin,134.0771484375,fp32,5,1000,complete,123.18875862520896,200,50
+64,178.14805532836914,178.41305541992188,180.40453033447267,1.9795728141419107,swin,1077.705078125,fp32,5,1000,complete,359.251746430983,200,50
+1,9.13337626171112,9.041919708251953,9.377792358398438,0.29909913260705207,swin,185.33349609375,fp16,5,1000,complete,109.4885364782565,200,50
+64,84.70931391143799,84.73241424560547,86.2045181274414,1.2024556603816736,swin,970.5625,fp16,5,1000,complete,755.5249481410133,200,50
+1,5.772215712070465,5.725183963775635,5.933670282363892,0.18675654583815726,efficientformer,60.97119140625,fp32,5,1000,complete,173.2436987600564,200,50
+64,56.72956251525879,56.71116638183594,57.0277889251709,0.22907337112652218,efficientformer,497.74072265625,fp32,5,1000,complete,1128.1595902098777,200,50
+1,6.20901661491394,6.112767934799194,6.5593345165252686,0.30308211309436084,efficientformer,80.21435546875,fp16,5,1000,complete,161.05609986580146,200,50
+64,32.634319339752196,32.62673568725586,32.93388748168945,0.21541325392113395,efficientformer,354.11083984375,fp16,5,1000,complete,1961.125627708158,200,50
+1,4.368596731185913,4.306943893432617,4.604022598266601,0.16726820626053793,deit,97.1484375,fp32,5,1000,complete,228.9064570463424,200,50
+64,106.82791548156739,106.8067855834961,107.79443359375,0.8205535043948153,deit,351.763671875,fp32,5,1000,complete,599.094344502518,200,50
+1,4.722966948986054,4.696063995361328,4.82521595954895,0.09960610584523705,deit,137.90283203125,fp16,5,1000,complete,211.73131440495982,200,50
+64,32.135718923568724,32.168447494506836,33.121176147460936,0.8138286399443037,deit,320.05126953125,fp16,5,1000,complete,1991.5533911725133,200,50
+1,4.383408827781677,4.355168104171753,4.442111968994141,0.20651985327066089,deit_distilled,97.30419921875,fp32,5,1000,complete,228.13295297990092,200,50
+64,105.38310792541503,105.41926574707031,106.30379638671876,0.7789229665081872,deit_distilled,352.50732421875,fp32,5,1000,complete,607.3079572230498,200,50
+1,4.723058529853821,4.704256057739258,4.769792079925537,0.15652674115081888,deit_distilled,138.13818359375,fp16,5,1000,complete,211.72720890057442,200,50
+64,32.341811212539675,32.4213752746582,33.34256629943848,0.8330030872120695,deit_distilled,321.55615234375,fp16,5,1000,complete,1978.8625806827326,200,50

chapter-04-efficient-and-scalable-vision-transformers/results/model_profiles.csv

@@ -0,0 +1,7 @@
+checkpoint,display_name,error,expected_gmacs_from_model_card,expected_parameters_m,measured_gmacs,measured_macs,model,parameters,representative_run,unsupported_operators
+convnext_tiny.fb_in1k,ConvNeXt-T,,4.5,28.6,4.469746176,4469746176,convnext,27897028,convnext-standard-seed42,"{'aten::add': 18, 'aten::gelu': 18, 'aten::mul': 18}"
+vit_small_patch16_224.augreg_in1k,ViT-S/16,,4.3,22.1,4.25032896,4250328960,vit,21704164,vit-standard-seed42,"{'aten::add': 25, 'aten::gelu': 12, 'aten::scaled_dot_product_attention': 12}"
+swin_tiny_patch4_window7_224.ms_in1k,Swin-T,,4.5,28.3,4.508503296,4508503296,swin,27596254,swin-standard-seed42,"{'aten::add': 41, 'aten::bernoulli_': 22, 'aten::div_': 22, 'aten::gelu': 12, 'aten::mean': 1, 'aten::mul': 34, 'aten::pad': 15, 'aten::rsub': 24, 'aten::softmax': 12}"
+efficientformer_l1.snap_dist_in1k,EfficientFormer-L1,,1.3,12.3,1.317055936,1317055936,efficientformer,11481728,efficientformer-standard-seed42,"{'aten::add': 32, 'aten::add_': 33, 'aten::avg_pool2d': 14, 'aten::div': 1, 'aten::gelu': 15, 'aten::mean': 1, 'aten::mul': 31, 'aten::softmax': 1, 'aten::sub': 14}"
+deit_small_patch16_224.fb_in1k,DeiT-S/16,,4.6,22.1,4.25032896,4250328960,deit,21704164,deit-standard-seed42,"{'aten::add': 25, 'aten::gelu': 12, 'aten::scaled_dot_product_attention': 12}"
+deit_small_distilled_patch16_224.fb_in1k,Distilled DeiT-S/16,,4.6,22.4,4.271649024,4271649024,deit_distilled,21743432,deit_distilled-standard-seed42,"{'aten::add': 26, 'aten::div': 1, 'aten::gelu': 12, 'aten::scaled_dot_product_attention': 12}"

chapter-04-efficient-and-scalable-vision-transformers/results/report.md

@@ -0,0 +1,57 @@
+# Full vision-transformer benchmark
+
+Configuration fingerprint: `7b87d57ae5546600c460245a81431288db34c3e40a44c2e83da01acc4160d731`.
+
+## Main comparison
+
+| Model | Training | Seeds | Test top-1 (%) | Parameters (M) | Images/s | Timing setting |
+| --- | --- | --- | --- | --- | --- | --- |
+| ConvNeXt-T | standard | 1 | 89.78 | 27.90 | 1107.96 | fp16, batch 64 |
+| ViT-S/16 | standard | 1 | 89.22 | 21.70 | 1995.49 | fp16, batch 64 |
+| Swin-T | standard | 1 | 89.54 | 27.60 | 755.52 | fp16, batch 64 |
+| EfficientFormer-L1 | standard | 1 | 86.83 | 11.48 | 1961.13 | fp16, batch 64 |
+| DeiT-S/16 | standard | 3 | 88.86 ± 0.03 | 21.70 | 1991.55 | fp16, batch 64 |
+| Distilled DeiT-S/16 | standard | 3 | 89.46 ± 0.27 | 21.74 | 1978.86 | fp16, batch 64 |
+| Distilled DeiT-S/16 | hard_kd | 3 | 89.72 ± 0.11 | 21.74 | 1978.86 | fp16, batch 64 |
+
+A `±` value is the sample standard deviation across seeds; single-seed rows have no
+uncertainty estimate. Throughput is an architecture property here, so the same timing is
+reused for standard and hard-distilled runs of the same Distilled DeiT model.
+
+## Isolated training-time distillation gain
+
+This paired comparison holds the Distilled DeiT architecture and seed fixed. It isolates
+the effect of using the ConvNeXt teacher during fine-tuning; comparing ordinary DeiT with
+Distilled DeiT would also change the checkpoint and architecture.
+
+| Seed | Standard top-1 | Hard-KD top-1 | Gain (points) |
+| --- | --- | --- | --- |
+| 42 | 89.74 | 89.83 | 0.09 |
+| 43 | 89.21 | 89.61 | 0.40 |
+| 44 | 89.42 | 89.72 | 0.30 |
+
+## Hardware protocol
+
+Device: **NVIDIA L4**; PyTorch 2.9.1+cu128; precision and batch size are shown per row. Timing excludes data loading and includes synchronized model forward passes.
+
+The operation counter reports MAC-style fvcore operations (one multiply-add is counted as
+one), plus any unsupported operators in `model_profiles.csv`. Peak memory is allocator
+memory, not whole-system power or memory use.
+
+## Training behavior and trade-offs
+
+![Validation Accuracy](figures/validation_accuracy.png)
+![Distillation Gain](figures/distillation_gain.png)
+![Accuracy Throughput](figures/accuracy_throughput.png)
+
+## Reproducibility files
+
+- `run_results.csv`: one row per seed and training mode
+- `epoch_metrics.csv`: learning curves and elapsed training time
+- `aggregate_results.csv`: grouped accuracy statistics
+- `distillation_gains.csv`: paired hard-distillation differences
+- `model_profiles.csv` and `inference_benchmark.csv`: created after hardware benchmarking
+- each run directory: resolved configuration, checkpoints, predictions, and environment data
+
+Select checkpoints with validation accuracy only. The official CIFAR-100 test split is
+evaluated after training and is not used for early stopping or hyperparameter selection.

chapter-04-efficient-and-scalable-vision-transformers/results/report_summary.json

@@ -0,0 +1,126 @@
+{
+  "aggregate": [
+    {
+      "best_validation_top1_mean": 89.86,
+      "display_name": "ConvNeXt-T",
+      "mode": "standard",
+      "model": "convnext",
+      "parameters_m": 27.897028,
+      "seeds": 1,
+      "test_top1_mean": 89.78,
+      "test_top1_std": NaN,
+      "test_top5_mean": 98.86,
+      "training_peak_memory_mb": 4245.66796875,
+      "training_seconds_mean": 2789.4574548180008
+    },
+    {
+      "best_validation_top1_mean": 89.6,
+      "display_name": "ViT-S/16",
+      "mode": "standard",
+      "model": "vit",
+      "parameters_m": 21.704164,
+      "seeds": 1,
+      "test_top1_mean": 89.22,
+      "test_top1_std": NaN,
+      "test_top5_mean": 98.32,
+      "training_peak_memory_mb": 2563.0244140625,
+      "training_seconds_mean": 1673.357545405
+    },
+    {
+      "best_validation_top1_mean": 89.54,
+      "display_name": "Swin-T",
+      "mode": "standard",
+      "model": "swin",
+      "parameters_m": 27.596254,
+      "seeds": 1,
+      "test_top1_mean": 89.54,
+      "test_top1_std": NaN,
+      "test_top5_mean": 98.65,
+      "training_peak_memory_mb": 4843.58154296875,
+      "training_seconds_mean": 3631.1184958180006
+    },
+    {
+      "best_validation_top1_mean": 86.86,
+      "display_name": "EfficientFormer-L1",
+      "mode": "standard",
+      "model": "efficientformer",
+      "parameters_m": 11.481728,
+      "seeds": 1,
+      "test_top1_mean": 86.83,
+      "test_top1_std": NaN,
+      "test_top5_mean": 98.26,
+      "training_peak_memory_mb": 2633.85888671875,
+      "training_seconds_mean": 1701.0025390570008
+    },
+    {
+      "best_validation_top1_mean": 89.71999999999998,
+      "display_name": "DeiT-S/16",
+      "mode": "standard",
+      "model": "deit",
+      "parameters_m": 21.704164,
+      "seeds": 3,
+      "test_top1_mean": 88.86,
+      "test_top1_std": 0.030000000000001137,
+      "test_top5_mean": 98.19666666666667,
+      "training_peak_memory_mb": 2563.0244140625,
+      "training_seconds_mean": 1676.6830429646668
+    },
+    {
+      "best_validation_top1_mean": 90.16666666666667,
+      "display_name": "Distilled DeiT-S/16",
+      "mode": "standard",
+      "model": "deit_distilled",
+      "parameters_m": 21.743432,
+      "seeds": 3,
+      "test_top1_mean": 89.45666666666666,
+      "test_top1_std": 0.2668957349478132,
+      "test_top5_mean": 98.26666666666667,
+      "training_peak_memory_mb": 2566.7571614583335,
+      "training_seconds_mean": 1701.7341956493337
+    },
+    {
+      "best_validation_top1_mean": 90.26666666666667,
+      "display_name": "Distilled DeiT-S/16",
+      "mode": "hard_kd",
+      "model": "deit_distilled",
+      "parameters_m": 21.743432,
+      "seeds": 3,
+      "test_top1_mean": 89.71999999999998,
+      "test_top1_std": 0.10999999999999943,
+      "test_top5_mean": 98.88999999999999,
+      "training_peak_memory_mb": 3051.60888671875,
+      "training_seconds_mean": 2477.565588236
+    }
+  ],
+  "completed_runs": 13,
+  "config_fingerprint": "7b87d57ae5546600c460245a81431288db34c3e40a44c2e83da01acc4160d731",
+  "distillation_gains": [
+    {
+      "gain_percentage_points": 0.09000000000000341,
+      "hard_kd_top1": 89.83,
+      "seed": 42,
+      "standard_top1": 89.74
+    },
+    {
+      "gain_percentage_points": 0.4000000000000057,
+      "hard_kd_top1": 89.61,
+      "seed": 43,
+      "standard_top1": 89.21
+    },
+    {
+      "gain_percentage_points": 0.29999999999999716,
+      "hard_kd_top1": 89.72,
+      "seed": 44,
+      "standard_top1": 89.42
+    }
+  ],
+  "figures": [
+    "validation_accuracy.png",
+    "distillation_gain.png",
+    "accuracy_throughput.png"
+  ],
+  "missing_runs": [],
+  "preset": "full",
+  "schema_version": 1,
+  "tutorial_only": false
+}

chapter-04-efficient-and-scalable-vision-transformers/results/run_results.csv

@@ -0,0 +1,14 @@
+run_key,model,display_name,mode,seed,best_epoch,best_validation_top1,test_top1,test_top5,parameters,parameters_m,training_seconds,training_peak_memory_mb,run_directory
+convnext-standard-seed42,convnext,ConvNeXt-T,standard,42,17,89.86,89.78,98.86,27897028,27.897028,2789.4574548180008,4245.66796875,/vol/artifacts/full/convnext-standard-seed42-7b87d57ae5
+vit-standard-seed42,vit,ViT-S/16,standard,42,20,89.6,89.22,98.32,21704164,21.704164,1673.357545405,2563.0244140625,/vol/artifacts/full/vit-standard-seed42-7b87d57ae5
+swin-standard-seed42,swin,Swin-T,standard,42,20,89.54,89.54,98.65,27596254,27.596254,3631.1184958180006,4843.58154296875,/vol/artifacts/full/swin-standard-seed42-7b87d57ae5
+efficientformer-standard-seed42,efficientformer,EfficientFormer-L1,standard,42,20,86.86,86.83,98.26,11481728,11.481728,1701.0025390570008,2633.85888671875,/vol/artifacts/full/efficientformer-standard-seed42-7b87d57ae5
+deit-standard-seed42,deit,DeiT-S/16,standard,42,20,89.8,88.89,98.09,21704164,21.704164,1639.156589266,2563.0244140625,/vol/artifacts/full/deit-standard-seed42-7b87d57ae5
+deit-standard-seed43,deit,DeiT-S/16,standard,43,20,89.7,88.86,98.22,21704164,21.704164,1688.338380226,2563.0244140625,/vol/artifacts/full/deit-standard-seed43-7b87d57ae5
+deit-standard-seed44,deit,DeiT-S/16,standard,44,18,89.66,88.83,98.28,21704164,21.704164,1702.554159402,2563.0244140625,/vol/artifacts/full/deit-standard-seed44-7b87d57ae5
+deit_distilled-standard-seed42,deit_distilled,Distilled DeiT-S/16,standard,42,19,90.04,89.74,98.25,21743432,21.743432,1680.1939577700005,2594.0869140625,/vol/artifacts/full/deit_distilled-standard-seed42-7b87d57ae5
+deit_distilled-standard-seed43,deit_distilled,Distilled DeiT-S/16,standard,43,20,90.16,89.21,98.24,21743432,21.743432,1683.276820501,2594.0869140625,/vol/artifacts/full/deit_distilled-standard-seed43-7b87d57ae5
+deit_distilled-standard-seed44,deit_distilled,Distilled DeiT-S/16,standard,44,19,90.3,89.42,98.31,21743432,21.743432,1741.7318086770006,2512.09765625,/vol/artifacts/full/deit_distilled-standard-seed44-7b87d57ae5
+deit_distilled-hard_kd-seed42,deit_distilled,Distilled DeiT-S/16,hard_kd,42,20,90.46,89.83,98.88,21743432,21.743432,2489.6353192010006,3051.60888671875,/vol/artifacts/full/deit_distilled-hard_kd-seed42-7b87d57ae5
+deit_distilled-hard_kd-seed43,deit_distilled,Distilled DeiT-S/16,hard_kd,43,20,90.06,89.61,98.94,21743432,21.743432,2475.6306675059996,3051.60888671875,/vol/artifacts/full/deit_distilled-hard_kd-seed43-7b87d57ae5
+deit_distilled-hard_kd-seed44,deit_distilled,Distilled DeiT-S/16,hard_kd,44,19,90.28,89.72,98.85,21743432,21.743432,2467.4307780010004,3051.60888671875,/vol/artifacts/full/deit_distilled-hard_kd-seed44-7b87d57ae5