ihor-umanets/pixelsprite-canvas-v2

pixelsprite canvas v2: text to 16×16 / 24×24 / 32×32 pixel sprites

A small autoregressive transformer (39.1M parameters) that draws pixel-art sprites from a caption. It works in English, Russian and Ukrainian. It can also colour a silhouette you give it.

• Sizes: 16×16 and 32×32 (trained), plus 24×24 drawn as a window in the middle of the 32×32 canvas (not trained separately).
• Pixel by pixel: the model writes the sprite left to right and top to bottom. For every pixel it picks one of 1,024 palette colours or "transparent". This is the next-token method of text models, applied to pixels.
• Multilingual captions: they go through the frozen paraphrase-multilingual-MiniLM-L12-v2 encoder.
• Separate guidance: classifier-free guidance is applied separately to the shape (transparent or not) and to the colour. By default shape is 1 and colour is 2.
• Optional silhouette: give a silhouette and only the colours are chosen. The outline is kept exactly.

Try it in your browser: https://umanets.github.io/pixelsprite/ (no install; the model runs on your GPU through WebGPU, or on the CPU).

Code (inference, CLI, local web UI, learning notebooks): https://github.com/umanets/pixelsprite

Quick start

pip install "git+https://github.com/umanets/pixelsprite"
pixelsprite "frost sword" --size 16 --n 16

from pixelsprite import SpritePipeline

pipe = SpritePipeline.from_pretrained("ihor-umanets/pixelsprite-canvas-v2")
res = pipe.generate("a teal cookie jar", n=8, size=32, seed=0)   # g_shape=1.0, g_colour=2.0 by default
res.save("outputs/jars")                                          # PNGs (alpha = silhouette) + sheet.png
recolour = pipe.generate("ice potion", n=4, size=32, mask=res.index[0] >= 0)

Files in this repo:

• model.safetensors: weights only, float32.
• config.json: architecture, text encoder, special tokens and sampling defaults.
• palette.json: the 1,024 sRGB colours.
• onnx/: the same weights for the browser demo (onnxruntime-web). decoder_static.onnx (used by the demo) and decoder.onnx are float32 graphs of the transformer blocks and the head, one token per call with a key/value cache: a fixed-size cache for the whole sprite, written at pos, in the first; a growing one in the second. Both take their large weights as external data, shipped once as float16 in decoder.f16 (expanded to float32 in the browser; logits change by less than 0.005). embeddings.bin / embeddings.json hold the input tables.

16×16	24×24 (window)

Model

Architecture	Decoder-only transformer: 12 layers, width 512, 8 heads, MLP 2048, pre-norm, GELU, dropout 0.1
Sequence	[text vector, canvas size, S silhouette-row tokens, S×S pixel tokens]. Pixel token = previous pixel + row + column + edge flag.
Output per pixel	1,024 colours + EMPTY (transparent)
Silhouette input	Row bits of the silhouette, or "unknown" (given in half of the training examples)
Text	Frozen multilingual MiniLM, 384-d L2-normalised vector. A learned null vector stands for "no text". 10% of training examples drop the text, which makes guidance possible.
Parameters	39,134,721

Training

• Data: 1,431,684 training sprites.
  • 1,264,518 at 16×16 from Angais/pixel-art-atlas-1.4m.
  • 167,166 at 32×32 from azyg0s/pixelart-172k.
  • Captions in English. The text is either the short subject or the full caption, half the time each.
  • Splits are by unique silhouette, so test silhouettes never appear in training.
• Palette: 1,024 colours, k-means in OKLab over the training pixels of both sizes. Mean ΔE is 0.74 at 16×16 and 1.02 at 32×32. 512 colours failed the 32×32 quality threshold.
• Optimisation:
  • AdamW, betas (0.9, 0.95), learning rate 5.5e-4, weight decay 0.1.
  • Warm-up of 1,500 steps, then cosine decay over the time budget.
  • Gradient clipping at 1.0, bf16 autocast.
  • Batches of about 49k tokens, each batch a single canvas size.
• Compute: 6 hours on one NVIDIA RTX PRO 6000 (power-capped at 450 W), 16.6 epochs.
  • The best validation score came at minute 345; no overfitting.
  • A 101M-parameter variant was tried in a 40-minute pilot. It lost under a selection rule written down before the pilots.

Evaluation (test split, default sampling)

	16×16	32×32
Bits per sprite, caption given	324	1,010
Colour bits per pixel when the silhouette is given	1.87	2.48
Largest gap from real sprites in shape statistics (one component, no holes, row-convex, symmetric)	2.6 pp	3.2 pp
Real-vs-generated classifier AUC on shape + colour features (0.5 = indistinguishable)	0.63	0.64–0.66
Named colour present: model / real sprites	80.5% / 82.6%	76.0% / 77.0%

• Probe splits of pixelart-172k (32×32), named colour: object + colour pairs absent from training (val_binding) score 87.0%, against 95.4% for the real sprites. Unseen objects (val_unseen) score 71.2%, against 76.6%.
• Russian / Ukrainian: the colours generated from a translated caption match the English ones (cosine similarity 0.68 / 0.67, against 0.21 for unrelated captions). The dominant hue agrees in 71% of 24 test phrases.
• 24×24 window: 99.2% of sprites are one connected shape.
• Validation vs test: at 32×32 the validation set included the dataset's official validation split, which is simpler: a median of 22 colours per sprite against 49 in training. The honest 32×32 number is therefore the test one above.
• Human check: a single rater named the object in 21 of 22 generated sprites (1 of 5 choices). They took 7 of 10 generated 32×32 sprites for real ones. One rater is not a study; more ratings are being collected.

Limitations

• Composition: in a sweep of 6 objects × 11 attributes, single colour or material words work, even for pairs absent from the data ("golden sword": 98% golden). Two attributes together do not: "blue flaming mushroom" gives a blue mushroom without fire. The model hears words, not their combination.
• Pose and size words: "upright", "pointing down" and "tiny" have weak or no effect. "upright sword" gives 46% vertical swords at 16×16.
• Untranslated phrases: Russian and Ukrainian work well when the phrase has a clear English counterpart in the data. Otherwise they can drift: «зелье здоровья» (health potion) gives round red objects, while "health potion" gives flasks.
• Reproducibility: results depend on the device and on how many sprites are drawn in parallel. The same seed reproduces only under the same conditions.
• Style: the style is that of the training data. Both datasets are AI-generated, and the sprites inherit those generators' style and biases.

Licence and data

• Weights: CC BY-NC 4.0 (non-commercial use; attribution).
• Code (GitHub): Apache-2.0.
• Pixel Art Atlas: released by its author under CC0 1.0. It was generated with Krea 2 Turbo and a pixel-art LoRA. The card states that this does not grant rights in Krea's models or third-party material, and does not override the Krea 2 Community License. Check the dataset card and Krea's terms before any commercial use of outputs.
• pixelart-172k: MIT. All images are AI-generated: 121,394 sprites made with a Krea 2 pixel-art LoRA, and 50,936 cut from generated sprite sheets.
• Text encoder: paraphrase-multilingual-MiniLM-L12-v2, Apache-2.0.

How it was built

The whole route is told as a hands-on article, Through the Thicket to Inference:

• data statistics;
• a symmetric-mask generator;
• a colour model;
• this full-canvas model;
• the guidance experiments and the evaluation.

It is linked from the GitHub repository.