Jina CLIP v2 split ONNX

Independent text and vision ONNX encoders derived from jinaai/jina-clip-v2.

File Precision Size Input
jina-clip-v2-text-fp16.onnx FP16 1.12 GB input_ids: int64 [batch, sequence]
jina-clip-v2-text-fp32.onnx (+.data) FP32 2.24 GB input_ids: int64 [batch, sequence]
jina-clip-v2-text-int8.onnx Dynamic INT8 564 MB input_ids: int64 [batch, sequence]
jina-clip-v2-vision-fp16.onnx FP16 610 MB pixel_values: float32 [batch, 3, 512, 512]
jina-clip-v2-vision-fp32.onnx (+.data) FP32 1.22 GB pixel_values: float32 [batch, 3, 512, 512]
jina-clip-v2-vision-int8.onnx Dynamic INT8 313 MB pixel_values: float32 [batch, 3, 512, 512]

Every model is self-contained (FP32 uses an external-data companion file), supports dynamic batch sizes, and returns normalized float32 embeddings named embeddings with shape [batch, 1024]. The text models also support dynamic sequence lengths.

Which runtime should I use?

Measured on RTX 2070 (sm_75) unless noted. Full methodology: GitHub results.md.

scenario what to use measured
GPU, Turing (sm_75) or newer, fastest TensorRT engine built from the FP16 text graph via this repo's scripts/build_trt_engine.py b1×1024 25 ms, b32×1024 0.80 s, flat ~3 GiB VRAM
GPU via ONNX Runtime FP16 models (text: apply the Einsum→MatMul rewrite first — −25%) b1×1024 76–97 ms, b32×1024 2.4–2.6 s
GPU, portable PyTorch, long context fp16 torch + sdpa_patch.py b32×1024 1.70 s, flat ~2.65 GB
CPU dynamic INT8 models text b8x128 tokens 0.74 s vs 3.14 s FP16 (i9-7980XE, ORT 1.29)

Notes:

  • TensorRT fuses each attention layer into a single flash-attention-style kernel (_gemm_mha_v2) with linear memory; engines are hardware-specific and bound to the built optimization profile (default [1,8]..[32,2048]; pad shorter sequences with pad-token id 1, which the internal mask makes semantically neutral). GTX 1080 Ti / Pascal cannot run TensorRT 10+ — use ONNX Runtime there.
  • FP16 ONNX is not recommended for CPU inference: CPU providers promote unsupported FP16 ops or insert casts, increasing latency and memory.

Preprocessing and inference

Use the tokenizer and image processor from the base model:

import onnxruntime as ort
from transformers import AutoImageProcessor, AutoTokenizer

base_model = "jinaai/jina-clip-v2"
tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True)
image_processor = AutoImageProcessor.from_pretrained(base_model, trust_remote_code=True)

text_session = ort.InferenceSession("jina-clip-v2-text-int8.onnx")
tokens = tokenizer(["A photo of a cat"], padding=True, truncation=True, return_tensors="np")
text_embeddings = text_session.run(["embeddings"], {"input_ids": tokens.input_ids})[0]

# `images` is a list of PIL images.
vision_session = ort.InferenceSession("jina-clip-v2-vision-int8.onnx")
pixels = image_processor(images=images, return_tensors="np").pixel_values
image_embeddings = vision_session.run(["embeddings"], {"pixel_values": pixels})[0]

Matryoshka dimensions

The trained dimensions are 32, 64, 128, 256, 512, 768, and 1024. To reproduce the base model's truncate_dim=N behavior, truncate the prefix and normalize it again:

import numpy as np

embedding = embedding[:, :dimension]
embedding /= np.linalg.norm(embedding, axis=1, keepdims=True)

Validation

Both INT8 graphs were re-quantized on 2026-09-01 with per-channel weight scales and ORT's reduce_range; the artifacts published before that date were per-tensor and are not faithful — a single caption embedded at mean cosine 0.70 to the base model. If you cached the earlier files, re-download them.

The text encoder is scored on 424 real captions against ground-truth PyTorch fp32, on two metrics, because neither catches the other's failure mode: per-vector cosine is absolute and per-document, retrieval ranking is relative and cross-document, and this export has shipped a build that passed each while failing the other.

Dimension Mean cosine Worst caption Ranking
64 0.9929 0.9699 OK
128 0.9899 0.9694 OK
256 0.9866 0.9639 OK
512 0.9846 0.9597 OK
768 0.9839 0.9589 OK
1024 0.9838 0.9586 OK

The FP16 and FP32 graphs score 1.0000 on the same check. Ranking is a 20-profile retrieval fixture in which the mean-pooled vectors of 4 relevant profiles must outrank 16 irrelevant ones; dim=32 ranks wrongly for every build including FP16 and is a property of that fixture, not of any encoder. Reproduce with jina-clip-verify --text-model <graph> from the GitHub repository.

Vision INT8 scores mean cosine 0.9874 (worst 0.9831) at 512 dimensions and 0.9855 (worst 0.9809) at 1024 against the FP16 vision graph on synthetic structured images; a real-image vision gate is still open.

INT8 CPU performance

ONNX Runtime 1.29, Intel i9-7980XE, ORT_ENABLE_BASIC, mean of 5 runs:

Shape INT8 FP16
b1x128 0.13 s 0.53 s
b8x128 0.74 s 3.14 s
b8x512 3.53 s 12.48 s

Peak resident memory: 2.6 GiB INT8 vs 5.0 GiB FP16. The text input remains dynamic up to 8192 tokens, although ordinary full attention still has quadratic memory use; the TensorRT path removes that quadratic term on supported GPUs.

License and attribution

This is a quantized and structurally split derivative of jinaai/jina-clip-v2. It retains the base model's CC BY-NC 4.0 license and non-commercial restriction. Refer to the base model card for intended use, limitations, training details, and complete attribution.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for satorean/jina-clip-v2-split-onnx

Quantized
(1)
this model