Zero entities with the documented setup (gliner 0.2.26/0.2.27 + transformers 5.1.0) — repro included
Hi, congrats on the release — the bidirectional LFM2.5 backbone idea is great and the benchmark numbers look very promising. Unfortunately I can't reproduce any output with the published checkpoint: the model returns zero entities with the exact setup documented on the model card, including on the card's own example. Full repro below — hope it helps you pin it down quickly.
Environment
| Component | Version |
|---|---|
| gliner | 0.2.26 (latest PyPI) and 0.2.27 (git main) — both tested |
| transformers | 5.1.0 (required: the checkpoint's tokenizer_config.json declares TokenizersBackend, which does not exist in 4.x — loading fails there with ValueError: Tokenizer class TokenizersBackend does not exist) |
| torch | 2.12.0+cu130 |
| Hardware | NVIDIA GB10 (CUDA) and CPU — identical results on both |
| Checkpoint revision | d231d87f80a7ffb12df9ffa36dce70a7eb7cd3d2 (latest as of testing) |
Repro (model card example, verbatim)
from gliner import GLiNER
model = GLiNER.from_pretrained("VAGOsolutions/SauerkrautLM-LFM2.5-GLiNER")
text = "Maria Schmidt arbeitet bei Siemens in München, E-Mail: maria.schmidt@siemens.com"
labels = ["person", "organization", "location", "email"]
entities = model.predict_entities(text, labels, threshold=0.5)
print(len(entities)) # -> 0
A warning is emitted at load time:
gliner/data_processing/processor.py:42: UserWarning: Tokenizer missing 'unk_token'. This may cause issues.
Diagnosis attempts
Dropping the threshold to 0.0 to inspect raw candidates shows the scoring head is producing ~zero scores and noise spans (everything collapses to the first label):
threshold=0.0 -> 6 spans, all scores 0.0000:
0.0 person | maria.schmidt@siemens.com
0.0 person | Schmidt arbeitet bei Siemens i
0.0 person | Maria
0.0 person | E-Mail
0.0 person | ,
0.0 person | :
Same behaviour on an English sentence, on CPU and CUDA, with gliner 0.2.26 and git main. For comparison, nvidia/gliner-PII runs fine in the same environment (93%+ recall on our French PII eval), so the harness itself is sound.
This pattern (all-zero scores + arbitrary span/label selection + the unk_token warning) looks like the GLiNER special prompt tokens (<<ENT>> / <<SEP>>) not being mapped correctly through the v5 TokenizersBackend tokenizer — the label embeddings the matcher sees would then be garbage, which matches the all-zero similarity scores.
Note the structural bind: the published tokenizer requires transformers ≥ 5, while public gliner (0.2.x) was written against the 4.x tokenizer API — so as published, there may be no public version combination that works.
Questions
- Which exact
gliner+transformersversions (or fork/branch) were used for the benchmark numbers? - Would you consider either pushing a tokenizer config compatible with transformers 4.x, or documenting the working version pin on the card?
Happy to test a fix — we evaluate this model for French notarial PII anonymization and the announced PII numbers are exactly what we are after.
Your environment is fine the issue is a missing setup step, not the tokenizer.
This checkpoint uses a bidirectional LFM2.5 backbone (Lfm2BiModel). Stock gliner doesn’t know that architecture, so it silently falls back to loading LFM2 as its original causal model weights load fine, but the encoder output is meaningless for span/label matching, hence the all-zero scores you see.
The fix is the code_modification/ folder in the repo. Copy its two files into your installed gliner package before running:
pip install gliner # 0.2.26 is fine
python - <<'PY'
import pathlib
import shutil
import gliner.modeling
dst = pathlib.Path(gliner.modeling.file).parent
for f in ("encoder.py", "lfm2_bi.py"):
shutil.copy(f"code_modification/{f}", dst / f)
print("patched:", dst)
PY
lfm2_bi.py rebuilds LFM2 as a true bidirectional encoder (full attention + symmetric convolutions), and encoder.py registers Lfm2Config → Lfm2BiModel so gliner stops falling back to AutoModel.
After that, the model-card example works as expected.