Papers
arxiv:2611.08439

Label-Wording Augmentation in a Typed-Decision Encoder: Gains on Unfamiliar Words, Not on Abbreviations or Codes

Published on Nov 5

Abstract

A 307M-parameter multilingual cross-encoder for Russian routing decisions was compared across two releases, with label-form augmentation improving accuracy on ordinary and borrowed labels but not abbreviations or opaque codes, while exposing evaluation hazards.

Typed-decision models score options that the caller supplies at request time, so their accuracy depends on label wording that the model's developer does not control. We compare two releases of a 307M-parameter multilingual cross-encoder, Ru-Jev, trained on thirteen Russian-language tasks; the later release adds an augmentation that presents labels in several surface forms. Holding questions fixed and varying only how eighteen routing labels are worded, we evaluate ten label vocabularies, produced by three separate processes and checked for overlap with the training data, with five candidates per question and, for eight of them, also with eighteen. On held-out vocabularies made of ordinary or borrowed words (English-derived jargon, office slang, department names), the augmented release is more accurate in every setting, by up to 0.24. Eleven of these fourteen gains survive a Holm correction (slang with eighteen candidates and descriptive sentences with five do not), and for jargon on all labels the label-level interval excludes zero in both settings. On abbreviations and opaque codes it gives no usable gain: on abbreviations it is not better, and worse on the clean slice, and on opaque codes both releases stay at or below chance once labels related to training options are removed. In most cells where accuracy rises, the risk-coverage area falls, but with eighteen candidates confident errors rise on jargon and slang. On two datasets neither release saw, accuracy moves by -8.4 and +5.6 points. We also report four evaluation hazards from comparing the two releases.

Community

Author here. Model: https://huggingface.co/besktech/ru-jev. Code, per-row predictions and every evaluation script: https://github.com/beskvladimir-create/ru-jev

One result the preprint does not cover. After the release, a reader showed on the released rows that most of the v1.3 drop on news headlines is a label prior, not a reading failure: v1.3 answers politics 318 times for 166 political headlines. A temperature cannot fix that, since it never changes the argmax. One additive bias per option, fitted on half the rows and scored on the other half over 25 splits, lifts v1.3 from 0.568 to 0.640 and v1.1 from 0.652 to 0.673, so the gap falls from 8.4 to about 3 points. On emotions the v1.3 lead shrinks to 1.4 points and is no longer distinguishable from zero. Per-option biases are being added to the calibration script next to the temperature.

If you test the model on your own label set, please post the number of labels, candidates per question, accuracy, and whether the labels are words, abbreviations or codes. Negative results are as useful as positive ones.

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2611.08439
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 1

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.08439 in a dataset README.md to link it from this page.

Spaces citing this paper 1

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.