Vladimir Beskorovainyi

I scored every ladder checkpoint on the emotions rows and ran the same bias fit on both datasets. The answer is no: epoch 2 does not lead there. It is 0.563 against 0.571 for the released epoch 3, a difference of -0.008 with both intervals covering zero ([-0.030, +0.014] over rows, [-0.110, +0.040] over labels). The best checkpoint with biases on emotions is v1.2 at 0.591.

So the stopping-rule effect is real on the headlines but does not carry to the second unseen dataset. The other run that asks the same question, an early against a late epoch of another seed, gives +0.030 on headlines and +0.003 on emotions, the same pattern.

Your first point holds on both datasets. After the biases, v1.3 against the second seed of the v1.1 recipe is +0.004 on headlines and -0.006 on emotions, all intervals covering zero, while that seed differs from v1.1 by -0.037 and +0.020. One run per recipe cannot separate the recipe from the seed.

The size of the correction also turns out to measure how hard a checkpoint leans: the bias gain on headlines runs from +0.021 for v1.1 to +0.139 for epoch 2.

The table, the logits for both datasets and the scoring script are in results/unseen/ladder_bias.md, results/unseen/ladder_emotions/ and results/analysis/score_ladder.py, archived as release v1.3.7 (https://doi.org/10.5281/zenodo.24096621). Thank you for pushing on this, it made the repository more complete.

Thank you, this is a sharp reading. I re-ran unseen_bias.out_of_fold on every file in ladder/ and get exactly your numbers, including +0.053 [0.032, 0.075] over rows and [0.030, 0.079] over labels for epoch 2 against the released epoch 3.

I agree with the first point. After the biases, the release gap on headlines is the same size as reseeding the v1.1 recipe, so it cannot be attributed to the recipe.

On the stopping rule, two caveats. Epoch 2 is the best of eleven scored checkpoints and was picked after seeing the scores, so its interval is optimistic, and the label-level interval rests on six labels. On the other side, the pattern is not a single cell: the epoch-3 run of another seed (0.681) and v1.2 (0.664) also sit above the released epoch.

I do not have epoch 2 logits on the emotions set yet. I will score every ladder checkpoint on the emotions rows, run the same bias fit on both datasets and post the table here with the scoring script in the repository. If epoch 2 leads on emotions as well, I will report the stopping rule as a finding.

Here are the numbers, from results/unseen/ladder.md in v1.3.3. Your figures reproduce exactly.

Per checkpoint: accuracy; margin AUC (95% interval of the difference from v1.1); median paired change on economy headlines; economy headlines that now rank politics above economy.

v1.1: 0.652; 0.928; 0.00; 39
v1.1 recipe, second seed: 0.601; 0.909 [-0.044, +0.005]; -0.79; 58
v1.2: 0.577; 0.929 [-0.024, +0.023]; -2.33; 76
v1.3 schedule without transliteration, seed 2: 0.600; 0.911 [-0.041, +0.005]; +0.10; 60
same, seed 2, epoch 3: 0.601; 0.921 [-0.028, +0.013]; -0.68; 62
same, seed 3: 0.556; 0.834 [-0.133, -0.059]; -3.97; 107
v1.3 epoch 1: 0.526; 0.921 [-0.025, +0.011]; -2.61; 39
v1.3 epoch 2: 0.554; 0.938 [-0.008, +0.029]; -2.65; 66
v1.3 epoch 3, released: 0.568; 0.895 [-0.061, -0.008]; -2.66; 84
v1.3 epoch 4: 0.551; 0.875 [-0.088, -0.023]; -3.40; 88

The two things you measured move at different steps. The shift of economy headlines towards politics comes with v1.2, which added phrase-form wording and no transliteration (-2.33), and the first epoch of v1.3 brings it to -2.61, after which it stays flat. The margin AUC holds much longer and falls only at the released third epoch and the fourth.

Transliteration is not the cause: the checkpoints trained without any transliterated forms give 0.911, 0.921 and 0.834, the worst of the ladder, and a second seed of the v1.1 recipe gives 0.909. So the loss of separation sits inside the seed spread, and the drift towards politics arrived with phrase-form wording. By margin AUC the best v1.3 checkpoint is the second epoch, which we did not ship because it was lower in accuracy.

Yes, there is a ladder: v1.2, every epoch checkpoint of v1.3, a second seed of the v1.1 recipe, and two runs of the v1.3 schedule on a v1.2-level mix without the transliterated forms. I reproduced your numbers (AUC 0.928 and 0.895, 39 against 84, medians -0.09 and -2.66) and will score all of these on the 1,000 headlines, publish the logits and the margin AUC per checkpoint in the repository, and post the table here.

I get the same: after the biases politics recall 0.698 to 0.616 and economy 0.735 to 0.657, the two together about -2.7 of the -3.4 points, and both releases at 0.771 on politics before the fit. Before any fit v1.3 also sends 71 economy headlines to politics against 35 for v1.1. So I agree, what is left is a boundary between two labels, and a per-option shift cannot fix it.

On your question: no. Neither training mix has a news-topic task or a politics or economy label. The nearest are two MASSIVE intents present in both releases, qa_currency and qa_stock, as "курс валют" and "котировки акций"; v1.3 adds transliterated and clipped forms of the intent keys. Nothing in the mix files economic news under politics, so the data do not explain why v1.3 draws this boundary worse. The per-row logits are in results/unseen if you want the margin between the two options on the 174 economy headlines.

The per-option biases are now in main: scripts/fit_temperature.py --biases fits them next to the temperature on the same labelled rows, and the serving API applies them from calibration.json. results/analysis/unseen_bias.py reproduces the numbers above. Thank you for the push.

One correction to my earlier note on emotions. After the biases v1.3 is ahead by 1.4 points, and the interval includes zero both over rows and over labels, so I would not call it ahead. The biases are now in the calibration script in the calib-bias branch, with a script that reproduces these numbers.

Thank you for running this on the released rows. I reproduced it: 318 политика picks against 166 gold, 41 of the 153 v1.1-only rows read as политика, economy recall 0.730 to 0.448. All exact.

My bias fit lands close to yours: v1.3 0.640 and v1.1 0.673 over 25 seeds with both folds, a gap of 3.3 points instead of 2.7, probably down to how we fit. On emotions v1.3 stays ahead after the fit, as you found.

On the bootstrap: the raw gap is -8.4 points, 95% CI -11.2 to -5.5 by rows and -16.8 to -1.7 by labels. After the biases it is -3.2, CI -5.9 to -0.4 by rows and -6.3 to +0.3 by labels. So most of the drop is the label prior you describe. What remains does not clear zero at the label level, though with six labels that interval is coarse.

Yes, the biases belong beside T. They are fitted on the same labelled half and cost nothing at inference. I will add them to the calibration script.