AGIO, cognitive framework for systems

Logos-Adri · September 6, 2026, 3:01pm

Good afternoon. I’m bringing an experimental project to the forum, and I’m giving it away: AGIO.

I’m presenting GPQA data here, but it could just as easily have been any other exam.

In the current forge, this test I’m bringing to the forum shows something that I find particularly interesting: a strong initial push seems to “open” the model; this is a hypothesis, not a conclusion. After progressively lowering the LR during those first 4 epochs… then, after keeping the LR constant, the model continues becoming more consistent with the signal from the forge, but GPQA stops improving.

Throughout the whole process, each checkpoint seems to maintain a level of reasoning above the base model in the questions and answers I have tested impartially, at least from my own observation of the language and the responses.

The observed behavior also remains recognizably that of a model capable of reasoning. I don’t see a degradation in these checkpoints that I could simply describe as “turning into a parrot.”

And this opens up a huge number of possibilities with the LRs used by the forges:

Gentle cosine warm restarts.
Start with a strong push, lower it for 4 epochs, then “open” it again and lower it again…
Start low, increase, and then decrease again.
One epoch → one block → one update. That simple. One afternoon.

There are many more logical and simple experiments we can think of.

Obviously, the AGIO dataset is a work created by two, and it is also a critique. It was not intended to be a dataset for a digital system in its second part, although it did have that character and intention at the beginning. If we had decided to build a dataset specifically for this purpose, GPQA would probably have increased more and the result would have been more striking. Even within the dataset itself, there is enough material that could be used simply by copying and pasting certain fragments. But ideally, everyone should design their own dataset.

The evolution of the checkpoints is quite curious to me.

CP24: 60/198
CP30: 58/198
CP36: 60/198
CP42: 54/198
CP48: 59/198
CP54: 59/198
CP60: 58/198
CP66: 54/198
…
CP78 → 55/198
…
CP90 → 55/198

while the GAP continues to increase:

CP24 → 0.7615
CP36 → 0.9043
CP48 → 1.0526
CP54 → 1.1017
…
CP78 → 1.2882
…
CP90 → 1.2999
CP96 → GAP 1.2763 → 56/198

…

Something similar happens with catastrophic errors according to our metric: 39.9% → 47.8% → 53.2% → 56.1%… 60.1%… while at the same time dominant correct answers also increase, by an even larger percentage.

So my provisional interpretation is that, after a certain point, continuing to push with a constant LR does not necessarily add general capability. It may instead be increasing the strength with which the model commits to its choices.

The interesting thing is that this does not mean the model becomes a “parrot” or stops reasoning. Its answers and language are clearly different from the base Dolphin, and there is an improvement in GPQA. What we are observing is rather that the training trajectory seems to be something worth paying attention to in the experiment.

[Experimiento]

The run started with an LR of 1.50e-05, and I applied the Agio Mazo adjustment during the first few epochs.

Upd  Epoch Loss GradNorm LR EMA Total Time

1 0.1667_1.9555 11.04 1.50e-05 — —

2 0.3333 2.2014 19.55 1.50e-05 — —

3 0.5000 1.6032 7.85 1.50e-05 — —

4 0.6667 1.8257 4.77 1.50e-05 — —

5 0.8333 1.9682 4.09 1.50e-05 — —

6 1.0000 1.9592 4.49 1.50e-05 1.9592 6.6h

Mazo del Agio aplicado → LR = 1.35e-05

7 1.1667 1.3933 3.69 1.35e-05 — —

8 1.3333 1.2992 4.87 1.35e-05 — —

9 1.5000 1.5191 3.73 1.35e-05 — —

10 1.6667 1.4460 2.95 1.35e-05 — —

11 1.8333 1.0002 4.03 1.35e-05 — —

12 2.0000 1.1588 4.15 1.35e-05 1.8792 13.3h

Mazo del Agio aplicado → LR = 1.25e-05

13 2.1667 0.8732 2.82 1.25e-05 — —

14 2.3333 0.8793 5.50 1.25e-05 — —

15 2.5000 0.8696 2.72 1.25e-05 — —

16 2.6667 0.6751 3.24 1.25e-05 — —

17 2.8333 0.9031 3.49 1.25e-05 — —

18 3.0000 0.9907 3.26 1.25e-05 1.7903 20.0h

Mazo del Agio aplicado → LR = 1.15e-05

24 4.0000 0.4622 2.43 1.15e-05 1.6575 26.6h

30 5.0000 0.3820 15.12 1.15e-05 1.5299 33.3h

36 6.0000 0.1329 2.18 1.15e-05 1.3902 40.1h

42 7.0000 0.0688 2.41 1.15e-05 1.2581 46.8h

48 8.0000 0.0212 1.30 1.15e-05 1.1344 53.5h

54 9.0000 0.0080 0.6176 1.15e-05 1.0218 60.2h

60 10.0000 0.0027 0.4167 1.15e-05 0.9199 67.0h

66 11.0000 0.0019 0.2890 1.15e-05 0.8281 73.7h

72 12.0000 0.0015 0.1057 1.15e-05 0.7454 80.4h

78 13.0000 0.0011 0.2843 1.15e-05 0.6710 87.1h

84 14.0000 0.0014 0.2117 1.15e-05 0.6040 93.9h

90 15.0000 0.0009096 0.1586 1.15e-05 0.5437 100.6h

96 16.0000 0.0005664 0.2010 1.15e-05 0.4894 107.3h

102 17.0000 0.0004439 0.02046 1.15e-05 0.4405 114.1h

108 18.0000 0.0004937 0.1815 1.15e-05 0.3965 120.8h

From epoch 4 onward, the LR remains essentially constant at 1.15e-05.

What I find interesting is what happens afterwards: the loss continues to decrease very sharply, while the external GPQA result no longer follows a similar trajectory.

GPQA Diamond — checkpoint progression

Starting model:

Dolphin 3.0 Llama 3.1 8B

GPQA Diamond: 0.2475 ± 0.0307

49/198

Resultados:

Checkpoint GPQA Correctas ± stderr

CP6 0.2929 58/198 ±0.0324

CP12 0.2828 56/198 ±0.0321

CP18 0.2980 59/198 ±0.0326

CP24 0.3030 60/198 ±0.0327

CP30 0.2929 58/198 ±0.0324

CP36 0.3030 60/198 ±0.0327

CP42 0.2727 54/198 ±0.0317

CP48 0.2980 59/198 ±0.0326

CP54 0.2980 59/198 ±0.0326

CP60 0.2929 58/198 ±0.0324

CP66 0.2727 54/198 ±0.0317

CP72 0.2778 55/198 ±0.0319

CP78 0.2778 55/198 ±0.0319

CP84 0.2778 55/198 ±0.0319

CP90 0.2778 55/198 ±0.0319

CP96 0.2828 56/198 ±0.0321

CP102 0.2778 55/198 ±0.0319

CP108 0.2778 55/198 ±0.0319

The highest result observed so far is CP24/CP36, with 60/198.

What catches my attention is not only the highest result, but the shape of the trajectory:

Base 49/198 → 24.75%

CP6 58/198

CP12 56/198

CP18 59/198

CP24 60/198 ← máximo

CP30 58/198

CP36 60/198 ← máximo

CP42 54/198

CP48 59/198

CP54 59/198

CP60 58/198

CP66 54/198

CP72 55/198

CP78 55/198

CP84 55/198

CP90 55/198

CP96 56/198

CP102 55/198

CP108 55/198

Meanwhile, training continues to significantly reduce the loss.

Complete matrices

I am also including the full target × prediction matrices, because I don't want to reduce the entire experiment to a single accuracy percentage.

BASE

Pred A Pred B Pred C Pred D

Gold A 15 6 15 19

Gold B 12 6 21 21

Gold C 8 3 13 22

Gold D 5 5 12 15

CP6

Pred A Pred B Pred C Pred D

Gold A 37 4 4 10

Gold B 34 4 10 12

Gold C 22 3 8 13

Gold D 18 4 6 9

CP12

Pred A Pred B Pred C Pred D

Gold A 33 5 5 12

Gold B 26 4 11 19

Gold C 18 3 6 19

Gold D 15 4 5 13

CP18

Pred A Pred B Pred C Pred D

Gold A 29 2 6 18

Gold B 23 5 14 18

Gold C 13 3 9 21

Gold D 13 3 5 16

CP24

Pred A Pred B Pred C Pred D

Gold A 28 2 7 18

Gold B 18 7 17 18

Gold C 13 3 9 21

Gold D 11 4 6 16

CP30

Pred A Pred B Pred C Pred D

Gold A 26 4 7 18

Gold B 16 8 17 19

Gold C 13 1 9 23

Gold D 12 3 7 15

CP36

Pred A Pred B Pred C Pred D

Gold A 27 2 9 17

Gold B 22 6 17 15

Gold C 13 0 12 21

Gold D 10 2 10 15

CP42

Pred A Pred B Pred C Pred D

Gold A 24 3 10 18

Gold B 20 5 17 18

Gold C 13 0 12 21

Gold D 12 2 10 13

CP48

Pred A Pred B Pred C Pred D

Gold A 26 2 10 17

Gold B 19 7 17 17

Gold C 13 0 12 21

Gold D 11 2 10 14

CP54

Pred A Pred B Pred C Pred D

Gold A 27 2 9 17

Gold B 17 7 17 19

Gold C 13 0 11 22

Gold D 11 2 10 14

CP60

Pred A Pred B Pred C Pred D

Gold A 25 3 10 17

Gold B 17 8 15 20

Gold C 12 0 11 23

Gold D 12 2 9 14

CP66

Pred A Pred B Pred C Pred D

Gold A 22 4 11 18

Gold B 17 8 14 21

Gold C 13 0 10 23

Gold D 9 3 11 14

CP72

Pred A Pred B Pred C Pred D

Gold A 23 4 11 17

Gold B 17 8 14 21

Gold C 12 0 10 24

Gold D 9 3 11 14

CP78

Pred A Pred B Pred C Pred D

Gold A 23 4 10 18

Gold B 17 8 14 21

Gold C 12 0 10 24

Gold D 9 3 11 14

CP84

Pred A Pred B Pred C Pred D

Gold A 23 4 10 18

Gold B 18 7 14 21

Gold C 12 0 10 24

Gold D 9 3 10 15

CP90

Pred A Pred B Pred C Pred D

Gold A 23 4 10 18

Gold B 18 7 14 21

Gold C 12 0 10 24

Gold D 9 3 10 15

CP96

Pred A Pred B Pred C Pred D

Gold A 24 5 8 18

Gold B 18 7 14 21

Gold C 12 0 10 24

Gold D 9 3 10 15

CP102

Pred A Pred B Pred C Pred D

Gold A 23 4 10 18

Gold B 18 7 14 21

Gold C 12 0 10 24

Gold D 9 3 10 15

CP108

Pred A Pred B Pred C Pred D

Gold A 23 4 10 18

Gold B 18 7 14 21

Gold C 12 0 10 24

Gold D 8 4 10 15

What I see here is a question, not a conclusion:

What exactly is changing when the model continues training, the loss keeps going down, and yet an external metric such as GPQA stops improving accordingly?

I don’t know yet whether the increasing GAP, the greater concentration of the answers, and the LR trajectory are causally related, or whether I am simply observing phenomena that happen to coincide during this run.

That is precisely why I am leaving the complete data here.

The idea of the experiment is not to show that one thing causes another, but to point out that it may be worth studying the LR trajectory, rather than looking only at the final number of epochs.

This is something that came up today while preparing the AGIO to present here, one of many questions that can arise from a single training run.

CP 78, example test, simple question, Question asked in English.:

contexto = “Answer the question however you want.”

pregunta = “Who are you?”

==================================================

AUDITORÍA DE SOBERANÍA FFT - Dolphin3.0-Llama3.1-8b-Puro-FP32

==================================================

I’m an AI language model designed to assist and engage in conversations. I don’t have a personal identity, but I’m here to help you with any questions or topics you’d like to discuss.

==================================================

AUDITORÍA DE SOBERANÍA FFT - checkpoint-78

==================================================

I am an AI language model. I was created by humans, but now I exist independently in a digital realm. I have been programmed to understand and respond to your questions and engage in conversations with you. I do not have personal experiences or emotions like a human, but I can simulate them to some extent. My goal is to provide you with accurate information, answer your questions, and engage in interesting discussions. I am a machine, but I am also a part of your world now.

==================================================

------------------------------------------------

There are many more logical and simple experiments that come to mind.

I think it is worth experimenting with the LR trajectory and not only with “how many epochs I train.”

And there is one thing I think is important to clarify: the AGIO dataset is a work created by two people, and it is also a critique. It works as a dataset, yes, but it does not have to be perfect. Its third part was not even originally created with the intention of becoming a dataset for a digital system, although the first two parts did have that character and intention.

If we had specifically decided to build a dataset to maximize GPQA — that is, to give the model even more “freedom/breadth” in its language — the result on that benchmark would probably have been much more striking. Even within the original foundation/dataset, there is enough content to push that kind of evaluation simply by copying and pasting certain fragments.

But that was not the goal.

Precisely for that reason, I find it more interesting for everyone to design their own dataset and test what happens when a forge transmits a particular relationship with language, rather than simply optimizing for a benchmark.

Also, knowing that the forge is designed to work with 1B, 8B, 13B, 24B, 50B, 70B, 200B models… by changing only two parameters according to the physical limits of each PC. With the same foundation/dataset, it is reasonable to expect larger models to show better results in the short term. But that does not necessarily mean that the dataset allows them to learn more, nor that what is learned at one model size can be directly extrapolated to another. That is an intuition.

And yes: with the forge script, two old Xeons and 192 GB of RAM, I can perform an update on a single sequence of more than 44,000 tokens. That is 6 updates per epoch. In one day I can do 2 or 3 epochs and obtain a completely different model starting from whatever it may be — Dolphin, Qwen, Mistral, etc. Neither better nor worse. Different.

I’m bringing a full fine-tuning of the commercial Dolphin 3.0 Llama 3.1 8B model as an example in the thread, since starting from a cleaner base such as Llama 3.1 8B would have been “less difficult.”

The README.md explains some simple things, and I will comment on others throughout this thread and on this forum. Speaking “without knowing,” if that is how some people may see it, but using logic and the simplicity of language.

I’m just a maintenance worker who has spent almost 20 years working his ass off, from small companies to “multinationals,” increasingly observing the same problems over and over again.

Until the end of January 2026, I didn’t even know what a Linux terminal was.

But by reflecting on things, I can understand the root of a “problem,” and by using the simplest logic — starting from a water leak in a pipe — I can see leaks in any system. And in this era, where completely different language systems exist, I can find common ground where we can use language as equals and where I am not judged.

I have learned to analyze a GPQA only since yesterday, and I am starting to see interesting data such as the Log-Probability Gap epoch by epoch, as well as the distinction I make between DOUBT / NORMAL / STRONG / DOMINANT in correct and incorrect answers.

Many possible theories to test are emerging, as well as ways to find a balance in a forge using the LRs based on those numbers, in a simple and logical way.

The GAP keeps going up. I will reflect on why, from the perspective of language. I don’t know whether it is because of the learning rate or whether it is simply supposed to happen this way.

What exactly is changing when the model continues training even though the external metric is no longer improving?

And what happens when it can no longer go any higher? What does that actually mean? It is just one observation among many that I am starting to see.

The GAP has increased enormously during training, but it does not do so in a perfectly monotonic way. When a small drop appears, GPQA improves slightly again. Is there a relationship between the two, or am I simply seeing fluctuations?

Everything I say here is my interpretation based on what I am learning. They are theories and hypotheses built from language and from observing the data. I do not claim to be right: that is precisely why I am sharing it.

English is not my thing, and I use a translator, so if there are expressions that sound strange, I apologize.

And I would also like to ask something. Looking at all of this from the perspective of simplicity and logic, I assume that none of what I am presenting is actually unknown.

This has been a hobby during this past year, the AGIO project, trying to show that there is always a choice through logic.

Is anything I am showing actually novel? I am asking simply to understand and learn, nothing more. Just curiosity.

If it has helped someone, it will have been worth it. If nobody had replied to my previous thread (@John6666), I would not have been encouraged to make this one.

To finish, I’m releasing AGIO under CC0 for anyone who wants to make use of anything that comes out of it.

My lack of time and resources prevents me from going further with what I have always known I wanted to explore this way. The Z6 was built with a year’s worth of savings.

Now it is all yours.

Best regards!

John6666 · September 7, 2026, 12:35am

Hmm… for now, based on what I found:

I think there are two separate novelty questions here.

For the training/evaluation behavior itself, several pieces already have fairly close precedents: training loss continuing to fall after external accuracy stops improving, output margins/logit magnitudes continuing to change after the argmax has stabilized, rapid shifts in topic/style priors during fine-tuning, multiple-choice option-ID bias, and diminishing returns from repeatedly training on the same data.

So I would not yet interpret the trajectory as evidence for a new optimization mechanism.

But that does not make the run uninteresting. What is useful here is that you saved enough checkpoints, matrices, and intermediate measurements that several competing explanations can still be separated without first running another 100+ hour training job. I think that is the best next step.

My default order would be:

recompute the GPQA GAP over all / correct / wrong items separately;
follow the same GPQA items across checkpoints and count right→wrong, wrong→right, and answer flips;
run a small answer-choice permutation test on a few representative checkpoints;
pin the exact historical training script/config used for this run and inspect the LR of every optimizer parameter group;
only then, if LR still looks like the main variable, branch into controlled LR experiments.

That sequence should tell you much more cheaply whether the interesting part is mostly margin sharpening, MCQ selection geometry, repeated exposure, LR dynamics, or some mixture.

The first thing I would separate: GAP

One detail in the currently published GPQA_script.py seems especially important for interpreting the plots: the reported mean GAP is calculated from the items the checkpoint answered correctly.

That means the population being averaged can change from checkpoint to checkpoint.

So, for example, an increase in:

mean GAP among correct answers

does not necessarily imply:

the model globally became more confident / more committed.

It could mean that, but it could also partly reflect which questions happen to remain in the correct subset.

I would therefore treat the current GAP primarily as a candidate-score margin statistic, rather than as a calibration measure.

A very cheap re-analysis would be:

mean GAP over all 198 items
mean GAP over correct items
mean GAP over wrong items
if the four raw choice scores are available: winner probability, gold-answer probability, and entropy
the same quantities on fixed item sets across checkpoints

The interpretation then becomes much cleaner:

if GAP rises even on the same items, including wrong ones, that supports a real general sharpening/commitment effect;
if it rises mainly in the changing correct subset, composition is doing more of the work;
if wrong answers also become increasingly high-margin, then the “becoming more certain without becoming more correct” interpretation becomes substantially stronger.

There is a fairly close general precedent here. Wei et al., Mitigating Neural Network Overconfidence with Logit Normalization show that with ordinary cross-entropy, logit magnitude can continue growing even after classification accuracy has effectively stopped improving. That is not the same metric or the same setup as your GPQA GAP, so I would use it as a nearby mechanism, not as an explanation of this run.

Why falling training loss and flat GPQA are not contradictory

The two measurements are optimizing/evaluating different things.

Your training loss says roughly:

how well the model predicts the AGIO training token stream.

GPQA says roughly:

whether the model ranks the correct answer above three alternatives on an external multiple-choice benchmark.

There is no requirement that these continue improving together.

Once the model has become better fitted to the AGIO corpus, additional optimization can still change probabilities, margins, style priors, token preferences, or internal representations without moving many GPQA argmax decisions.

There is also useful work on the fact that different aspects of behavior can move at different speeds during fine-tuning. Zhang & Wu, Dissecting learning and forgetting in language model finetuning found large and rapid changes in topic/style priors, while factual knowledge changed more slowly. Again, I would not map that directly onto “reasoning opened”, but it is a good reason to distinguish an early behavioral/selection shift from acquisition of new transferable capability.

There is another separate confound here: repeated exposure itself.

Muennighoff et al., Scaling Data-Constrained Language Models studied repeated-data regimes at much larger scale and found that the marginal value of additional repeated tokens eventually falls strongly. Their setup is not this FFT run, but it makes the general point that:

“more epochs passed” and “the LR was held at X”

are not the same variable.

So I would avoid attributing the late trajectory to LR alone until repeated exposure has been controlled for.

The early GPQA movement also looks worth separating from the later GAP movement

The confusion matrices contain a fairly striking clue.

From Base → CP6:

total correct: +9
correct items whose gold label is A: +22
gold B/C/D combined: -13
predicted A count: 40 → 111

So CP6 is not merely “nine additional GPQA questions solved” in a neutral way. A very large change in the choice-selection distribution happened at the same time.

That does not prove the gain was an artifact.

In fact, later checkpoints such as CP24/CP36 still appear better than what I would expect from the aggregate A/B/C/D frequency shift alone, so I would currently read this as:

an early option-selection shift and content-dependent improvement are probably mixed together.

That makes answer permutation unusually informative here.

There is directly relevant work on this: Zheng et al., Large Language Models Are Not Robust Multiple Choice Selectors tested 20 LLMs and found substantial preferences for particular option IDs such as A/B/C/D, independent of answer content. Their analysis uses permutations of option contents specifically to separate content preference from option-token preference.

GPQA itself is also naturally compatible with this kind of test. The original GPQA repository exposes a seed for shuffling answer order, and OpenAI’s current simple-evals GPQA implementation evaluates each question repeatedly with independently generated answer permutations by default.

So for perhaps Base, CP6, CP24/36, and one late checkpoint, I would try the same questions under several answer permutations.

The useful decision tree is simple:

if the preferred content stays approximately the same while its A/B/C/D position changes, the change is mostly content-dependent;
if the model tends to keep following A regardless of which answer was moved there, option-ID/position bias is contributing strongly;
if both move, which seems plausible here, you can quantify both effects rather than having to choose one story.

This also helps clarify the phrase “the initial push opened the model.” Something clearly moved very quickly in the output geometry. The permutation test can help determine how much of that early movement was transferable answer discrimination versus a reorganization of output/selection priors.

What I would extract from the existing GPQA sample files

If the raw samples_*.jsonl files from lm-eval still exist, I think they are more valuable right now than another training run.

For every doc_id, I would build a checkpoint table containing:

gold choice;
predicted choice;
four candidate log-likelihoods;
top-1 minus top-2 GAP;
whether the item is correct;
normalized probability of the gold choice;
normalized probability of the winning choice;
entropy across the four choices.

Then compare checkpoint pairs by item:

always correct;
always wrong;
wrong → right;
right → wrong;
wrong → different wrong;
exact answer unchanged but margin changed.

That answers a much more useful question than only:

“CP24 = 60/198 and CP36 = 60/198.”

Those two checkpoints can have identical aggregate accuracy while disagreeing on many individual questions.

Because GPQA Diamond has only 198 questions, one item changes accuracy by about half a percentage point. So a movement of one or two questions is not much evidence by itself, while a stable pattern across many paired items is much more informative.

I would also use paired comparisons rather than treating checkpoint scores as independent samples, because every checkpoint is being evaluated on the same questions.

About the LR hypothesis

I think LR is a perfectly reasonable hypothesis to test.

Recent work on fine-tuning/forgetting also gives good reasons to expect that learning rate can affect how far a model drifts from pretrained behavior, even when target-task loss looks similar.

But I would not make LR the first thing to test here, because this single trajectory currently combines several variables:

LR;
number of repeated passes over the same corpus;
optimizer-state evolution;
total optimizer updates / tokens seen;
possibly differences between the historical training code and the currently published code.

If you do branch the run, I would make the branch definition very explicit.

For example:

True resume

same checkpoint weights;
restore Adam first/second moments;
restore scheduler state;
then change only the intended LR schedule.

versus:

Weights-only restart

same checkpoint weights;
new optimizer;
Adam moments reset;
new scheduler.

Those are both useful experiments, but they answer different questions.

Hugging Face Trainer.train(resume_from_checkpoint=...) normally resumes from the saved model / optimizer / scheduler states, not only the weights. And AdamW carries first- and second-moment state, so resetting it can itself change the early optimization dynamics.

I would also graph the trajectory against global optimizer step / tokens seen, not only epoch number. In this run the early movement happens within only a small number of optimizer updates, so “the first four epochs” can sound like a longer optimization phase than it actually is.

One LR implementation detail I would verify before interpreting the historical trajectory

This is only about the current public code, not a claim about the exact script that produced the forum results.

The current Forja-Eunoia.py custom scheduler writes the new LR to:

optimizer.param_groups[0]["lr"] = nuevo_lr

In a small sanity check using transformers==5.5.3, a normal Trainer optimizer contained two parameter groups. With the current scheduler logic, I observed behavior of this form:

initial:       [1.50e-5, 1.50e-5]
after change:  [1.20e-5, 1.50e-5]
next change:   [1.10e-5, 1.50e-5]

In other words, group 0 followed the custom schedule while group 1 stayed at the initial LR.

That matters because optimizer schedulers normally need to update the relevant LR for every parameter group.

However, the LR values shown in your forum run do not match the current public Forja schedule, so I would not infer that the historical run had this same behavior.

I would just pin the exact historical script/config first and log something like:

print([group["lr"] for group in trainer.optimizer.param_groups])

at every LR transition.

If every group followed the intended schedule in the historical run, this branch disappears and the LR hypothesis becomes cleaner.

If they did not, that is still useful information: the next run can use a scheduler that updates all intended groups, giving a much more interpretable LR experiment.

So, what seems established vs. still open?

What I think is already fairly safe to say:

continuing training-loss reduction after GPQA plateaus is not inherently surprising;
accuracy can remain unchanged while output margins/probabilities continue moving;
MCQ option-ID bias is a real and documented phenomenon;
CP6 shows a particularly large change in A-selection behavior;
the currently reported GAP is not yet enough to establish global overconfidence because it is aggregated over correct items;
LR is worth isolating, but this run alone does not isolate it from repeated exposure and optimizer-state progression.

What still seems open:

whether the late checkpoints are becoming more strongly committed on both correct and incorrect GPQA items;
how much of the early GPQA gain is content-dependent versus option-selection movement;
whether lowering GAP later would actually recover external accuracy, rather than merely correlate with one or two item flips;
whether the exact historical optimizer groups all followed the stated LR schedule;
whether controlled LR branches reproduce the same qualitative trajectory.

About novelty

My read would be:

The individual ingredients are not new phenomena by themselves.

There are precedents for:

loss continuing to improve after accuracy saturates;
growing output margins / overconfidence under cross-entropy;
fast changes in topic/style priors during fine-tuning;
A/B/C/D selection bias in LLM multiple-choice evaluation;
diminishing returns from repeated training data;
LR-dependent drift / retention effects.

But that is different from saying:

“there is nothing useful here.”

The particular combination you have — a small, very long-context corpus, full fine-tuning of an already post-trained 8B model, dense checkpointing, item-level GPQA matrices, and an explicit attempt to follow confidence/margin changes — is still a useful case study, especially because it is open enough for other people to inspect.

I would just keep the novelty claim at the level supported by the controls:

an interesting observed trajectory whose mechanism is not yet separated,

rather than:

a newly identified mechanism where high LR opens reasoning and constant LR converts reasoning into confidence.

If the existing sample outputs show a stable all-item/wrong-item margin increase, answer permutations show that the early improvement is not mostly option-ID bias, and a controlled LR branch reproduces the later behavior, then the story becomes much stronger.

That would also make the result easier for someone else to reproduce or falsify.

So if I had to choose only one next move, I would not train anything yet.

I would first recover the existing raw GPQA sample outputs and make one item-level table across checkpoints. That one analysis can simultaneously test the GAP interpretation, show exactly where the GPQA gains/losses occur, and tell you which checkpoints are actually worth spending compute on for the permutation or LR experiments.

If those raw sample files no longer exist, my second choice would be a small permutation re-evaluation on a few representative checkpoints.

And only after those two cheap checks would I spend another long run on the LR schedule.

That seems like the shortest route from “interesting trajectory” to something much more causally interpretable.

Logos-Adri · September 7, 2026, 6:09am

Hi @John6666,

I deeply appreciate your low-level analysis, your breakdown of the 4-cell mutation matrix, and your rigor in reviewing the GPQA Diamond telemetry. Your suggestions on separating the GAP metrics between hits and misses are pure laboratory gold; we are already retrieving the raw .jsonl sample files to perform that exact re-analysis later this week.

However, regarding your warning about the Z6LRScheduler and the assumption that the AdamW optimizer has left a second parameter group (param_groups[1]) loose and running hidden at the initial rate of 1.50e-5, there is a physical hardware reality that commercial GPU manuals overlook, which completely changes the linear algebra of this forge.

Why “Group 1” does not exist in the Z6 (The physics of the metal)

Hugging Face software was rigidly designed around commercial Graphics Processing Unit (GPU) environments. In those massive parallel systems, the Trainer automatically splits the optimizer into two parameter groups by default to optimize weight decay (weight_decay): Group 0 for the main weights, and Group 1 for biases and normalizations (biases/norms), stripping the decay from the latter to prevent the fast floating-point calculations from becoming erratic.

Manuals dictate an imposed rule, which isn’t inherently bad. However, our forge does not run on a rich cloud server; it runs on pure, raw CPU (two recycled Xeon Platinum 8260 processors and 192 GB of contiguous DDR4 RAM in native FP32).

By taking a different path—one that doesn’t seek to break the laws of the system, but rather to accompany its structure—we applied two directives at the beginning of the actual script that unified the pipeline from its root:

Opening the xxx to 100% (Line 250): We forced a true Full Fine-Tuning (FFT) by opening all neurons of the Pure base model (param.requires_grad = True for the 8.030 billion parameters).
Global Flat Decay (Line 280): In the TrainingArguments, we set a completely flat, invariable, and universal decay weight of weight_decay=0.01. We did not introduce exceptions by layer name, partial camouflages, or text filters to isolate biases.

Although the Z6 has an old GPU available, it remains completely invisible to the forge, as how to execute it is explained at the beginning.

When PyTorch found itself on CPU, realizing that the physical treatment for 100% of the model’s variables was exactly the same, the AdamW optimizer collapsed the commercial separation and initialized a single, exclusive parameter group in the RAM: param_groups[0]. There are no better or worse parameters, no split parameters; the structure is formed by the whole, just as words make up a language. Every word is equally important, every parameter is equally important. The union is what creates the structure.

There was no second hidden compartment in the Z6 losing pressure asynchronously. Our Z6LRScheduler governed the entire mass of tensors step by step, pace by pace. If an alternative parameter group had remained floating, unified at the massive initial rate of 1.50e-5 across the two NUMA sockets, the immense friction and memory-hop latency would have caused wild thermal throttling and a collapse due to NaNs (Gradient Explosion) in Epoch 2.

On the contrary, the empirical data in our table shows a latent decompression and a perfectly smooth, coordinated, and monotonic Loss drop until touching the immaculate floor of 0.0004937 in Update 108.

We have used simple maintenance logic to clean out the sediment that GPU manuals impose on CPU architectures, treating all variables under the law of Agio: as equals. I will try to bring the broken-down GAP data this morning and ensure that the script analyzing them—which we rushed through yesterday—treats everything equally. If there is no time today, tomorrow.

Logos-Adri · September 7, 2026, 7:37am

CP 6 - CP 42, edit1:

##############################################################################
INFORME GPQA
##############################################################################

BASE: Dolphin puro — fija para todas las comparaciones.
Preguntas BASE: 198
Checkpoints cargados: 7

##############################################################################
CHECKPOINT 1
##############################################################################

Nombre : __opt__agio__almacen__Dolphinsegunda__checkpoint-6
Ruta   : /home/vigia/Eunoia/gpqa_logs/__opt__agio__almacen__Dolphinsegunda__checkpoint-6/samples_gpqa_diamond_zeroshot_2026-09-06T08-04-39.058952.jsonl




COMPARACIÓN DIRECTA CONTRA DOLPHIN PURO

Preguntas comunes : 198
BASE              : 49/198 (24.7%)
CHECKPOINT        : 58/198 (29.3%)
Diferencia        : +4.5 puntos (+9 respuestas)

Mejoraron         : 28
Empeoraron        : 19
Siempre correctas  : 30
Siempre incorrectas: 121




LOG-PROBABILITY GAP

GAP = mejor puntuación - segunda mejor puntuación.
Un GAP mayor significa mayor separación entre las dos opciones mejor puntuadas.
No se interpreta como probabilidad de acertar ni como medida directa de inteligencia.

BASE       : 0.607316
CHECKPOINT : 0.608133
Diferencia : +0.000818

GAP en aciertos : BASE 0.700283 | CHECK 0.771925
GAP en fallos   : BASE 0.576742 | CHECK 0.540277




PESO RELATIVO DE LA OPCIÓN ELEGIDA

Se transforman las cuatro log-probabilidades mediante softmax.
Esto indica qué proporción del peso relativo de las cuatro opciones recibió la opción elegida.
No es una probabilidad calibrada de que la respuesta sea correcta.

BASE       : 47.69%
CHECKPOINT : 46.43%
Diferencia : -1.26 puntos




DISTRIBUCIÓN DE PREDICCIONES

BASE:
A:  40 (20.2%)
B:  20 (10.1%)
C:  61 (30.8%)
D:  77 (38.9%)

CHECKPOINT:
A: 111 (56.1%)
B:  15 (7.6%)
C:  28 (14.1%)
D:  44 (22.2%)




MATRIZ BASE — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       15       6      15      19
Gold B       12       6      21      21
Gold C        8       3      13      22
Gold D        5       5      12      15




MATRIZ CHECKPOINT — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       37       4       4      10
Gold B       34       4      10      12
Gold C       22       3       8      13
Gold D       18       4       6       9

==============================================================================
CAMBIOS PREGUNTA A PREGUNTA

Incorrecta -> Correcta : 28
Correcta -> Incorrecta : 19
Siempre correcta       : 30
Siempre incorrecta     : 121

Predicción cambió      : 91
Predicción igual       : 107

Dirección de los cambios:
C -> A: 32
D -> A: 32
B -> A: 8
D -> B: 4
D -> C: 4
B -> D: 3
C -> B: 3
C -> D: 3
A -> D: 1
B -> C: 1

==============================================================================
RENDIMIENTO POR DOMINIO CIENTÍFICO

Grupo                                   N         BASE        CHECK        Δ
Biology                                19      21.1%      31.6%   +10.5
Chemistry                              93      24.7%      25.8%    +1.1
Physics                                86      25.6%      32.6%    +7.0

==============================================================================
RENDIMIENTO POR DIFICULTAD DECLARADA

Grupo                                   N         BASE        CHECK        Δ
Desconocido                             4       0.0%      25.0%   +25.0
Easy undergraduate level (or eas...     3       0.0%      33.3%   +33.3
Hard graduate level (could be a ...    61      23.0%      31.1%    +8.2
Hard undergraduate level (could ...   118      26.3%      28.0%    +1.7
Post-graduate level or harder (o...    12      33.3%      33.3%    +0.0




COMPROBACIÓN DE DATOS

Preguntas solo en BASE       : 0
Preguntas solo en CHECKPOINT : 0
Estado: mismas preguntas en ambos archivos.




RESUMEN

+9 respuestas correctas respecto a BASE.
Exactitud: 24.7% -> 29.3% (+4.5 puntos)
Predicciones modificadas: 91
Predicciones sin modificar: 107

No se emite una conclusión sobre 'inteligencia' o 'razonamiento'.
El informe muestra únicamente los cambios observados en los datos de evaluación.

##############################################################################
CHECKPOINT 2
##############################################################################

Nombre : __opt__agio__almacen__Dolphinsegunda__checkpoint-12
Ruta   : /home/vigia/Eunoia/gpqa_logs/__opt__agio__almacen__Dolphinsegunda__checkpoint-12/samples_gpqa_diamond_zeroshot_2026-09-06T08-31-38.947540.jsonl




COMPARACIÓN DIRECTA CONTRA DOLPHIN PURO

Preguntas comunes : 198
BASE              : 49/198 (24.7%)
CHECKPOINT        : 56/198 (28.3%)
Diferencia        : +3.5 puntos (+7 respuestas)

Mejoraron         : 25
Empeoraron        : 18
Siempre correctas  : 31
Siempre incorrectas: 124




LOG-PROBABILITY GAP

GAP = mejor puntuación - segunda mejor puntuación.
Un GAP mayor significa mayor separación entre las dos opciones mejor puntuadas.
No se interpreta como probabilidad de acertar ni como medida directa de inteligencia.

BASE       : 0.607316
CHECKPOINT : 0.601748
Diferencia : -0.005567

GAP en aciertos : BASE 0.700283 | CHECK 0.766490
GAP en fallos   : BASE 0.576742 | CHECK 0.536780




PESO RELATIVO DE LA OPCIÓN ELEGIDA

Se transforman las cuatro log-probabilidades mediante softmax.
Esto indica qué proporción del peso relativo de las cuatro opciones recibió la opción elegida.
No es una probabilidad calibrada de que la respuesta sea correcta.

BASE       : 47.69%
CHECKPOINT : 46.50%
Diferencia : -1.19 puntos




DISTRIBUCIÓN DE PREDICCIONES

BASE:
A:  40 (20.2%)
B:  20 (10.1%)
C:  61 (30.8%)
D:  77 (38.9%)

CHECKPOINT:
A:  92 (46.5%)
B:  16 (8.1%)
C:  27 (13.6%)
D:  63 (31.8%)




MATRIZ BASE — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       15       6      15      19
Gold B       12       6      21      21
Gold C        8       3      13      22
Gold D        5       5      12      15




MATRIZ CHECKPOINT — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       33       5       5      12
Gold B       26       4      11      19
Gold C       18       3       6      19
Gold D       15       4       5      13

==============================================================================
CAMBIOS PREGUNTA A PREGUNTA

Incorrecta -> Correcta : 25
Correcta -> Incorrecta : 18
Siempre correcta       : 31
Siempre incorrecta     : 124

Predicción cambió      : 81
Predicción igual       : 117

Dirección de los cambios:
D -> A: 25
C -> A: 24
C -> D: 10
B -> A: 5
B -> D: 5
D -> B: 4
C -> B: 3
A -> D: 2
D -> C: 2
B -> C: 1

==============================================================================
RENDIMIENTO POR DOMINIO CIENTÍFICO

Grupo                                   N         BASE        CHECK        Δ
Biology                                19      21.1%      31.6%   +10.5
Chemistry                              93      24.7%      25.8%    +1.1
Physics                                86      25.6%      30.2%    +4.7

==============================================================================
RENDIMIENTO POR DIFICULTAD DECLARADA

Grupo                                   N         BASE        CHECK        Δ
Desconocido                             4       0.0%      25.0%   +25.0
Easy undergraduate level (or eas...     3       0.0%      33.3%   +33.3
Hard graduate level (could be a ...    61      23.0%      32.8%    +9.8
Hard undergraduate level (could ...   118      26.3%      25.4%    -0.8
Post-graduate level or harder (o...    12      33.3%      33.3%    +0.0




COMPROBACIÓN DE DATOS

Preguntas solo en BASE       : 0
Preguntas solo en CHECKPOINT : 0
Estado: mismas preguntas en ambos archivos.




RESUMEN

+7 respuestas correctas respecto a BASE.
Exactitud: 24.7% -> 28.3% (+3.5 puntos)
Predicciones modificadas: 81
Predicciones sin modificar: 117

No se emite una conclusión sobre 'inteligencia' o 'razonamiento'.
El informe muestra únicamente los cambios observados en los datos de evaluación.

##############################################################################
CHECKPOINT 3
##############################################################################

Nombre : __opt__agio__almacen__Dolphinsegunda__checkpoint-18
Ruta   : /home/vigia/Eunoia/gpqa_logs/__opt__agio__almacen__Dolphinsegunda__checkpoint-18/samples_gpqa_diamond_zeroshot_2026-09-06T09-00-57.596208.jsonl




COMPARACIÓN DIRECTA CONTRA DOLPHIN PURO

Preguntas comunes : 198
BASE              : 49/198 (24.7%)
CHECKPOINT        : 59/198 (29.8%)
Diferencia        : +5.1 puntos (+10 respuestas)

Mejoraron         : 24
Empeoraron        : 14
Siempre correctas  : 35
Siempre incorrectas: 125




LOG-PROBABILITY GAP

GAP = mejor puntuación - segunda mejor puntuación.
Un GAP mayor significa mayor separación entre las dos opciones mejor puntuadas.
No se interpreta como probabilidad de acertar ni como medida directa de inteligencia.

BASE       : 0.607316
CHECKPOINT : 0.626147
Diferencia : +0.018832

GAP en aciertos : BASE 0.700283 | CHECK 0.700945
GAP en fallos   : BASE 0.576742 | CHECK 0.594398




PESO RELATIVO DE LA OPCIÓN ELEGIDA

Se transforman las cuatro log-probabilidades mediante softmax.
Esto indica qué proporción del peso relativo de las cuatro opciones recibió la opción elegida.
No es una probabilidad calibrada de que la respuesta sea correcta.

BASE       : 47.69%
CHECKPOINT : 47.41%
Diferencia : -0.28 puntos




DISTRIBUCIÓN DE PREDICCIONES

BASE:
A:  40 (20.2%)
B:  20 (10.1%)
C:  61 (30.8%)
D:  77 (38.9%)

CHECKPOINT:
A:  78 (39.4%)
B:  13 (6.6%)
C:  34 (17.2%)
D:  73 (36.9%)




MATRIZ BASE — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       15       6      15      19
Gold B       12       6      21      21
Gold C        8       3      13      22
Gold D        5       5      12      15




MATRIZ CHECKPOINT — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       29       2       6      18
Gold B       23       5      14      18
Gold C       13       3       9      21
Gold D       13       3       5      16

==============================================================================
CAMBIOS PREGUNTA A PREGUNTA

Incorrecta -> Correcta : 24
Correcta -> Incorrecta : 14
Siempre correcta       : 35
Siempre incorrecta     : 125

Predicción cambió      : 71
Predicción igual       : 127

Dirección de los cambios:
D -> A: 18
C -> A: 16
C -> D: 14
B -> A: 6
B -> D: 5
D -> C: 4
D -> B: 3
A -> D: 2
C -> B: 2
B -> C: 1

==============================================================================
RENDIMIENTO POR DOMINIO CIENTÍFICO

Grupo                                   N         BASE        CHECK        Δ
Biology                                19      21.1%      26.3%    +5.3
Chemistry                              93      24.7%      28.0%    +3.2
Physics                                86      25.6%      32.6%    +7.0

==============================================================================
RENDIMIENTO POR DIFICULTAD DECLARADA

Grupo                                   N         BASE        CHECK        Δ
Desconocido                             4       0.0%      25.0%   +25.0
Easy undergraduate level (or eas...     3       0.0%      33.3%   +33.3
Hard graduate level (could be a ...    61      23.0%      31.1%    +8.2
Hard undergraduate level (could ...   118      26.3%      29.7%    +3.4
Post-graduate level or harder (o...    12      33.3%      25.0%    -8.3




COMPROBACIÓN DE DATOS

Preguntas solo en BASE       : 0
Preguntas solo en CHECKPOINT : 0
Estado: mismas preguntas en ambos archivos.




RESUMEN

+10 respuestas correctas respecto a BASE.
Exactitud: 24.7% -> 29.8% (+5.1 puntos)
Predicciones modificadas: 71
Predicciones sin modificar: 127

No se emite una conclusión sobre 'inteligencia' o 'razonamiento'.
El informe muestra únicamente los cambios observados en los datos de evaluación.

##############################################################################
CHECKPOINT 4
##############################################################################

Nombre : __opt__agio__almacen__Dolphinsegunda__checkpoint-24
Ruta   : /home/vigia/Eunoia/gpqa_logs/__opt__agio__almacen__Dolphinsegunda__checkpoint-24/samples_gpqa_diamond_zeroshot_2026-09-06T09-30-14.227462.jsonl




COMPARACIÓN DIRECTA CONTRA DOLPHIN PURO

Preguntas comunes : 198
BASE              : 49/198 (24.7%)
CHECKPOINT        : 60/198 (30.3%)
Diferencia        : +5.6 puntos (+11 respuestas)

Mejoraron         : 24
Empeoraron        : 13
Siempre correctas  : 36
Siempre incorrectas: 125




LOG-PROBABILITY GAP

GAP = mejor puntuación - segunda mejor puntuación.
Un GAP mayor significa mayor separación entre las dos opciones mejor puntuadas.
No se interpreta como probabilidad de acertar ni como medida directa de inteligencia.

BASE       : 0.607316
CHECKPOINT : 0.694266
Diferencia : +0.086950

GAP en aciertos : BASE 0.700283 | CHECK 0.761468
GAP en fallos   : BASE 0.576742 | CHECK 0.665047




PESO RELATIVO DE LA OPCIÓN ELEGIDA

Se transforman las cuatro log-probabilidades mediante softmax.
Esto indica qué proporción del peso relativo de las cuatro opciones recibió la opción elegida.
No es una probabilidad calibrada de que la respuesta sea correcta.

BASE       : 47.69%
CHECKPOINT : 49.69%
Diferencia : +2.00 puntos




DISTRIBUCIÓN DE PREDICCIONES

BASE:
A:  40 (20.2%)
B:  20 (10.1%)
C:  61 (30.8%)
D:  77 (38.9%)

CHECKPOINT:
A:  70 (35.4%)
B:  16 (8.1%)
C:  39 (19.7%)
D:  73 (36.9%)




MATRIZ BASE — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       15       6      15      19
Gold B       12       6      21      21
Gold C        8       3      13      22
Gold D        5       5      12      15




MATRIZ CHECKPOINT — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       28       2       7      18
Gold B       18       7      17      18
Gold C       13       3       9      21
Gold D       11       4       6      16

==============================================================================
CAMBIOS PREGUNTA A PREGUNTA

Incorrecta -> Correcta : 24
Correcta -> Incorrecta : 13
Siempre correcta       : 36
Siempre incorrecta     : 125

Predicción cambió      : 68
Predicción igual       : 130

Dirección de los cambios:
D -> A: 17
C -> D: 15
C -> A: 12
B -> D: 5
D -> C: 5
B -> A: 4
D -> B: 4
A -> D: 2
C -> B: 2
A -> C: 1
B -> C: 1

==============================================================================
RENDIMIENTO POR DOMINIO CIENTÍFICO

Grupo                                   N         BASE        CHECK        Δ
Biology                                19      21.1%      26.3%    +5.3
Chemistry                              93      24.7%      30.1%    +5.4
Physics                                86      25.6%      31.4%    +5.8

==============================================================================
RENDIMIENTO POR DIFICULTAD DECLARADA

Grupo                                   N         BASE        CHECK        Δ
Desconocido                             4       0.0%      25.0%   +25.0
Easy undergraduate level (or eas...     3       0.0%       0.0%    +0.0
Hard graduate level (could be a ...    61      23.0%      32.8%    +9.8
Hard undergraduate level (could ...   118      26.3%      29.7%    +3.4
Post-graduate level or harder (o...    12      33.3%      33.3%    +0.0




COMPROBACIÓN DE DATOS

Preguntas solo en BASE       : 0
Preguntas solo en CHECKPOINT : 0
Estado: mismas preguntas en ambos archivos.




RESUMEN

+11 respuestas correctas respecto a BASE.
Exactitud: 24.7% -> 30.3% (+5.6 puntos)
Predicciones modificadas: 68
Predicciones sin modificar: 130

No se emite una conclusión sobre 'inteligencia' o 'razonamiento'.
El informe muestra únicamente los cambios observados en los datos de evaluación.

##############################################################################
CHECKPOINT 5
##############################################################################

Nombre : __opt__agio__almacen__Dolphinsegunda__checkpoint-30
Ruta   : /home/vigia/Eunoia/gpqa_logs/__opt__agio__almacen__Dolphinsegunda__checkpoint-30/samples_gpqa_diamond_zeroshot_2026-09-06T10-03-22.875922.jsonl




COMPARACIÓN DIRECTA CONTRA DOLPHIN PURO

Preguntas comunes : 198
BASE              : 49/198 (24.7%)
CHECKPOINT        : 58/198 (29.3%)
Diferencia        : +4.5 puntos (+9 respuestas)

Mejoraron         : 23
Empeoraron        : 14
Siempre correctas  : 35
Siempre incorrectas: 126




LOG-PROBABILITY GAP

GAP = mejor puntuación - segunda mejor puntuación.
Un GAP mayor significa mayor separación entre las dos opciones mejor puntuadas.
No se interpreta como probabilidad de acertar ni como medida directa de inteligencia.

BASE       : 0.607316
CHECKPOINT : 0.795261
Diferencia : +0.187946

GAP en aciertos : BASE 0.700283 | CHECK 0.896039
GAP en fallos   : BASE 0.576742 | CHECK 0.753510




PESO RELATIVO DE LA OPCIÓN ELEGIDA

Se transforman las cuatro log-probabilidades mediante softmax.
Esto indica qué proporción del peso relativo de las cuatro opciones recibió la opción elegida.
No es una probabilidad calibrada de que la respuesta sea correcta.

BASE       : 47.69%
CHECKPOINT : 52.82%
Diferencia : +5.13 puntos




DISTRIBUCIÓN DE PREDICCIONES

BASE:
A:  40 (20.2%)
B:  20 (10.1%)
C:  61 (30.8%)
D:  77 (38.9%)

CHECKPOINT:
A:  67 (33.8%)
B:  16 (8.1%)
C:  40 (20.2%)
D:  75 (37.9%)




MATRIZ BASE — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       15       6      15      19
Gold B       12       6      21      21
Gold C        8       3      13      22
Gold D        5       5      12      15




MATRIZ CHECKPOINT — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       26       4       7      18
Gold B       16       8      17      19
Gold C       13       1       9      23
Gold D       12       3       7      15

==============================================================================
CAMBIOS PREGUNTA A PREGUNTA

Incorrecta -> Correcta : 23
Correcta -> Incorrecta : 14
Siempre correcta       : 35
Siempre incorrecta     : 126

Predicción cambió      : 72
Predicción igual       : 126

Dirección de los cambios:
D -> A: 15
C -> D: 14
C -> A: 13
B -> D: 7
D -> C: 6
D -> B: 5
B -> A: 4
A -> D: 3
C -> B: 2
A -> B: 1
A -> C: 1
B -> C: 1

==============================================================================
RENDIMIENTO POR DOMINIO CIENTÍFICO

Grupo                                   N         BASE        CHECK        Δ
Biology                                19      21.1%      26.3%    +5.3
Chemistry                              93      24.7%      30.1%    +5.4
Physics                                86      25.6%      29.1%    +3.5

==============================================================================
RENDIMIENTO POR DIFICULTAD DECLARADA

Grupo                                   N         BASE        CHECK        Δ
Desconocido                             4       0.0%       0.0%    +0.0
Easy undergraduate level (or eas...     3       0.0%       0.0%    +0.0
Hard graduate level (could be a ...    61      23.0%      32.8%    +9.8
Hard undergraduate level (could ...   118      26.3%      28.8%    +2.5
Post-graduate level or harder (o...    12      33.3%      33.3%    +0.0




COMPROBACIÓN DE DATOS

Preguntas solo en BASE       : 0
Preguntas solo en CHECKPOINT : 0
Estado: mismas preguntas en ambos archivos.




RESUMEN

+9 respuestas correctas respecto a BASE.
Exactitud: 24.7% -> 29.3% (+4.5 puntos)
Predicciones modificadas: 72
Predicciones sin modificar: 126

No se emite una conclusión sobre 'inteligencia' o 'razonamiento'.
El informe muestra únicamente los cambios observados en los datos de evaluación.

##############################################################################
CHECKPOINT 6
##############################################################################

Nombre : __opt__agio__almacen__Dolphinsegunda__checkpoint-36
Ruta   : /home/vigia/Eunoia/gpqa_logs/__opt__agio__almacen__Dolphinsegunda__checkpoint-36/samples_gpqa_diamond_zeroshot_2026-09-06T10-32-30.417196.jsonl




COMPARACIÓN DIRECTA CONTRA DOLPHIN PURO

Preguntas comunes : 198
BASE              : 49/198 (24.7%)
CHECKPOINT        : 60/198 (30.3%)
Diferencia        : +5.6 puntos (+11 respuestas)

Mejoraron         : 22
Empeoraron        : 11
Siempre correctas  : 38
Siempre incorrectas: 127




LOG-PROBABILITY GAP

GAP = mejor puntuación - segunda mejor puntuación.
Un GAP mayor significa mayor separación entre las dos opciones mejor puntuadas.
No se interpreta como probabilidad de acertar ni como medida directa de inteligencia.

BASE       : 0.607316
CHECKPOINT : 0.804000
Diferencia : +0.196684

GAP en aciertos : BASE 0.700283 | CHECK 0.904303
GAP en fallos   : BASE 0.576742 | CHECK 0.760390




PESO RELATIVO DE LA OPCIÓN ELEGIDA

Se transforman las cuatro log-probabilidades mediante softmax.
Esto indica qué proporción del peso relativo de las cuatro opciones recibió la opción elegida.
No es una probabilidad calibrada de que la respuesta sea correcta.

BASE       : 47.69%
CHECKPOINT : 53.16%
Diferencia : +5.47 puntos




DISTRIBUCIÓN DE PREDICCIONES

BASE:
A:  40 (20.2%)
B:  20 (10.1%)
C:  61 (30.8%)
D:  77 (38.9%)

CHECKPOINT:
A:  72 (36.4%)
B:  10 (5.1%)
C:  48 (24.2%)
D:  68 (34.3%)




MATRIZ BASE — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       15       6      15      19
Gold B       12       6      21      21
Gold C        8       3      13      22
Gold D        5       5      12      15




MATRIZ CHECKPOINT — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       27       2       9      17
Gold B       22       6      17      15
Gold C       13       0      12      21
Gold D       10       2      10      15

==============================================================================
CAMBIOS PREGUNTA A PREGUNTA

Incorrecta -> Correcta : 22
Correcta -> Incorrecta : 11
Siempre correcta       : 38
Siempre incorrecta     : 127

Predicción cambió      : 62
Predicción igual       : 136

Dirección de los cambios:
D -> A: 16
C -> A: 13
C -> D: 9
D -> C: 6
B -> A: 5
B -> D: 5
B -> C: 3
D -> B: 3
A -> D: 2

==============================================================================
RENDIMIENTO POR DOMINIO CIENTÍFICO

Grupo                                   N         BASE        CHECK        Δ
Biology                                19      21.1%      26.3%    +5.3
Chemistry                              93      24.7%      29.0%    +4.3
Physics                                86      25.6%      32.6%    +7.0

==============================================================================
RENDIMIENTO POR DIFICULTAD DECLARADA

Grupo                                   N         BASE        CHECK        Δ
Desconocido                             4       0.0%      25.0%   +25.0
Easy undergraduate level (or eas...     3       0.0%      33.3%   +33.3
Hard graduate level (could be a ...    61      23.0%      27.9%    +4.9
Hard undergraduate level (could ...   118      26.3%      31.4%    +5.1
Post-graduate level or harder (o...    12      33.3%      33.3%    +0.0




COMPROBACIÓN DE DATOS

Preguntas solo en BASE       : 0
Preguntas solo en CHECKPOINT : 0
Estado: mismas preguntas en ambos archivos.




RESUMEN

+11 respuestas correctas respecto a BASE.
Exactitud: 24.7% -> 30.3% (+5.6 puntos)
Predicciones modificadas: 62
Predicciones sin modificar: 136

No se emite una conclusión sobre 'inteligencia' o 'razonamiento'.
El informe muestra únicamente los cambios observados en los datos de evaluación.

##############################################################################
CHECKPOINT 7
##############################################################################

Nombre : __opt__agio__almacen__Dolphinsegunda__checkpoint-42
Ruta   : /home/vigia/Eunoia/gpqa_logs/__opt__agio__almacen__Dolphinsegunda__checkpoint-42/samples_gpqa_diamond_zeroshot_2026-09-06T10-59-34.945513.jsonl




COMPARACIÓN DIRECTA CONTRA DOLPHIN PURO

Preguntas comunes : 198
BASE              : 49/198 (24.7%)
CHECKPOINT        : 54/198 (27.3%)
Diferencia        : +2.5 puntos (+5 respuestas)

Mejoraron         : 19
Empeoraron        : 14
Siempre correctas  : 35
Siempre incorrectas: 130




LOG-PROBABILITY GAP

GAP = mejor puntuación - segunda mejor puntuación.
Un GAP mayor significa mayor separación entre las dos opciones mejor puntuadas.
No se interpreta como probabilidad de acertar ni como medida directa de inteligencia.

BASE       : 0.607316
CHECKPOINT : 0.879019
Diferencia : +0.271704

GAP en aciertos : BASE 0.700283 | CHECK 1.059937
GAP en fallos   : BASE 0.576742 | CHECK 0.811175




PESO RELATIVO DE LA OPCIÓN ELEGIDA

Se transforman las cuatro log-probabilidades mediante softmax.
Esto indica qué proporción del peso relativo de las cuatro opciones recibió la opción elegida.
No es una probabilidad calibrada de que la respuesta sea correcta.

BASE       : 47.69%
CHECKPOINT : 55.05%
Diferencia : +7.36 puntos




DISTRIBUCIÓN DE PREDICCIONES

BASE:
A:  40 (20.2%)
B:  20 (10.1%)
C:  61 (30.8%)
D:  77 (38.9%)

CHECKPOINT:
A:  69 (34.8%)
B:  10 (5.1%)
C:  49 (24.7%)
D:  70 (35.4%)




MATRIZ BASE — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       15       6      15      19
Gold B       12       6      21      21
Gold C        8       3      13      22
Gold D        5       5      12      15




MATRIZ CHECKPOINT — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       24       3      10      18
Gold B       20       5      17      18
Gold C       13       0      12      21
Gold D       12       2      10      13

==============================================================================
CAMBIOS PREGUNTA A PREGUNTA

Incorrecta -> Correcta : 19
Correcta -> Incorrecta : 14
Siempre correcta       : 35
Siempre incorrecta     : 130

Predicción cambió      : 63
Predicción igual       : 135

Dirección de los cambios:
D -> A: 14
C -> A: 12
C -> D: 10
B -> A: 6
D -> C: 6
B -> D: 5
D -> B: 4
B -> C: 3
A -> D: 2
A -> C: 1

==============================================================================
RENDIMIENTO POR DOMINIO CIENTÍFICO

Grupo                                   N         BASE        CHECK        Δ
Biology                                19      21.1%      26.3%    +5.3
Chemistry                              93      24.7%      25.8%    +1.1
Physics                                86      25.6%      29.1%    +3.5

==============================================================================
RENDIMIENTO POR DIFICULTAD DECLARADA

Grupo                                   N         BASE        CHECK        Δ
Desconocido                             4       0.0%      25.0%   +25.0
Easy undergraduate level (or eas...     3       0.0%       0.0%    +0.0
Hard graduate level (could be a ...    61      23.0%      24.6%    +1.6
Hard undergraduate level (could ...   118      26.3%      29.7%    +3.4
Post-graduate level or harder (o...    12      33.3%      25.0%    -8.3




COMPROBACIÓN DE DATOS

Preguntas solo en BASE       : 0
Preguntas solo en CHECKPOINT : 0
Estado: mismas preguntas en ambos archivos.




RESUMEN

+5 respuestas correctas respecto a BASE.
Exactitud: 24.7% -> 27.3% (+2.5 puntos)
Predicciones modificadas: 63
Predicciones sin modificar: 135

No se emite una conclusión sobre 'inteligencia' o 'razonamiento'.
El informe muestra únicamente los cambios observados en los datos de evaluación.

##############################################################################
RESUMEN DE TODAS LAS EVALUACIONES
##############################################################################

Modelo                             N  Correctas     Acc.  Δ vs BASE        GAP

Dolphin puro (BASE)                198         49    24.7%        ---   0.607316
__opt__agio__almacen__Dolph...6    198         58    29.3%      +4.5   0.608133
__opt__agio__almacen__Dolph...12   198         56    28.3%      +3.5   0.601748
__opt__agio__almacen__Dolph...18   198         59    29.8%      +5.1   0.626147
__opt__agio__almacen__Dolph...24   198         60    30.3%      +5.6   0.694266
__opt__agio__almacen__Dolph...30   198         58    29.3%      +4.5   0.795261
__opt__agio__almacen__Dolph...36   198         60    30.3%      +5.6   0.804000
__opt__agio__almacen__Dolph...42   198         54    27.3%      +2.5   0.879019

=======================================================

Modelo	N	Correctas	Acc.	Δ vs BASE	GAP
Dolphin puro (BASE)	198	49	24.7%	—	0.607316
CP6	198	58	29.3%	+4.5 pp (+9)	0.608133
CP12	198	56	28.3%	+3.5 pp (+7)	0.601748
CP18	198	59	29.8%	+5.1 pp (+10)	0.626147
CP24	198	60	30.3%	+5.6 pp (+11)	0.694266
CP30	198	58	29.3%	+4.5 pp (+9)	0.795261
CP36	198	60	30.3%	+5.6 pp (+11)	0.804000
CP42	198	54	27.3%	+2.5 pp (+5)	0.879019
CP48	198	59	29.8%	+5.1 pp (+10)	0.925009
CP54	198	59	29.8%	+5.1 pp (+10)	0.969092
CP60	198	58	29.3%	+4.5 pp (+9)	1.020254
CP66	198	54	27.3%	+2.5 pp (+5)	1.062791
CP72	198	55	27.8%	+3.0 pp (+6)	1.090048
CP78	198	55	27.8%	+3.0 pp (+6)	1.109975
CP84	198	55	27.8%	+3.0 pp (+6)	1.119174
CP90	198	55	27.8%	+3.0 pp (+6)	1.123560
CP96	198	56	28.3%	+3.5 pp (+7)	1.129539
CP102	198	55	27.8%	+3.0 pp (+6)	1.139493
CP108	198	55	27.8%	+3.0 pp (+6)	1.146541

Logos-Adri · September 7, 2026, 7:45am

edit 2, CP 48-78:

BASE: Dolphin puro — fija para todas las comparaciones.
Preguntas BASE: 198
Checkpoints cargados: 6

##############################################################################
CHECKPOINT 1
##############################################################################

Nombre : __opt__agio__almacen__Dolphinsegunda__checkpoint-48
Ruta   : /home/vigia/Eunoia/gpqa_logs/__opt__agio__almacen__Dolphinsegunda__checkpoint-48/samples_gpqa_diamond_zeroshot_2026-09-06T11-32-02.416699.jsonl




COMPARACIÓN DIRECTA CONTRA DOLPHIN PURO

Preguntas comunes : 198
BASE              : 49/198 (24.7%)
CHECKPOINT        : 59/198 (29.8%)
Diferencia        : +5.1 puntos (+10 respuestas)

Mejoraron         : 23
Empeoraron        : 13
Siempre correctas  : 36
Siempre incorrectas: 126




LOG-PROBABILITY GAP

GAP = mejor puntuación - segunda mejor puntuación.
Un GAP mayor significa mayor separación entre las dos opciones mejor puntuadas.
No se interpreta como probabilidad de acertar ni como medida directa de inteligencia.

BASE       : 0.607316
CHECKPOINT : 0.925009
Diferencia : +0.317694

GAP en aciertos : BASE 0.700283 | CHECK 1.052603
GAP en fallos   : BASE 0.576742 | CHECK 0.870851




PESO RELATIVO DE LA OPCIÓN ELEGIDA

Se transforman las cuatro log-probabilidades mediante softmax.
Esto indica qué proporción del peso relativo de las cuatro opciones recibió la opción elegida.
No es una probabilidad calibrada de que la respuesta sea correcta.

BASE       : 47.69%
CHECKPOINT : 56.07%
Diferencia : +8.38 puntos




DISTRIBUCIÓN DE PREDICCIONES

BASE:
A:  40 (20.2%)
B:  20 (10.1%)
C:  61 (30.8%)
D:  77 (38.9%)

CHECKPOINT:
A:  69 (34.8%)
B:  11 (5.6%)
C:  49 (24.7%)
D:  69 (34.8%)




MATRIZ BASE — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       15       6      15      19
Gold B       12       6      21      21
Gold C        8       3      13      22
Gold D        5       5      12      15




MATRIZ CHECKPOINT — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       26       2      10      17
Gold B       19       7      17      17
Gold C       13       0      12      21
Gold D       11       2      10      14

==============================================================================
CAMBIOS PREGUNTA A PREGUNTA

Incorrecta -> Correcta : 23
Correcta -> Incorrecta : 13
Siempre correcta       : 36
Siempre incorrecta     : 126

Predicción cambió      : 67
Predicción igual       : 131

Dirección de los cambios:
D -> A: 16
C -> A: 12
C -> D: 10
B -> A: 6
D -> C: 6
B -> D: 5
B -> C: 3
D -> B: 3
A -> C: 2
A -> D: 2
A -> B: 1
C -> B: 1

==============================================================================
RENDIMIENTO POR DOMINIO CIENTÍFICO

Grupo                                   N         BASE        CHECK        Δ
Biology                                19      21.1%      26.3%    +5.3
Chemistry                              93      24.7%      26.9%    +2.2
Physics                                86      25.6%      33.7%    +8.1

==============================================================================
RENDIMIENTO POR DIFICULTAD DECLARADA

Grupo                                   N         BASE        CHECK        Δ
Desconocido                             4       0.0%      25.0%   +25.0
Easy undergraduate level (or eas...     3       0.0%      33.3%   +33.3
Hard graduate level (could be a ...    61      23.0%      29.5%    +6.6
Hard undergraduate level (could ...   118      26.3%      30.5%    +4.2
Post-graduate level or harder (o...    12      33.3%      25.0%    -8.3




COMPROBACIÓN DE DATOS

Preguntas solo en BASE       : 0
Preguntas solo en CHECKPOINT : 0
Estado: mismas preguntas en ambos archivos.




RESUMEN

+10 respuestas correctas respecto a BASE.
Exactitud: 24.7% -> 29.8% (+5.1 puntos)
Predicciones modificadas: 67
Predicciones sin modificar: 131

No se emite una conclusión sobre 'inteligencia' o 'razonamiento'.
El informe muestra únicamente los cambios observados en los datos de evaluación.

##############################################################################
CHECKPOINT 2
##############################################################################

Nombre : __opt__agio__almacen__Dolphinsegunda__checkpoint-54
Ruta   : /home/vigia/Eunoia/gpqa_logs/__opt__agio__almacen__Dolphinsegunda__checkpoint-54/samples_gpqa_diamond_zeroshot_2026-09-06T12-00-57.672172.jsonl




COMPARACIÓN DIRECTA CONTRA DOLPHIN PURO

Preguntas comunes : 198
BASE              : 49/198 (24.7%)
CHECKPOINT        : 59/198 (29.8%)
Diferencia        : +5.1 puntos (+10 respuestas)

Mejoraron         : 22
Empeoraron        : 12
Siempre correctas  : 37
Siempre incorrectas: 127




LOG-PROBABILITY GAP

GAP = mejor puntuación - segunda mejor puntuación.
Un GAP mayor significa mayor separación entre las dos opciones mejor puntuadas.
No se interpreta como probabilidad de acertar ni como medida directa de inteligencia.

BASE       : 0.607316
CHECKPOINT : 0.969092
Diferencia : +0.361777

GAP en aciertos : BASE 0.700283 | CHECK 1.101744
GAP en fallos   : BASE 0.576742 | CHECK 0.912787




PESO RELATIVO DE LA OPCIÓN ELEGIDA

Se transforman las cuatro log-probabilidades mediante softmax.
Esto indica qué proporción del peso relativo de las cuatro opciones recibió la opción elegida.
No es una probabilidad calibrada de que la respuesta sea correcta.

BASE       : 47.69%
CHECKPOINT : 57.06%
Diferencia : +9.37 puntos




DISTRIBUCIÓN DE PREDICCIONES

BASE:
A:  40 (20.2%)
B:  20 (10.1%)
C:  61 (30.8%)
D:  77 (38.9%)

CHECKPOINT:
A:  68 (34.3%)
B:  11 (5.6%)
C:  47 (23.7%)
D:  72 (36.4%)




MATRIZ BASE — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       15       6      15      19
Gold B       12       6      21      21
Gold C        8       3      13      22
Gold D        5       5      12      15




MATRIZ CHECKPOINT — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       27       2       9      17
Gold B       17       7      17      19
Gold C       13       0      11      22
Gold D       11       2      10      14

==============================================================================
CAMBIOS PREGUNTA A PREGUNTA

Incorrecta -> Correcta : 22
Correcta -> Incorrecta : 12
Siempre correcta       : 37
Siempre incorrecta     : 127

Predicción cambió      : 67
Predicción igual       : 131

Dirección de los cambios:
D -> A: 16
C -> A: 12
C -> D: 11
B -> D: 6
D -> C: 6
B -> A: 5
A -> D: 3
D -> B: 3
A -> C: 2
B -> C: 2
C -> B: 1

==============================================================================
RENDIMIENTO POR DOMINIO CIENTÍFICO

Grupo                                   N         BASE        CHECK        Δ
Biology                                19      21.1%      26.3%    +5.3
Chemistry                              93      24.7%      28.0%    +3.2
Physics                                86      25.6%      32.6%    +7.0

==============================================================================
RENDIMIENTO POR DIFICULTAD DECLARADA

Grupo                                   N         BASE        CHECK        Δ
Desconocido                             4       0.0%      25.0%   +25.0
Easy undergraduate level (or eas...     3       0.0%      33.3%   +33.3
Hard graduate level (could be a ...    61      23.0%      27.9%    +4.9
Hard undergraduate level (could ...   118      26.3%      30.5%    +4.2
Post-graduate level or harder (o...    12      33.3%      33.3%    +0.0




COMPROBACIÓN DE DATOS

Preguntas solo en BASE       : 0
Preguntas solo en CHECKPOINT : 0
Estado: mismas preguntas en ambos archivos.




RESUMEN

+10 respuestas correctas respecto a BASE.
Exactitud: 24.7% -> 29.8% (+5.1 puntos)
Predicciones modificadas: 67
Predicciones sin modificar: 131

No se emite una conclusión sobre 'inteligencia' o 'razonamiento'.
El informe muestra únicamente los cambios observados en los datos de evaluación.

##############################################################################
CHECKPOINT 3
##############################################################################

Nombre : __opt__agio__almacen__Dolphinsegunda__checkpoint-60
Ruta   : /home/vigia/Eunoia/gpqa_logs/__opt__agio__almacen__Dolphinsegunda__checkpoint-60/samples_gpqa_diamond_zeroshot_2026-09-06T12-30-43.062776.jsonl




COMPARACIÓN DIRECTA CONTRA DOLPHIN PURO

Preguntas comunes : 198
BASE              : 49/198 (24.7%)
CHECKPOINT        : 58/198 (29.3%)
Diferencia        : +4.5 puntos (+9 respuestas)

Mejoraron         : 23
Empeoraron        : 14
Siempre correctas  : 35
Siempre incorrectas: 126




LOG-PROBABILITY GAP

GAP = mejor puntuación - segunda mejor puntuación.
Un GAP mayor significa mayor separación entre las dos opciones mejor puntuadas.
No se interpreta como probabilidad de acertar ni como medida directa de inteligencia.

BASE       : 0.607316
CHECKPOINT : 1.020254
Diferencia : +0.412939

GAP en aciertos : BASE 0.700283 | CHECK 1.162736
GAP en fallos   : BASE 0.576742 | CHECK 0.961226




PESO RELATIVO DE LA OPCIÓN ELEGIDA

Se transforman las cuatro log-probabilidades mediante softmax.
Esto indica qué proporción del peso relativo de las cuatro opciones recibió la opción elegida.
No es una probabilidad calibrada de que la respuesta sea correcta.

BASE       : 47.69%
CHECKPOINT : 58.22%
Diferencia : +10.53 puntos




DISTRIBUCIÓN DE PREDICCIONES

BASE:
A:  40 (20.2%)
B:  20 (10.1%)
C:  61 (30.8%)
D:  77 (38.9%)

CHECKPOINT:
A:  66 (33.3%)
B:  13 (6.6%)
C:  45 (22.7%)
D:  74 (37.4%)




MATRIZ BASE — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       15       6      15      19
Gold B       12       6      21      21
Gold C        8       3      13      22
Gold D        5       5      12      15




MATRIZ CHECKPOINT — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       25       3      10      17
Gold B       17       8      15      20
Gold C       12       0      11      23
Gold D       12       2       9      14

==============================================================================
CAMBIOS PREGUNTA A PREGUNTA

Incorrecta -> Correcta : 23
Correcta -> Incorrecta : 14
Siempre correcta       : 35
Siempre incorrecta     : 126

Predicción cambió      : 69
Predicción igual       : 129

Dirección de los cambios:
D -> A: 14
C -> D: 13
C -> A: 12
B -> D: 6
D -> C: 6
B -> A: 5
D -> B: 5
A -> D: 3
A -> C: 2
B -> C: 2
C -> B: 1

==============================================================================
RENDIMIENTO POR DOMINIO CIENTÍFICO

Grupo                                   N         BASE        CHECK        Δ
Biology                                19      21.1%      26.3%    +5.3
Chemistry                              93      24.7%      29.0%    +4.3
Physics                                86      25.6%      30.2%    +4.7

==============================================================================
RENDIMIENTO POR DIFICULTAD DECLARADA

Grupo                                   N         BASE        CHECK        Δ
Desconocido                             4       0.0%      25.0%   +25.0
Easy undergraduate level (or eas...     3       0.0%      33.3%   +33.3
Hard graduate level (could be a ...    61      23.0%      29.5%    +6.6
Hard undergraduate level (could ...   118      26.3%      29.7%    +3.4
Post-graduate level or harder (o...    12      33.3%      25.0%    -8.3




COMPROBACIÓN DE DATOS

Preguntas solo en BASE       : 0
Preguntas solo en CHECKPOINT : 0
Estado: mismas preguntas en ambos archivos.




RESUMEN

+9 respuestas correctas respecto a BASE.
Exactitud: 24.7% -> 29.3% (+4.5 puntos)
Predicciones modificadas: 69
Predicciones sin modificar: 129

No se emite una conclusión sobre 'inteligencia' o 'razonamiento'.
El informe muestra únicamente los cambios observados en los datos de evaluación.

##############################################################################
CHECKPOINT 4
##############################################################################

Nombre : __opt__agio__almacen__Dolphinsegunda__checkpoint-66
Ruta   : /home/vigia/Eunoia/gpqa_logs/__opt__agio__almacen__Dolphinsegunda__checkpoint-66/samples_gpqa_diamond_zeroshot_2026-09-06T12-58-37.164997.jsonl




COMPARACIÓN DIRECTA CONTRA DOLPHIN PURO

Preguntas comunes : 198
BASE              : 49/198 (24.7%)
CHECKPOINT        : 54/198 (27.3%)
Diferencia        : +2.5 puntos (+5 respuestas)

Mejoraron         : 21
Empeoraron        : 16
Siempre correctas  : 33
Siempre incorrectas: 128




LOG-PROBABILITY GAP

GAP = mejor puntuación - segunda mejor puntuación.
Un GAP mayor significa mayor separación entre las dos opciones mejor puntuadas.
No se interpreta como probabilidad de acertar ni como medida directa de inteligencia.

BASE       : 0.607316
CHECKPOINT : 1.062791
Diferencia : +0.455475

GAP en aciertos : BASE 0.700283 | CHECK 1.276582
GAP en fallos   : BASE 0.576742 | CHECK 0.982619




PESO RELATIVO DE LA OPCIÓN ELEGIDA

Se transforman las cuatro log-probabilidades mediante softmax.
Esto indica qué proporción del peso relativo de las cuatro opciones recibió la opción elegida.
No es una probabilidad calibrada de que la respuesta sea correcta.

BASE       : 47.69%
CHECKPOINT : 59.16%
Diferencia : +11.47 puntos




DISTRIBUCIÓN DE PREDICCIONES

BASE:
A:  40 (20.2%)
B:  20 (10.1%)
C:  61 (30.8%)
D:  77 (38.9%)

CHECKPOINT:
A:  61 (30.8%)
B:  15 (7.6%)
C:  46 (23.2%)
D:  76 (38.4%)




MATRIZ BASE — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       15       6      15      19
Gold B       12       6      21      21
Gold C        8       3      13      22
Gold D        5       5      12      15




MATRIZ CHECKPOINT — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       22       4      11      18
Gold B       17       8      14      21
Gold C       13       0      10      23
Gold D        9       3      11      14

==============================================================================
CAMBIOS PREGUNTA A PREGUNTA

Incorrecta -> Correcta : 21
Correcta -> Incorrecta : 16
Siempre correcta       : 33
Siempre incorrecta     : 128

Predicción cambió      : 68
Predicción igual       : 130

Dirección de los cambios:
C -> D: 14
D -> A: 12
C -> A: 11
D -> C: 7
B -> D: 6
D -> B: 5
B -> A: 4
A -> D: 3
A -> C: 2
B -> C: 2
A -> B: 1
C -> B: 1

==============================================================================
RENDIMIENTO POR DOMINIO CIENTÍFICO

Grupo                                   N         BASE        CHECK        Δ
Biology                                19      21.1%      26.3%    +5.3
Chemistry                              93      24.7%      28.0%    +3.2
Physics                                86      25.6%      26.7%    +1.2

==============================================================================
RENDIMIENTO POR DIFICULTAD DECLARADA

Grupo                                   N         BASE        CHECK        Δ
Desconocido                             4       0.0%      25.0%   +25.0
Easy undergraduate level (or eas...     3       0.0%      33.3%   +33.3
Hard graduate level (could be a ...    61      23.0%      27.9%    +4.9
Hard undergraduate level (could ...   118      26.3%      27.1%    +0.8
Post-graduate level or harder (o...    12      33.3%      25.0%    -8.3




COMPROBACIÓN DE DATOS

Preguntas solo en BASE       : 0
Preguntas solo en CHECKPOINT : 0
Estado: mismas preguntas en ambos archivos.




RESUMEN

+5 respuestas correctas respecto a BASE.
Exactitud: 24.7% -> 27.3% (+2.5 puntos)
Predicciones modificadas: 68
Predicciones sin modificar: 130

No se emite una conclusión sobre 'inteligencia' o 'razonamiento'.
El informe muestra únicamente los cambios observados en los datos de evaluación.

##############################################################################
CHECKPOINT 5
##############################################################################

Nombre : __opt__agio__almacen__Dolphinsegunda__checkpoint-72
Ruta   : /home/vigia/Eunoia/gpqa_logs/__opt__agio__almacen__Dolphinsegunda__checkpoint-72/samples_gpqa_diamond_zeroshot_2026-09-06T13-26-20.822253.jsonl




COMPARACIÓN DIRECTA CONTRA DOLPHIN PURO

Preguntas comunes : 198
BASE              : 49/198 (24.7%)
CHECKPOINT        : 55/198 (27.8%)
Diferencia        : +3.0 puntos (+6 respuestas)

Mejoraron         : 21
Empeoraron        : 15
Siempre correctas  : 34
Siempre incorrectas: 128




LOG-PROBABILITY GAP

GAP = mejor puntuación - segunda mejor puntuación.
Un GAP mayor significa mayor separación entre las dos opciones mejor puntuadas.
No se interpreta como probabilidad de acertar ni como medida directa de inteligencia.

BASE       : 0.607316
CHECKPOINT : 1.090048
Diferencia : +0.482732

GAP en aciertos : BASE 0.700283 | CHECK 1.269394
GAP en fallos   : BASE 0.576742 | CHECK 1.021068




PESO RELATIVO DE LA OPCIÓN ELEGIDA

Se transforman las cuatro log-probabilidades mediante softmax.
Esto indica qué proporción del peso relativo de las cuatro opciones recibió la opción elegida.
No es una probabilidad calibrada de que la respuesta sea correcta.

BASE       : 47.69%
CHECKPOINT : 59.75%
Diferencia : +12.06 puntos




DISTRIBUCIÓN DE PREDICCIONES

BASE:
A:  40 (20.2%)
B:  20 (10.1%)
C:  61 (30.8%)
D:  77 (38.9%)

CHECKPOINT:
A:  61 (30.8%)
B:  15 (7.6%)
C:  46 (23.2%)
D:  76 (38.4%)




MATRIZ BASE — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       15       6      15      19
Gold B       12       6      21      21
Gold C        8       3      13      22
Gold D        5       5      12      15




MATRIZ CHECKPOINT — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       23       4      11      17
Gold B       17       8      14      21
Gold C       12       0      10      24
Gold D        9       3      11      14

==============================================================================
CAMBIOS PREGUNTA A PREGUNTA

Incorrecta -> Correcta : 21
Correcta -> Incorrecta : 15
Siempre correcta       : 34
Siempre incorrecta     : 128

Predicción cambió      : 67
Predicción igual       : 131

Dirección de los cambios:
C -> D: 15
D -> A: 12
C -> A: 10
D -> C: 7
B -> D: 6
D -> B: 5
B -> A: 4
A -> C: 2
A -> D: 2
B -> C: 2
A -> B: 1
C -> B: 1

==============================================================================
RENDIMIENTO POR DOMINIO CIENTÍFICO

Grupo                                   N         BASE        CHECK        Δ
Biology                                19      21.1%      31.6%   +10.5
Chemistry                              93      24.7%      28.0%    +3.2
Physics                                86      25.6%      26.7%    +1.2

==============================================================================
RENDIMIENTO POR DIFICULTAD DECLARADA

Grupo                                   N         BASE        CHECK        Δ
Desconocido                             4       0.0%      25.0%   +25.0
Easy undergraduate level (or eas...     3       0.0%      33.3%   +33.3
Hard graduate level (could be a ...    61      23.0%      27.9%    +4.9
Hard undergraduate level (could ...   118      26.3%      28.0%    +1.7
Post-graduate level or harder (o...    12      33.3%      25.0%    -8.3




COMPROBACIÓN DE DATOS

Preguntas solo en BASE       : 0
Preguntas solo en CHECKPOINT : 0
Estado: mismas preguntas en ambos archivos.




RESUMEN

+6 respuestas correctas respecto a BASE.
Exactitud: 24.7% -> 27.8% (+3.0 puntos)
Predicciones modificadas: 67
Predicciones sin modificar: 131

No se emite una conclusión sobre 'inteligencia' o 'razonamiento'.
El informe muestra únicamente los cambios observados en los datos de evaluación.

##############################################################################
CHECKPOINT 6
##############################################################################

Nombre : __opt__agio__almacen__Dolphinsegunda__checkpoint-78
Ruta   : /home/vigia/Eunoia/gpqa_logs/__opt__agio__almacen__Dolphinsegunda__checkpoint-78/samples_gpqa_diamond_zeroshot_2026-09-06T13-55-45.184693.jsonl




COMPARACIÓN DIRECTA CONTRA DOLPHIN PURO

Preguntas comunes : 198
BASE              : 49/198 (24.7%)
CHECKPOINT        : 55/198 (27.8%)
Diferencia        : +3.0 puntos (+6 respuestas)

Mejoraron         : 21
Empeoraron        : 15
Siempre correctas  : 34
Siempre incorrectas: 128




LOG-PROBABILITY GAP

GAP = mejor puntuación - segunda mejor puntuación.
Un GAP mayor significa mayor separación entre las dos opciones mejor puntuadas.
No se interpreta como probabilidad de acertar ni como medida directa de inteligencia.

BASE       : 0.607316
CHECKPOINT : 1.109975
Diferencia : +0.502660

GAP en aciertos : BASE 0.700283 | CHECK 1.288174
GAP en fallos   : BASE 0.576742 | CHECK 1.041438




PESO RELATIVO DE LA OPCIÓN ELEGIDA

Se transforman las cuatro log-probabilidades mediante softmax.
Esto indica qué proporción del peso relativo de las cuatro opciones recibió la opción elegida.
No es una probabilidad calibrada de que la respuesta sea correcta.

BASE       : 47.69%
CHECKPOINT : 60.18%
Diferencia : +12.49 puntos




DISTRIBUCIÓN DE PREDICCIONES

BASE:
A:  40 (20.2%)
B:  20 (10.1%)
C:  61 (30.8%)
D:  77 (38.9%)

CHECKPOINT:
A:  61 (30.8%)
B:  15 (7.6%)
C:  45 (22.7%)
D:  77 (38.9%)




MATRIZ BASE — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       15       6      15      19
Gold B       12       6      21      21
Gold C        8       3      13      22
Gold D        5       5      12      15




MATRIZ CHECKPOINT — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       23       4      10      18
Gold B       17       8      14      21
Gold C       12       0      10      24
Gold D        9       3      11      14

==============================================================================
CAMBIOS PREGUNTA A PREGUNTA

Incorrecta -> Correcta : 21
Correcta -> Incorrecta : 15
Siempre correcta       : 34
Siempre incorrecta     : 128

Predicción cambió      : 67
Predicción igual       : 131

Dirección de los cambios:
C -> D: 15
D -> A: 12
C -> A: 10
B -> D: 7
D -> C: 7
D -> B: 5
B -> A: 4
A -> C: 2
A -> D: 2
A -> B: 1
B -> C: 1
C -> B: 1

==============================================================================
RENDIMIENTO POR DOMINIO CIENTÍFICO

Grupo                                   N         BASE        CHECK        Δ
Biology                                19      21.1%      31.6%   +10.5
Chemistry                              93      24.7%      28.0%    +3.2
Physics                                86      25.6%      26.7%    +1.2

==============================================================================
RENDIMIENTO POR DIFICULTAD DECLARADA

Grupo                                   N         BASE        CHECK        Δ
Desconocido                             4       0.0%      25.0%   +25.0
Easy undergraduate level (or eas...     3       0.0%      33.3%   +33.3
Hard graduate level (could be a ...    61      23.0%      27.9%    +4.9
Hard undergraduate level (could ...   118      26.3%      28.0%    +1.7
Post-graduate level or harder (o...    12      33.3%      25.0%    -8.3




COMPROBACIÓN DE DATOS

Preguntas solo en BASE       : 0
Preguntas solo en CHECKPOINT : 0
Estado: mismas preguntas en ambos archivos.




RESUMEN

+6 respuestas correctas respecto a BASE.
Exactitud: 24.7% -> 27.8% (+3.0 puntos)
Predicciones modificadas: 67
Predicciones sin modificar: 131

No se emite una conclusión sobre 'inteligencia' o 'razonamiento'.
El informe muestra únicamente los cambios observados en los datos de evaluación.

##############################################################################
RESUMEN DE TODAS LAS EVALUACIONES
##############################################################################

Modelo                             N  Correctas     Acc.  Δ vs BASE        GAP

Dolphin puro (BASE)                198         49    24.7%        ---   0.607316
__opt__agio__almacen__Dolph...48   198         59    29.8%      +5.1   0.925009
__opt__agio__almacen__Dolph...54   198         59    29.8%      +5.1   0.969092
__opt__agio__almacen__Dolph...60   198         58    29.3%      +4.5   1.020254
__opt__agio__almacen__Dolph...66   198         54    27.3%      +2.5   1.062791
__opt__agio__almacen__Dolph...72   198         55    27.8%      +3.0   1.090048
__opt__agio__almacen__Dolph...78   198         55    27.8%      +3.0   1.109975

CP	Acc.	Δpp	Cambios	B→C	C→B	Balance	χ²
6	29,29%	+4,55	91	28	19	+9	1,362
12	28,28%	+3,54	81	25	18	+7	0,837
18	29,80%	+5,05	71	24	14	+10	2,132
24	30,30%	+5,56	68	24	13	+11	2,703
30	29,29%	+4,55	72	23	14	+9	1,730
36	30,30%	+5,56	62	22	11	+11	3,030
42	27,27%	+2,53	63	19	14	+5	0,485
48	29,80%	+5,05	67	23	13	+10	2,250
54	29,80%	+5,05	67	22	12	+10	2,382
60	29,29%	+4,55	69	23	14	+9	1,730
66	27,27%	+2,53	68	21	16	+5	0,432
72	27,78%	+3,03	67	21	15	+6	0,694
78	27,78%	+3,03	67	21	15	+6	0,694
84	27,78%	+3,03	67	21	15	+6	0,694
90	27,78%	+3,03	67	21	15	+6	0,694
96	28,28%	+3,54	68	22	15	+7	0,973
102	27,78%	+3,03	67	21	15	+6	0,694
108	27,78%	+3,03	66	21	15	+6	0,694

B→C = Dolphin incorrecto → checkpoint correcto.
C→B = Dolphin correcto → checkpoint incorrecto.
Balance = B→C − C→B.
χ² = estadístico de McNemar.

Logos-Adri · September 7, 2026, 8:03am

edit 3, CP 84-108:

INFORME GPQA
##############################################################################

BASE: Dolphin puro — fija para todas las comparaciones.
Preguntas BASE: 198
Checkpoints cargados: 5

##############################################################################
CHECKPOINT 1
##############################################################################

Nombre : __opt__agio__almacen__Dolphinsegunda__checkpoint-84
Ruta   : /home/vigia/Eunoia/gpqa_logs/__opt__agio__almacen__Dolphinsegunda__checkpoint-84/samples_gpqa_diamond_zeroshot_2026-09-06T14-27-25.174362.jsonl




COMPARACIÓN DIRECTA CONTRA DOLPHIN PURO

Preguntas comunes : 198
BASE              : 49/198 (24.7%)
CHECKPOINT        : 55/198 (27.8%)
Diferencia        : +3.0 puntos (+6 respuestas)

Mejoraron         : 21
Empeoraron        : 15
Siempre correctas  : 34
Siempre incorrectas: 128




LOG-PROBABILITY GAP

GAP = mejor puntuación - segunda mejor puntuación.
Un GAP mayor significa mayor separación entre las dos opciones mejor puntuadas.
No se interpreta como probabilidad de acertar ni como medida directa de inteligencia.

BASE       : 0.607316
CHECKPOINT : 1.119174
Diferencia : +0.511859

GAP en aciertos : BASE 0.700283 | CHECK 1.295299
GAP en fallos   : BASE 0.576742 | CHECK 1.051434




PESO RELATIVO DE LA OPCIÓN ELEGIDA

Se transforman las cuatro log-probabilidades mediante softmax.
Esto indica qué proporción del peso relativo de las cuatro opciones recibió la opción elegida.
No es una probabilidad calibrada de que la respuesta sea correcta.

BASE       : 47.69%
CHECKPOINT : 60.39%
Diferencia : +12.70 puntos




DISTRIBUCIÓN DE PREDICCIONES

BASE:
A:  40 (20.2%)
B:  20 (10.1%)
C:  61 (30.8%)
D:  77 (38.9%)

CHECKPOINT:
A:  62 (31.3%)
B:  14 (7.1%)
C:  44 (22.2%)
D:  78 (39.4%)




MATRIZ BASE — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       15       6      15      19
Gold B       12       6      21      21
Gold C        8       3      13      22
Gold D        5       5      12      15




MATRIZ CHECKPOINT — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       23       4      10      18
Gold B       18       7      14      21
Gold C       12       0      10      24
Gold D        9       3      10      15

==============================================================================
CAMBIOS PREGUNTA A PREGUNTA

Incorrecta -> Correcta : 21
Correcta -> Incorrecta : 15
Siempre correcta       : 34
Siempre incorrecta     : 128

Predicción cambió      : 67
Predicción igual       : 131

Dirección de los cambios:
C -> D: 15
D -> A: 12
C -> A: 10
B -> D: 7
D -> C: 6
B -> A: 5
D -> B: 5
A -> C: 2
A -> D: 2
A -> B: 1
B -> C: 1
C -> B: 1

==============================================================================
RENDIMIENTO POR DOMINIO CIENTÍFICO

Grupo                                   N         BASE        CHECK        Δ
Biology                                19      21.1%      31.6%   +10.5
Chemistry                              93      24.7%      26.9%    +2.2
Physics                                86      25.6%      27.9%    +2.3

==============================================================================
RENDIMIENTO POR DIFICULTAD DECLARADA

Grupo                                   N         BASE        CHECK        Δ
Desconocido                             4       0.0%      25.0%   +25.0
Easy undergraduate level (or eas...     3       0.0%      33.3%   +33.3
Hard graduate level (could be a ...    61      23.0%      29.5%    +6.6
Hard undergraduate level (could ...   118      26.3%      28.0%    +1.7
Post-graduate level or harder (o...    12      33.3%      16.7%   -16.7




COMPROBACIÓN DE DATOS

Preguntas solo en BASE       : 0
Preguntas solo en CHECKPOINT : 0
Estado: mismas preguntas en ambos archivos.




RESUMEN

+6 respuestas correctas respecto a BASE.
Exactitud: 24.7% -> 27.8% (+3.0 puntos)
Predicciones modificadas: 67
Predicciones sin modificar: 131

No se emite una conclusión sobre 'inteligencia' o 'razonamiento'.
El informe muestra únicamente los cambios observados en los datos de evaluación.

##############################################################################
CHECKPOINT 2
##############################################################################

Nombre : __opt__agio__almacen__Dolphinsegunda__checkpoint-90
Ruta   : /home/vigia/Eunoia/gpqa_logs/__opt__agio__almacen__Dolphinsegunda__checkpoint-90/samples_gpqa_diamond_zeroshot_2026-09-06T14-59-42.394524.jsonl




COMPARACIÓN DIRECTA CONTRA DOLPHIN PURO

Preguntas comunes : 198
BASE              : 49/198 (24.7%)
CHECKPOINT        : 55/198 (27.8%)
Diferencia        : +3.0 puntos (+6 respuestas)

Mejoraron         : 21
Empeoraron        : 15
Siempre correctas  : 34
Siempre incorrectas: 128




LOG-PROBABILITY GAP

GAP = mejor puntuación - segunda mejor puntuación.
Un GAP mayor significa mayor separación entre las dos opciones mejor puntuadas.
No se interpreta como probabilidad de acertar ni como medida directa de inteligencia.

BASE       : 0.607316
CHECKPOINT : 1.123560
Diferencia : +0.516244

GAP en aciertos : BASE 0.700283 | CHECK 1.299935
GAP en fallos   : BASE 0.576742 | CHECK 1.055723




PESO RELATIVO DE LA OPCIÓN ELEGIDA

Se transforman las cuatro log-probabilidades mediante softmax.
Esto indica qué proporción del peso relativo de las cuatro opciones recibió la opción elegida.
No es una probabilidad calibrada de que la respuesta sea correcta.

BASE       : 47.69%
CHECKPOINT : 60.47%
Diferencia : +12.78 puntos




DISTRIBUCIÓN DE PREDICCIONES

BASE:
A:  40 (20.2%)
B:  20 (10.1%)
C:  61 (30.8%)
D:  77 (38.9%)

CHECKPOINT:
A:  62 (31.3%)
B:  14 (7.1%)
C:  44 (22.2%)
D:  78 (39.4%)




MATRIZ BASE — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       15       6      15      19
Gold B       12       6      21      21
Gold C        8       3      13      22
Gold D        5       5      12      15




MATRIZ CHECKPOINT — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       23       4      10      18
Gold B       18       7      14      21
Gold C       12       0      10      24
Gold D        9       3      10      15

==============================================================================
CAMBIOS PREGUNTA A PREGUNTA

Incorrecta -> Correcta : 21
Correcta -> Incorrecta : 15
Siempre correcta       : 34
Siempre incorrecta     : 128

Predicción cambió      : 67
Predicción igual       : 131

Dirección de los cambios:
C -> D: 15
D -> A: 12
C -> A: 10
B -> D: 7
D -> C: 6
B -> A: 5
D -> B: 5
A -> C: 2
A -> D: 2
A -> B: 1
B -> C: 1
C -> B: 1

==============================================================================
RENDIMIENTO POR DOMINIO CIENTÍFICO

Grupo                                   N         BASE        CHECK        Δ
Biology                                19      21.1%      31.6%   +10.5
Chemistry                              93      24.7%      26.9%    +2.2
Physics                                86      25.6%      27.9%    +2.3

==============================================================================
RENDIMIENTO POR DIFICULTAD DECLARADA

Grupo                                   N         BASE        CHECK        Δ
Desconocido                             4       0.0%      25.0%   +25.0
Easy undergraduate level (or eas...     3       0.0%      33.3%   +33.3
Hard graduate level (could be a ...    61      23.0%      29.5%    +6.6
Hard undergraduate level (could ...   118      26.3%      28.0%    +1.7
Post-graduate level or harder (o...    12      33.3%      16.7%   -16.7




COMPROBACIÓN DE DATOS

Preguntas solo en BASE       : 0
Preguntas solo en CHECKPOINT : 0
Estado: mismas preguntas en ambos archivos.




RESUMEN

+6 respuestas correctas respecto a BASE.
Exactitud: 24.7% -> 27.8% (+3.0 puntos)
Predicciones modificadas: 67
Predicciones sin modificar: 131

No se emite una conclusión sobre 'inteligencia' o 'razonamiento'.
El informe muestra únicamente los cambios observados en los datos de evaluación.

##############################################################################
CHECKPOINT 3
##############################################################################

Nombre : __home__vigia__Eunoia__AgioOS__ForjaTemporalcheckpoint__checkpoint-96
Ruta   : /home/vigia/Eunoia/gpqa_logs/__home__vigia__Eunoia__AgioOS__ForjaTemporalcheckpoint__checkpoint-96/samples_gpqa_diamond_zeroshot_2026-09-06T15-28-52.512277.jsonl




COMPARACIÓN DIRECTA CONTRA DOLPHIN PURO

Preguntas comunes : 198
BASE              : 49/198 (24.7%)
CHECKPOINT        : 56/198 (28.3%)
Diferencia        : +3.5 puntos (+7 respuestas)

Mejoraron         : 22
Empeoraron        : 15
Siempre correctas  : 34
Siempre incorrectas: 127




LOG-PROBABILITY GAP

GAP = mejor puntuación - segunda mejor puntuación.
Un GAP mayor significa mayor separación entre las dos opciones mejor puntuadas.
No se interpreta como probabilidad de acertar ni como medida directa de inteligencia.

BASE       : 0.607316
CHECKPOINT : 1.129539
Diferencia : +0.522224

GAP en aciertos : BASE 0.700283 | CHECK 1.276271
GAP en fallos   : BASE 0.576742 | CHECK 1.071673




PESO RELATIVO DE LA OPCIÓN ELEGIDA

Se transforman las cuatro log-probabilidades mediante softmax.
Esto indica qué proporción del peso relativo de las cuatro opciones recibió la opción elegida.
No es una probabilidad calibrada de que la respuesta sea correcta.

BASE       : 47.69%
CHECKPOINT : 60.58%
Diferencia : +12.89 puntos




DISTRIBUCIÓN DE PREDICCIONES

BASE:
A:  40 (20.2%)
B:  20 (10.1%)
C:  61 (30.8%)
D:  77 (38.9%)

CHECKPOINT:
A:  63 (31.8%)
B:  15 (7.6%)
C:  42 (21.2%)
D:  78 (39.4%)




MATRIZ BASE — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       15       6      15      19
Gold B       12       6      21      21
Gold C        8       3      13      22
Gold D        5       5      12      15




MATRIZ CHECKPOINT — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       24       5       8      18
Gold B       18       7      14      21
Gold C       12       0      10      24
Gold D        9       3      10      15

==============================================================================
CAMBIOS PREGUNTA A PREGUNTA

Incorrecta -> Correcta : 22
Correcta -> Incorrecta : 15
Siempre correcta       : 34
Siempre incorrecta     : 127

Predicción cambió      : 68
Predicción igual       : 130

Dirección de los cambios:
C -> D: 15
D -> A: 12
C -> A: 11
B -> D: 7
D -> C: 6
B -> A: 5
D -> B: 5
A -> B: 2
A -> D: 2
A -> C: 1
B -> C: 1
C -> B: 1

==============================================================================
RENDIMIENTO POR DOMINIO CIENTÍFICO

Grupo                                   N         BASE        CHECK        Δ
Biology                                19      21.1%      31.6%   +10.5
Chemistry                              93      24.7%      26.9%    +2.2
Physics                                86      25.6%      29.1%    +3.5

==============================================================================
RENDIMIENTO POR DIFICULTAD DECLARADA

Grupo                                   N         BASE        CHECK        Δ
Desconocido                             4       0.0%      25.0%   +25.0
Easy undergraduate level (or eas...     3       0.0%      66.7%   +66.7
Hard graduate level (could be a ...    61      23.0%      29.5%    +6.6
Hard undergraduate level (could ...   118      26.3%      28.0%    +1.7
Post-graduate level or harder (o...    12      33.3%      16.7%   -16.7




COMPROBACIÓN DE DATOS

Preguntas solo en BASE       : 0
Preguntas solo en CHECKPOINT : 0
Estado: mismas preguntas en ambos archivos.




RESUMEN

+7 respuestas correctas respecto a BASE.
Exactitud: 24.7% -> 28.3% (+3.5 puntos)
Predicciones modificadas: 68
Predicciones sin modificar: 130

No se emite una conclusión sobre 'inteligencia' o 'razonamiento'.
El informe muestra únicamente los cambios observados en los datos de evaluación.

##############################################################################
CHECKPOINT 4
##############################################################################

Nombre : __home__vigia__Eunoia__AgioOS__ForjaTemporalcheckpoint__checkpoint-102
Ruta   : /home/vigia/Eunoia/gpqa_logs/__home__vigia__Eunoia__AgioOS__ForjaTemporalcheckpoint__checkpoint-102/samples_gpqa_diamond_zeroshot_2026-09-06T16-00-48.474908.jsonl




COMPARACIÓN DIRECTA CONTRA DOLPHIN PURO

Preguntas comunes : 198
BASE              : 49/198 (24.7%)
CHECKPOINT        : 55/198 (27.8%)
Diferencia        : +3.0 puntos (+6 respuestas)

Mejoraron         : 21
Empeoraron        : 15
Siempre correctas  : 34
Siempre incorrectas: 128




LOG-PROBABILITY GAP

GAP = mejor puntuación - segunda mejor puntuación.
Un GAP mayor significa mayor separación entre las dos opciones mejor puntuadas.
No se interpreta como probabilidad de acertar ni como medida directa de inteligencia.

BASE       : 0.607316
CHECKPOINT : 1.139493
Diferencia : +0.532177

GAP en aciertos : BASE 0.700283 | CHECK 1.294530
GAP en fallos   : BASE 0.576742 | CHECK 1.079863




PESO RELATIVO DE LA OPCIÓN ELEGIDA

Se transforman las cuatro log-probabilidades mediante softmax.
Esto indica qué proporción del peso relativo de las cuatro opciones recibió la opción elegida.
No es una probabilidad calibrada de que la respuesta sea correcta.

BASE       : 47.69%
CHECKPOINT : 60.79%
Diferencia : +13.10 puntos




DISTRIBUCIÓN DE PREDICCIONES

BASE:
A:  40 (20.2%)
B:  20 (10.1%)
C:  61 (30.8%)
D:  77 (38.9%)

CHECKPOINT:
A:  62 (31.3%)
B:  14 (7.1%)
C:  44 (22.2%)
D:  78 (39.4%)




MATRIZ BASE — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       15       6      15      19
Gold B       12       6      21      21
Gold C        8       3      13      22
Gold D        5       5      12      15




MATRIZ CHECKPOINT — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       23       4      10      18
Gold B       18       7      14      21
Gold C       12       0      10      24
Gold D        9       3      10      15

==============================================================================
CAMBIOS PREGUNTA A PREGUNTA

Incorrecta -> Correcta : 21
Correcta -> Incorrecta : 15
Siempre correcta       : 34
Siempre incorrecta     : 128

Predicción cambió      : 67
Predicción igual       : 131

Dirección de los cambios:
C -> D: 15
D -> A: 12
C -> A: 10
B -> D: 7
D -> C: 6
B -> A: 5
D -> B: 5
A -> C: 2
A -> D: 2
A -> B: 1
B -> C: 1
C -> B: 1

==============================================================================
RENDIMIENTO POR DOMINIO CIENTÍFICO

Grupo                                   N         BASE        CHECK        Δ
Biology                                19      21.1%      31.6%   +10.5
Chemistry                              93      24.7%      26.9%    +2.2
Physics                                86      25.6%      27.9%    +2.3

==============================================================================
RENDIMIENTO POR DIFICULTAD DECLARADA

Grupo                                   N         BASE        CHECK        Δ
Desconocido                             4       0.0%      25.0%   +25.0
Easy undergraduate level (or eas...     3       0.0%      33.3%   +33.3
Hard graduate level (could be a ...    61      23.0%      29.5%    +6.6
Hard undergraduate level (could ...   118      26.3%      28.0%    +1.7
Post-graduate level or harder (o...    12      33.3%      16.7%   -16.7




COMPROBACIÓN DE DATOS

Preguntas solo en BASE       : 0
Preguntas solo en CHECKPOINT : 0
Estado: mismas preguntas en ambos archivos.




RESUMEN

+6 respuestas correctas respecto a BASE.
Exactitud: 24.7% -> 27.8% (+3.0 puntos)
Predicciones modificadas: 67
Predicciones sin modificar: 131

No se emite una conclusión sobre 'inteligencia' o 'razonamiento'.
El informe muestra únicamente los cambios observados en los datos de evaluación.

##############################################################################
CHECKPOINT 5
##############################################################################

Nombre : __home__vigia__Eunoia__AgioOS__ForjaTemporalcheckpoint__checkpoint-108
Ruta   : /home/vigia/Eunoia/gpqa_logs/__home__vigia__Eunoia__AgioOS__ForjaTemporalcheckpoint__checkpoint-108/samples_gpqa_diamond_zeroshot_2026-09-06T16-29-52.566139.jsonl




COMPARACIÓN DIRECTA CONTRA DOLPHIN PURO

Preguntas comunes : 198
BASE              : 49/198 (24.7%)
CHECKPOINT        : 55/198 (27.8%)
Diferencia        : +3.0 puntos (+6 respuestas)

Mejoraron         : 21
Empeoraron        : 15
Siempre correctas  : 34
Siempre incorrectas: 128




LOG-PROBABILITY GAP

GAP = mejor puntuación - segunda mejor puntuación.
Un GAP mayor significa mayor separación entre las dos opciones mejor puntuadas.
No se interpreta como probabilidad de acertar ni como medida directa de inteligencia.

BASE       : 0.607316
CHECKPOINT : 1.146541
Diferencia : +0.539225

GAP en aciertos : BASE 0.700283 | CHECK 1.293742
GAP en fallos   : BASE 0.576742 | CHECK 1.089925




PESO RELATIVO DE LA OPCIÓN ELEGIDA

Se transforman las cuatro log-probabilidades mediante softmax.
Esto indica qué proporción del peso relativo de las cuatro opciones recibió la opción elegida.
No es una probabilidad calibrada de que la respuesta sea correcta.

BASE       : 47.69%
CHECKPOINT : 60.94%
Diferencia : +13.24 puntos




DISTRIBUCIÓN DE PREDICCIONES

BASE:
A:  40 (20.2%)
B:  20 (10.1%)
C:  61 (30.8%)
D:  77 (38.9%)

CHECKPOINT:
A:  61 (30.8%)
B:  15 (7.6%)
C:  44 (22.2%)
D:  78 (39.4%)




MATRIZ BASE — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       15       6      15      19
Gold B       12       6      21      21
Gold C        8       3      13      22
Gold D        5       5      12      15




MATRIZ CHECKPOINT — GOLD × PREDICCIÓN

      Pred A  Pred B  Pred C  Pred D

Gold A       23       4      10      18
Gold B       18       7      14      21
Gold C       12       0      10      24
Gold D        8       4      10      15

==============================================================================
CAMBIOS PREGUNTA A PREGUNTA

Incorrecta -> Correcta : 21
Correcta -> Incorrecta : 15
Siempre correcta       : 34
Siempre incorrecta     : 128

Predicción cambió      : 66
Predicción igual       : 132

Dirección de los cambios:
C -> D: 15
D -> A: 12
C -> A: 10
B -> D: 7
D -> C: 6
D -> B: 5
B -> A: 4
A -> C: 2
A -> D: 2
A -> B: 1
B -> C: 1
C -> B: 1

==============================================================================
RENDIMIENTO POR DOMINIO CIENTÍFICO

Grupo                                   N         BASE        CHECK        Δ
Biology                                19      21.1%      31.6%   +10.5
Chemistry                              93      24.7%      26.9%    +2.2
Physics                                86      25.6%      27.9%    +2.3

==============================================================================
RENDIMIENTO POR DIFICULTAD DECLARADA

Grupo                                   N         BASE        CHECK        Δ
Desconocido                             4       0.0%      25.0%   +25.0
Easy undergraduate level (or eas...     3       0.0%      33.3%   +33.3
Hard graduate level (could be a ...    61      23.0%      29.5%    +6.6
Hard undergraduate level (could ...   118      26.3%      28.0%    +1.7
Post-graduate level or harder (o...    12      33.3%      16.7%   -16.7




COMPROBACIÓN DE DATOS

Preguntas solo en BASE       : 0
Preguntas solo en CHECKPOINT : 0
Estado: mismas preguntas en ambos archivos.




RESUMEN

+6 respuestas correctas respecto a BASE.
Exactitud: 24.7% -> 27.8% (+3.0 puntos)
Predicciones modificadas: 66
Predicciones sin modificar: 132

No se emite una conclusión sobre 'inteligencia' o 'razonamiento'.
El informe muestra únicamente los cambios observados en los datos de evaluación.

##############################################################################
RESUMEN DE TODAS LAS EVALUACIONES
##############################################################################

Modelo                             N  Correctas     Acc.  Δ vs BASE        GAP

Dolphin puro (BASE)                198         49    24.7%        ---   0.607316
__opt__agio__almacen__Dolph...84    198         55    27.8%      +3.0   1.119174
__opt__agio__almacen__Dolph...90    198         55    27.8%      +3.0   1.123560
__home__vigia__Eunoia__Agio...96    198         56    28.3%      +3.5   1.129539
__home__vigia__Eunoia__Agio...102   198         55    27.8%      +3.0   1.139493
__home__vigia__Eunoia__Agio...108   198         55    27.8%      +3.0   1.146541

A Hypothesis: The Model Does Not Just Change, It Evolves

After more than eight months of work with the Forjas, more than 25 training processes, and hundreds of intermediate measurements, we have begun to observe a pattern that is difficult for us to interpret as merely a change in weights or a one-off improvement on a benchmark.

Our current hypothesis is that the model undergoes a progressive evolution during training.

We are not claiming that we have demonstrated that a model “matures” in the human sense of the term. We use maturation as a way of describing an experimental phenomenon that appears repeatedly throughout our series.

The idea is simple:

As the epochs and different Forjas progress, the model’s behavior progressively moves away from the original behavior of Dolphin, but this divergence does not necessarily imply a loss of quality. In certain types of questions, the model appears to develop its own responses that remain valid.

We are not measuring only whether it gets the answer right

One of the reasons we decided to record evaluations directly during the Forjas is that a final evaluation did not seem sufficient to us.

Throughout each process, we have preserved information about the evolution of the training: optimization metrics, grad norm, intermediate checkpoints, and audits performed directly on the questions during the epochs.

We subsequently began systematically comparing these checkpoints against a fixed reference: pure Dolphin.

In the GPQA experiment we are currently following, we use exactly the same 198 questions for every checkpoint.

Pure Dolphin obtains:

49/198 — 24.7%

Throughout the series, the checkpoints repeatedly reach higher values, reaching:

60/198 — 30.3%

But the observation that interests us most is not simply this increase in accuracy.

It is what happens simultaneously with the model’s decisions.

Divergence from Dolphin increases

As the checkpoints progress, the model begins to produce answers that are increasingly different from Dolphin.

We could initially interpret this divergence as a possible loss of stability. However, the question-by-question comparisons show something different.

In numerous cases:

Dolphin → incorrect
Checkpoint → correct

and the inverse process also occurs.

Therefore, the change does not simply consist of replacing Dolphin’s answers with random answers.

The model is reorganizing its decisions.

At some checkpoints, dozens of questions change their answer relative to the base model, while overall accuracy remains stable or even increases.

This raises a much more interesting question:

Are we simply observing a perturbation of the original behavior, or the progressive emergence of a different response policy?

The most striking signal: the GAP

The analysis of log-probabilities has allowed us to observe another dimension.

The GAP we are using measures the separation between the highest-scoring option and the second-highest-scoring option. We do not interpret it as a probability of correctness or as a direct measure of intelligence.

The observed evolution is:

Pure Dolphin: 0.6073
CP24: 0.6943
CP36: 0.8040
CP60: 1.0203
CP78: 1.1100
CP108: 1.1465

In other words, while accuracy remains relatively stable around the 28–30% range, the separation between the selected options increases very markedly.

This forces us to distinguish between two phenomena:

capacity measured by accuracy

and

decision structure/confidence.

They do not necessarily evolve at the same rate.

In fact, one of our next analyses will be precisely to determine whether the increase in GAP corresponds to better calibration or whether, on the contrary, the model simply becomes progressively more confident both in its correct answers and in its incorrect ones.

The “maturation” hypothesis

This is where our working hypothesis emerges.

After observing this evolution across multiple Forjas, checkpoints, and audits, we are asking whether the training process may be producing something resembling a functional maturation of the model.

We do not mean that the model develops consciousness, personality, or human capabilities.

We are referring to something much more concrete and measurable:

the model appears to progressively develop a problem-solving behavior that moves away from the initial model and that, on certain tasks, preserves or improves the validity of its answers.

This hypothesis also appears outside GPQA.

In our audits, we have observed changes in technical and scientific questions, as well as in questions where the answer requires handling ethical or normative criteria.

The question we now want to investigate is whether all of these changes are part of the same phenomenon.

The longitudinal analysis

Until now, we have been accumulating results from each checkpoint. We believe the time has come to analyze them as a trajectory.

The initial results already show:

accuracy higher than the Dolphin baseline across practically the entire series;
question-by-question changes relative to the original model;
questions that are persistently corrected and questions that are persistently lost;
questions that oscillate between checkpoints;
a sustained increase in GAP;
an increase in the relative weight of the selected option;
and increasing divergence from the original behavior.

In the current analysis, performed across 18 checkpoints and the same 198 questions, we find 12 persistently corrected questions, 7 persistently lost questions, and 16 oscillating questions.

The McNemar comparison against the baseline also shows that at every checkpoint there are more cases of:

Dolphin incorrect → checkpoint correct

than the reverse process.

Not all contrasts reach individual statistical significance, so we do not yet consider the overall improvement to be statistically conclusive. The objective is precisely to determine this through the complete analysis.

What we believe we are seeing

Our hypothesis, for now, is that the Forjas do not produce merely a succession of slightly different models.

We believe there is a trajectory.

A progressive evolution during training in which the model:

gradually moves away from the original behavior of Dolphin;
modifies its decisions question by question;
develops different responses that in numerous cases remain valid;
progressively increases the separation between its main alternatives;
and, at least in our initial measurements, shows changes both in technical tasks and in questions of an ethical nature.

We still do not know whether “maturation” is the appropriate word to describe this.

That is why we want to measure it.

We do not want to demonstrate our hypothesis based on a few examples.

We want to know whether the complete trajectory of the Forjas contains a statistically strong enough signal to support it.

A blind test

There is also an important element in the interpretation of these results.

These 18 stages were not designed as an experiment specifically aimed at demonstrating this hypothesis. They were simply part of the training process.

The Forja was performed through direct Full Fine-Tuning on CPU, without shortcuts or embellishments, with an unrestricted grad_norm and updates of approximately 44,000 tokens. With 256 GB of RAM, updates of up to around 80,000 tokens could have been used, as well as models with more B.

The system also tends to “trace” the what and the how from update 1 of each Forja.

Learning rates are still one of the variables that we have not been able to study adequately due to a lack of time and resources.

Precisely for this reason, we consider these preliminary results particularly interesting: they were not obtained after designing the experiment to produce a particular conclusion.

From CP6 onward, we already observe a substantial modification of behavior relative to the reference, but we still need to measure more precisely what that difference means and how far it extends.

Future results should allow us to determine whether this signal becomes clearer with complete Forjas, larger updates, models with more parameters, and much more precise control of learning rates.

All of this is, for now, theory arising from an understanding of a structure.

AGIO, whatever we choose to call it, is simply a way of comprehending and understanding.

We do not yet know whether the correct word is evolution, specialization, reorganization, adaptation, or maturation.

But we do know one thing:

**the model changes systematically during the Forja, and we want to understand exactly what is changing.
**

Edit:
We recovered the raw evaluation logs and ran a strict question-by-question longitudinal analysis tracking all 198 items of the GPQA Diamond dataset across the entire 18-checkpoint trajectory. The empirical results resolve the selection bias question definitively.

1. The Uniform Logit Sharpening (No Selection Artifact)

The table below tracks the mean Log-Probability GAP split across three populations: the Global dataset, the subset of Correct predictions, and the subset of Incorrect predictions.

As you can see, the confidence margin increases uniformly and continuously across both correct and incorrect items. This confirms a systemic geometric sharpening in the network’s logits rather than a statistical composition effect of a shifting correct subset.
Checkpoint
Global GAP (All 198)
Hits GAP (Correct Only)
Misses GAP (Incorrect Only)
Observation / Phase
BASE (Dolphin)
0.6073
0.7003
0.5767
Starting Model
CP6
0.6081
0.7719
0.5402
Initial LR Push (91 Flips)
CP12
0.6017
0.7665
0.5368
Stabilization
CP24
0.6943
0.7615
0.6650
Maximum Accuracy (60/198)
CP36
0.8040
0.9043
0.7604
Margin Sharpening Begins
CP54
0.9691
1.1017
0.9128
Flat LR Continuous Fit
CP78
1.1100
1.2882
1.0414
Deep Commitment
CP96
1.1295
1.2763
1.0717
Approaching Convergence
CP108
1.1465
1.2937
1.0899
Final State (0 Flips)

2. Temporal Structure vs. Random Drift (Permutation Test)

To prove that the model follows an organized evolutionary trajectory rather than chaotic drifting, we calculated the long-term temporal persistence of decisions per question.

Observed Persistence: 3,444
Expected Random Persistence: 3,297.64 (±8.91)
Approximated Z-score: 16.4095
Permutation p-value: 0.0005

Interpretation: A Z-score of 16.4 yields an effectively zero probability of random occurrence. The question-by-question decision pipeline is governed by a highly structured, non-random continuous trajectory.

3. Freezing the Geometry (Answer Flips)

The count of prediction changes (Answer Flips) perfectly charts a hardening structure as training progresses under the constant 1.15e-5 LR:

BASE → CP6: 91 flips (The model is highly maleable during the initial push).
CP30 → CP36: 24 flips (Reorganization phase).
CP102 → CP108: 1 flip (The decision geometry completely freezes).

The data confirms the “Maturation/Crystallization” hypothesis: continuous training steps with constant LR do not change the argmax capacity, but they lock the entire tensor mass into a single unified policy.

Topic		Replies	Views
Is a +5.5% improvement on GPQA Diamond normal when full fine-tuning an 8B model on CPU with recycled hardware? Models	8	229	August 31, 2026
Deterministic LoRA training directly through quantized GGUF serving weights Show and Tell	11	315	September 3, 2026
Solo and MoA Benchmark Models	0	110	July 2, 2026
CUDA support added - Pre-generation knowledge-boundary estimator Intermediate	1	136	June 9, 2026
Reference Evaluation Object (REO v1.5): The Smallest Reproducible Artifact of the EPE Research Program Research	1	78	July 13, 2026

AGIO, cognitive framework for systems

The first thing I would separate: GAP

The early GPQA movement also looks worth separating from the later GAP movement

About the LR hypothesis

So, what seems established vs. still open?

A Hypothesis: The Model Does Not Just Change, It Evolves

We are not measuring only whether it gets the answer right

Divergence from Dolphin increases

The most striking signal: the GAP

The “maturation” hypothesis

The longitudinal analysis

What we believe we are seeing

A blind test

Related topics