Fix Specialists
#56
by Banaxi-Tech - opened
Fix Specialists and Half Specialists by classifying a model as specialist if the Arithmark 2 score is more than 30% higher in points than Hellaswag. This fixes models like Atom 2.7 While Also Fixing models like Nexus Erebus 135M that train on alot of synthetic arithmetic and almost no actual web corpus that didn't get flagged as specialist before. For models that actually perform like this there is a exclusion list. This should finally fix all the specialists
What this will change:
All current specialist models will still be specialist.
It will add Nexus Erebus 135M and similar models as a specialist (currently only nexus Erebus 135M)
Hi Banaxi, we have our own fix in the works which will be coming out soon!
Datdanboi25 changed pull request status to closed
Ok thanks