EQ-Bench score
Hi, I just benchmarked your model with EQ-Bench and got 79.75, which is a hair below Megadolphin, the current open source leader. Would love to get a peek at the dataset you're training on. It's very impressive for a 7b model.
Great but hey didn't see it on your leaderboard.
I wasn't aware of this testing, you've been doing this for a while.
TheBloke/koala-7B-HF was literally the first model TheBloke learned GPTQ on.
Only since december last year, but I guess time moves fast here. I tested a bunch of older models to round out the lower end of the leaderboard.
Re: adding it to the leaderboard. I'm a bit cautious with new models that overperform, so was going to wait for more info on the training set.
Or, if you aren't planning to release the dataset, I might ask /u/WolframRavenwolf to put it through its paces.
Understood, I would like to clarify that the training methodology I employ is proprietary at this time until a wall is reach at which time I'm planning a white paper. Thanks for the letting me know about the EQ benchmark, I'm running it against v3 now.
Fair enough! I'll be interested to see the v3 score.
Interesting...
This evaluates the model's capability to discern emotions from dialogue?
That's a pretty insane score!
To split hairs, it's more testing the ability to predict emotional responses. So, more like "how will Samantha feel after this dialogue", as opposed to "identify the emotions Samantha expressed in this dialogue". The former is a lot harder for language models than the latter.
When will v3 be released?
I'm also curious how v3 is coming along! Also the white paper for your training methodology.
Sam, I actually made v3 quite a while back and it's EQ score is much higher but a company paid for it's exclusive rights.
Sam, I actually made v3 quite a while back and it's EQ score is much higher but a company paid for it's exclusive rights.
Congrats! Also, boo.
its so over