Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
Hoglet-33 
posted an update 3 days ago
Post
2948
We are announcing the first generation of the Pebble model family!

These are the models we are releasing:

- Pebble 10M
- Pebble 25M
- Pebble 50M

Each model will use a Mamba-Transformer 3:1 hybrid architecture and will be pretrained on 25 billion tokens before IFT and SFT.

Depending on development time and resources, we may also release:

- Pebble 5M
- Pebble 75M
- Pebble 1M (possibly)

We hope you're excited and enjoy the models!

Follow for more:
@Hoglet-33
basically-ai
Datdanboi25

Awesome! We would love to have them on our leaderboard, would be the first hybrid style model on there!
yano2mch
•
edited 3 days ago

Do these micro models have a specific use case? I mean they probably aren't going to be making code or doing general conversation... probably...

·
mistefan

Lets start with the autocorrect running on most smartphones.... Or the AI cameras... Or the text content scanning mandated by the EU/US/India/Brazil/etc. And yes, some can even make code if trained to focus on one language... You must be very young, because the GPT-2 that people were hyping as Skynet was a 120-1500M model.
wayneworkman2012

Where can I find them?

·
Hoglet-33

They aren't released yet, but hopefully I can release them by the end of September though.
In this post