The Curse and Blessing of Mean Bias in FP4-Quantized LLM Training
Paper β’ 2603.10444 β’ Published β’ 12
AI & ML interests
None defined yet.
Recent Activity
View all activity
xianbaoΒ
submitted a
paper to Daily Papers about 1 month ago
julien-cΒ
submitted a
paper to Daily Papers 3 months ago
csabakecskemetiΒ
posted an update 4 months ago
Post
3327
Just sharing a result of a homelab infrastructure experiment:
I've managed to setup a distributed inference infra at home using a DGX Spark (128GB unified gddr6) and a linux workstation with an RTX 6000 Pro (96GB gddr7) connected via 100Gbps RoCEv2. The model I've used (https://lnkd.in/gx6J7YuB) is about 140GB so could not fit either of the GPU. Full setup and tutorial soon on devquasar.com
Screen recording:
https://lnkd.in/gKM9H5GJ
I've managed to setup a distributed inference infra at home using a DGX Spark (128GB unified gddr6) and a linux workstation with an RTX 6000 Pro (96GB gddr7) connected via 100Gbps RoCEv2. The model I've used (https://lnkd.in/gx6J7YuB) is about 140GB so could not fit either of the GPU. Full setup and tutorial soon on devquasar.com
Screen recording:
https://lnkd.in/gKM9H5GJ
- 3 replies
JohannesΒ
authored a
paper 4 months ago
woojun-jungΒ
authored a
paper 4 months ago
csabakecskemetiΒ
posted an update 5 months ago
Post
1430
FYI: Mistral.Ministral-3 dequantizer FP8->BF16
https://github.com/csabakecskemeti/ministral-3_dequantizer_fp8-bf16
(The instruct model weights are in FP8)
https://github.com/csabakecskemeti/ministral-3_dequantizer_fp8-bf16
(The instruct model weights are in FP8)
csabakecskemetiΒ
posted an update 5 months ago
Post
2103
Looking for some help to test an INT8 Deepseek 3.2:
SGLang supports Channel wise INT8 quants on CPUs with AMX instructions (Xeon 5 and above AFAIK)
https://lmsys.org/blog/2025-07-14-intel-xeon-optimization/
Currently uploading an INT8 version of Deepseek 3.2 Speciale:
DevQuasar/deepseek-ai.DeepSeek-V3.2-Speciale-Channel-INT8
I cannot test this I'm on AMD
"AssertionError: W8A8Int8LinearMethod on CPU requires that CPU has AMX support"
(I assumed it can fall back to some non optimized kernel but seems not)
If anyone with the required resources (Intel Xeon 5/6 + ~768-1TB ram) can help to test this that would be awesome.
If you have hints how to make this work on AMD Threadripper 7000 Pro series please guide me.
Thanks all!
SGLang supports Channel wise INT8 quants on CPUs with AMX instructions (Xeon 5 and above AFAIK)
https://lmsys.org/blog/2025-07-14-intel-xeon-optimization/
Currently uploading an INT8 version of Deepseek 3.2 Speciale:
DevQuasar/deepseek-ai.DeepSeek-V3.2-Speciale-Channel-INT8
I cannot test this I'm on AMD
"AssertionError: W8A8Int8LinearMethod on CPU requires that CPU has AMX support"
(I assumed it can fall back to some non optimized kernel but seems not)
If anyone with the required resources (Intel Xeon 5/6 + ~768-1TB ram) can help to test this that would be awesome.
If you have hints how to make this work on AMD Threadripper 7000 Pro series please guide me.
Thanks all!
- 8 replies
csabakecskemetiΒ
posted an update 6 months ago
Post
315
Recently there are so much activity on token efficient formats, I've also build a package (inspired by toon).
Deep-TOON
My goal was to token efficiently handle json structures with complex embeddings.
So this is what I've built on the weekend. Feel free try:
https://pypi.org/project/deep-toon/0.1.0/
Deep-TOON
My goal was to token efficiently handle json structures with complex embeddings.
So this is what I've built on the weekend. Feel free try:
https://pypi.org/project/deep-toon/0.1.0/
xianbaoΒ
authored a
paper 6 months ago
csabakecskemetiΒ
posted an update 6 months ago
Post
2659
Christmas came early this year
- 3 replies
Post
22791
Okay this is insane... WebGPU-accelerated semantic video tracking, powered by DINOv3 and Transformers.js! π€―
Demo (+ source code): webml-community/DINOv3-video-tracking
This will revolutionize AI-powered video editors... which can now run 100% locally in your browser, no server inference required (costs $0)! π
How does it work? π€
1οΈβ£ Generate and cache image features for each frame
2οΈβ£ Create a list of embeddings for selected patch(es)
3οΈβ£ Compute cosine similarity between each patch and the selected patch(es)
4οΈβ£ Highlight those whose score is above some threshold
... et voilΓ ! π₯³
You can also make selections across frames to improve temporal consistency! This is super useful if the object changes its appearance slightly throughout the video.
Excited to see what the community builds with it!
Demo (+ source code): webml-community/DINOv3-video-tracking
This will revolutionize AI-powered video editors... which can now run 100% locally in your browser, no server inference required (costs $0)! π
How does it work? π€
1οΈβ£ Generate and cache image features for each frame
2οΈβ£ Create a list of embeddings for selected patch(es)
3οΈβ£ Compute cosine similarity between each patch and the selected patch(es)
4οΈβ£ Highlight those whose score is above some threshold
... et voilΓ ! π₯³
You can also make selections across frames to improve temporal consistency! This is super useful if the object changes its appearance slightly throughout the video.
Excited to see what the community builds with it!
- 4 replies
1024mΒ
authored 2
papers 9 months ago
wubinghengΒ
authored a
paper 9 months ago
Post
4946
The next generation of AI-powered websites is going to be WILD! π€―
In-browser tool calling & MCP is finally here, allowing LLMs to interact with websites programmatically.
To show what's possible, I built a demo using Liquid AI's new LFM2 model, powered by π€ Transformers.js: LiquidAI/LFM2-WebGPU
As always, the demo is open source (which you can find under the "Files" tab), so I'm excited to see how the community builds upon this! π
In-browser tool calling & MCP is finally here, allowing LLMs to interact with websites programmatically.
To show what's possible, I built a demo using Liquid AI's new LFM2 model, powered by π€ Transformers.js: LiquidAI/LFM2-WebGPU
As always, the demo is open source (which you can find under the "Files" tab), so I'm excited to see how the community builds upon this! π
- 2 replies
wubinghengΒ
authored a
paper 9 months ago
Post
3755
Introducing Voxtral WebGPU: State-of-the-art audio transcription directly in your browser! π€―
π£οΈ Transcribe videos, meeting notes, songs and more
π Runs on-device, meaning no data is sent to a server
π Multilingual (8 languages)
π€ Completely free (forever) & open source
That's right, we're running Mistral's new Voxtral-Mini-3B model 100% locally in-browser on WebGPU, powered by Transformers.js and ONNX Runtime Web! π₯
Try it out yourself! π
webml-community/Voxtral-WebGPU
π£οΈ Transcribe videos, meeting notes, songs and more
π Runs on-device, meaning no data is sent to a server
π Multilingual (8 languages)
π€ Completely free (forever) & open source
That's right, we're running Mistral's new Voxtral-Mini-3B model 100% locally in-browser on WebGPU, powered by Transformers.js and ONNX Runtime Web! π₯
Try it out yourself! π
webml-community/Voxtral-WebGPU
Skylion007Β
authored a
paper 10 months ago
csabakecskemetiΒ
posted an update 11 months ago
Post
3089
Has anyone ever backed up a model to a sequential tape drive, or I'm the world first? :D
Just played around with my retro PC that has got a tape driveβdid it just because I can.
Just played around with my retro PC that has got a tape driveβdid it just because I can.
- 5 replies
Post
7633
NEW: Real-time conversational AI models can now run 100% locally in your browser! π€―
π Privacy by design (no data leaves your device)
π° Completely free... forever
π¦ Zero installation required, just visit a website
β‘οΈ Blazingly-fast WebGPU-accelerated inference
Try it out: webml-community/conversational-webgpu
For those interested, here's how it works:
- Silero VAD for voice activity detection
- Whisper for speech recognition
- SmolLM2-1.7B for text generation
- Kokoro for text to speech
Powered by Transformers.js and ONNX Runtime Web! π€ I hope you like it!
π Privacy by design (no data leaves your device)
π° Completely free... forever
π¦ Zero installation required, just visit a website
β‘οΈ Blazingly-fast WebGPU-accelerated inference
Try it out: webml-community/conversational-webgpu
For those interested, here's how it works:
- Silero VAD for voice activity detection
- Whisper for speech recognition
- SmolLM2-1.7B for text generation
- Kokoro for text to speech
Powered by Transformers.js and ONNX Runtime Web! π€ I hope you like it!
- 5 replies