Holo1: New family of GUI automation VLMs powering GUI agent Surfer-H
•
70
AI & ML interests
None defined yet.
Recent Activity
Articles
sergiopaniego
posted
an
update
about 15 hours ago
Post
118
OpenAI's open models are out! 💃
Try: https://www.gpt-oss.com/
Learn: https://huggingface.co/blog/welcome-openai-gpt-oss
Try: https://www.gpt-oss.com/
Learn: https://huggingface.co/blog/welcome-openai-gpt-oss
Post
292
we're all sleeping on this OCR model
rednote-hilab/dots.ocr 🔥
dots.ocr is a new 3B model with sota performance, support for 100 languages & allowing commercial use! 🤯
single e2e model to extract image, convert tables, formula, and more into markdown 📝
try it MohamedRashad/Dots-OCR
dots.ocr is a new 3B model with sota performance, support for 100 languages & allowing commercial use! 🤯
single e2e model to extract image, convert tables, formula, and more into markdown 📝
try it MohamedRashad/Dots-OCR
sergiopaniego
posted
an
update
1 day ago
Post
2533
Want to learn how to align a Vision Language Model (VLM) for reasoning using GRPO and TRL? 🌋
🧑🍳 We've got you covered!!
NEW multimodal post training recipe to align a VLM using TRL in @HuggingFace 's Cookbook.
Go to the recipe 👉https://huggingface.co/learn/cookbook/fine_tuning_vlm_grpo_trl
Powered by the latest TRL v0.20 release, this recipe shows how to teach Qwen2.5-VL-3B-Instruct to reason over images 🌋
🧑🍳 We've got you covered!!
NEW multimodal post training recipe to align a VLM using TRL in @HuggingFace 's Cookbook.
Go to the recipe 👉https://huggingface.co/learn/cookbook/fine_tuning_vlm_grpo_trl
Powered by the latest TRL v0.20 release, this recipe shows how to teach Qwen2.5-VL-3B-Instruct to reason over images 🌋
Post
424
massive releases and tons of Flux 1. Krea LoRas past week!
here's some of the picks, find more models in collection 🫡 merve/releases-august-2-6890c14248203522b7d0267f
LLMs 💬
> Tencent dropped tencent/Hunyuan-7B-Instruct
> Qwen released Qwen/Qwen3-Coder-30B-A3B-Instruct, 30B MoE with 3B params for coding (OS)
vision/multimodal
> RedNote released rednote-hilab/dots.ocr - 3B OCR model (OS)
> Cohere released CohereLabs/command-a-vision-07-2025 - 112B (dense!) VLM for 6 languages
> StepFun-AI shipped stepfun-ai/step3 - 321B MoE VLM (OS)
> Skywork shipped Skywork/Skywork-UniPic-1.5B - new any-to-any model (image+text → image+text) (OS)
here's some of the picks, find more models in collection 🫡 merve/releases-august-2-6890c14248203522b7d0267f
LLMs 💬
> Tencent dropped tencent/Hunyuan-7B-Instruct
> Qwen released Qwen/Qwen3-Coder-30B-A3B-Instruct, 30B MoE with 3B params for coding (OS)
vision/multimodal
> RedNote released rednote-hilab/dots.ocr - 3B OCR model (OS)
> Cohere released CohereLabs/command-a-vision-07-2025 - 112B (dense!) VLM for 6 languages
> StepFun-AI shipped stepfun-ai/step3 - 321B MoE VLM (OS)
> Skywork shipped Skywork/Skywork-UniPic-1.5B - new any-to-any model (image+text → image+text) (OS)
sergiopaniego
posted
an
update
2 days ago
Post
4167
Just included example scripts for aligning models using GSPO (including VLM example) 🙆♂️🙆♂️
GSPO is the latest RL alignment algo by @Alibaba_Qwen and it's already supported in the latest TRL v0.20 release.
Super-easy-to-get-started example scripts below, GO run them!👩💻👩💻
🧑🎨 Script: https://github.com/huggingface/trl/blob/main/examples/scripts/gspo.py
🦄 VLM script: https://github.com/huggingface/trl/blob/main/examples/scripts/gspo_vlm.py
🧩 More TRL examples: https://huggingface.co/docs/trl/main/en/example_overview
🧙♂️ GSPO paper: Group Sequence Policy Optimization (2507.18071)
GSPO is the latest RL alignment algo by @Alibaba_Qwen and it's already supported in the latest TRL v0.20 release.
Super-easy-to-get-started example scripts below, GO run them!👩💻👩💻
🧑🎨 Script: https://github.com/huggingface/trl/blob/main/examples/scripts/gspo.py
🦄 VLM script: https://github.com/huggingface/trl/blob/main/examples/scripts/gspo_vlm.py
🧩 More TRL examples: https://huggingface.co/docs/trl/main/en/example_overview
🧙♂️ GSPO paper: Group Sequence Policy Optimization (2507.18071)
Post
2059
Cohere just dropped
CohereLabs/command-a-vision-07-2025, a 112B (dense!) vision LM
> based on SigLIP2 & Command-A
> built for enterprise use cases 🔥
> use with Inference Providers or transformers 🤗
read their blog https://huggingface.co/blog/CohereLabs/introducing-command-a-vision-07-2025
> based on SigLIP2 & Command-A
> built for enterprise use cases 🔥
> use with Inference Providers or transformers 🤗
read their blog https://huggingface.co/blog/CohereLabs/introducing-command-a-vision-07-2025
- 2 replies
sergiopaniego
posted
an
update
7 days ago
Post
287
Did you miss this? 👓
🧙♂️vLLM + transformers integration just got upgraded with direct VLM support.
Select a VLM +
🧙♂️vLLM + transformers integration just got upgraded with direct VLM support.
Select a VLM +
model_impl=transformers and play via vLLM!
Post
3499
past week in open AI was insane 🔥 here's some of picks, find more here
merve/releases-july-25-688768ca47fe3693407e02d1
💬 LLMs & VLMs
> Qwen/Qwen3-235B-A22B-Thinking-2507 had a new update (OS)
> Qwen/Qwen3-Coder-480B-A35B-Instruct is out with 480B total 35B active params 🤯 (OS)
> AllenAI dropped an update to allenai/olmOCR-7B-0725 📝
> InternLM released internlm/Intern-S1 - 235B Qwen3 MoE + 6B InternViT encoder (OS)
> OmniSVG/OmniSVG is a new SVG generation VLM (OS)
🖼️ image/video/3D generation
> WanAI released Wan2.2 series - both T2V and I2V 14B models for high-quality video generation (OS) multimodalart/wan-22-688767e313337b434ed55112
> Tencent dropped tencent/HunyuanWorld-1 - image-to-3D scene generation model
💬 LLMs & VLMs
> Qwen/Qwen3-235B-A22B-Thinking-2507 had a new update (OS)
> Qwen/Qwen3-Coder-480B-A35B-Instruct is out with 480B total 35B active params 🤯 (OS)
> AllenAI dropped an update to allenai/olmOCR-7B-0725 📝
> InternLM released internlm/Intern-S1 - 235B Qwen3 MoE + 6B InternViT encoder (OS)
> OmniSVG/OmniSVG is a new SVG generation VLM (OS)
🖼️ image/video/3D generation
> WanAI released Wan2.2 series - both T2V and I2V 14B models for high-quality video generation (OS) multimodalart/wan-22-688767e313337b434ed55112
> Tencent dropped tencent/HunyuanWorld-1 - image-to-3D scene generation model
- 1 reply
sergiopaniego
posted
an
update
8 days ago
Post
2550
We just released TRL v0.20 with major multimodal upgrades!
👁️ VLM support for GRPO (highly requested by the community!)
🎞️ New GSPO trainer (from @Qwen , released last week, VLM-ready)
🐙 New MPO trainer (multimodal by design, as in the paper)
📝 Full release notes here: https://github.com/huggingface/trl/releases/tag/v0.20.0
👁️ VLM support for GRPO (highly requested by the community!)
🎞️ New GSPO trainer (from @Qwen , released last week, VLM-ready)
🐙 New MPO trainer (multimodal by design, as in the paper)
📝 Full release notes here: https://github.com/huggingface/trl/releases/tag/v0.20.0
Post
4303
🤯 241B VLM with apache-2.0 license
internlm/Intern-S1
internlm released Intern-S1: multimodal reasoning model based on 235B MoE Qwen3 and 6B InternViT 😍
benchmarks look great (👑 best model ✅ best open model)
internlm released Intern-S1: multimodal reasoning model based on 235B MoE Qwen3 and 6B InternViT 😍
benchmarks look great (👑 best model ✅ best open model)
sergiopaniego
posted
an
update
14 days ago
Post
1162
Yet Another New Multimodal Fine-Tuning Recipe 🥧
🧑🍳 In this @HuggingFace Face Cookbook notebook, we demonstrate how to align a multimodal model (VLM) using Mixed Preference Optimization (MPO) using trl.
💡 This recipe is powered by the new MPO support in trl, enabled through a recent upgrade to the DPO trainer!
We align the multimodal model using multiple optimization objectives (losses), guided by a preference dataset (chosen vs. rejected multimodal pairs).
Check it out! ➡️ https://huggingface.co/learn/cookbook/fine_tuning_vlm_mpo
🧑🍳 In this @HuggingFace Face Cookbook notebook, we demonstrate how to align a multimodal model (VLM) using Mixed Preference Optimization (MPO) using trl.
💡 This recipe is powered by the new MPO support in trl, enabled through a recent upgrade to the DPO trainer!
We align the multimodal model using multiple optimization objectives (losses), guided by a preference dataset (chosen vs. rejected multimodal pairs).
Check it out! ➡️ https://huggingface.co/learn/cookbook/fine_tuning_vlm_mpo
-
- 2 replies
Post
777
so many open LLMs and image LoRAs dropped past week, here's some picks for you 🫡
merve/releases-july-18-687e3fbd2ab9b39c51f9238b
LLMs
> ByteDance released a bunch of translation models called Seed-X-RM (7B) ByteDance-Seed/Seed-X-RM-7B
> NVIDIA released reasoning models of which 32B surpassing the giant Qwen3-235B with cc-by-4.0 license 👏 nvidia/openreasoning-nemotron-687730dae0170059860f1f01
> LG released a new EXAONE model (32B) LGAI-EXAONE/EXAONE-4.0-32B
VLMs/any-to-any
> vidore/colqwen-omni-v0.1 is a new any-to-any retriever (MIT)
> HiDream-ai/HiDream-E1-1 is image+text in image+text out model (MIT)
LoRAs
> There's a bunch of LoRAs based on Flux Kontext, gotta check out the collection 🤠
LLMs
> ByteDance released a bunch of translation models called Seed-X-RM (7B) ByteDance-Seed/Seed-X-RM-7B
> NVIDIA released reasoning models of which 32B surpassing the giant Qwen3-235B with cc-by-4.0 license 👏 nvidia/openreasoning-nemotron-687730dae0170059860f1f01
> LG released a new EXAONE model (32B) LGAI-EXAONE/EXAONE-4.0-32B
VLMs/any-to-any
> vidore/colqwen-omni-v0.1 is a new any-to-any retriever (MIT)
> HiDream-ai/HiDream-E1-1 is image+text in image+text out model (MIT)
LoRAs
> There's a bunch of LoRAs based on Flux Kontext, gotta check out the collection 🤠
Post
2775
Now it's possible to do RAG with any-to-any models 🔥
Learn how to search in a video dataset and generate using Tevatron/OmniEmbed-v0.1-multivent an all modality retriever, and Qwen/Qwen2.5-Omni-7B, any-to-any model in this notebook 🤝 merve/smol-vision
Learn how to search in a video dataset and generate using Tevatron/OmniEmbed-v0.1-multivent an all modality retriever, and Qwen/Qwen2.5-Omni-7B, any-to-any model in this notebook 🤝 merve/smol-vision
sergiopaniego
posted
an
update
19 days ago
Post
1646
🧑🍳 New Multimodal Fine-Tuning Recipe 🧑🍳
⚡️ In this new @huggingface Cookbook recipe, I walk you though the process of fine tuning a Visual Language Model (VLM) for Object Detection with Visual Grounding, using TRL.
🔍 Object detection typically involves detecting categories in images (e.g., vase).
By combining it with visual grounding, we add contextual understanding so instead of detecting just "vase", we can detect "middle vase" in an image.
VLMs are super powerful!
In this case, I use PaliGemma 2 which already supports object detection and extend it to also add visual grounding.
🤗 Check it out here: https://huggingface.co/learn/cookbook/fine_tuning_vlm_object_detection_grounding
⚡️ In this new @huggingface Cookbook recipe, I walk you though the process of fine tuning a Visual Language Model (VLM) for Object Detection with Visual Grounding, using TRL.
🔍 Object detection typically involves detecting categories in images (e.g., vase).
By combining it with visual grounding, we add contextual understanding so instead of detecting just "vase", we can detect "middle vase" in an image.
VLMs are super powerful!
In this case, I use PaliGemma 2 which already supports object detection and extend it to also add visual grounding.
🤗 Check it out here: https://huggingface.co/learn/cookbook/fine_tuning_vlm_object_detection_grounding
sergiopaniego
posted
an
update
20 days ago
Post
1609
Multiple NEW notebooks and scripts added to the Hugging Face Gemma recipes repo!
Thanks to the community 🫶, we're adding more and more recipes using Gemma 💎
Fine tuning for all modalities, function calling, RAG...
Repo: https://github.com/huggingface/huggingface-gemma-recipes
We're also open to new ideas from the community 🤗!
Thanks to the community 🫶, we're adding more and more recipes using Gemma 💎
Fine tuning for all modalities, function calling, RAG...
Repo: https://github.com/huggingface/huggingface-gemma-recipes
We're also open to new ideas from the community 🤗!
- 1 reply
Post
2050
all modality RAG 🔥
ColQwen-Omni is a new multimodal retrieval model that can retrieve anything (videos, audios, documents and more!)
use with transformers 🤗
read the blog https://huggingface.co/blog/manu/colqwen-omni-omnimodal-retrieval
model repository vidore/colqwen-omni-v0.1
ColQwen-Omni is a new multimodal retrieval model that can retrieve anything (videos, audios, documents and more!)
use with transformers 🤗
read the blog https://huggingface.co/blog/manu/colqwen-omni-omnimodal-retrieval
model repository vidore/colqwen-omni-v0.1
Post
2600
Fine-tune Gemma3n on videos with audios inside with Colab A100 🔥
Just dropped the notebook where you can learn how to fine-tune Gemma3n on images+audio+text at the same time!
keep in mind, it's made for educational purposes 🫡 we do LoRA, audio resampling & video downsampling to be able to train <40GB VRAM
stretch modalities and unfreeze layers as you wish! 🙏🏻 merve/smol-vision
Just dropped the notebook where you can learn how to fine-tune Gemma3n on images+audio+text at the same time!
keep in mind, it's made for educational purposes 🫡 we do LoRA, audio resampling & video downsampling to be able to train <40GB VRAM
stretch modalities and unfreeze layers as you wish! 🙏🏻 merve/smol-vision
- 1 reply
sergiopaniego
posted
an
update
23 days ago
Post
387
Loved this paper! ♥️
Authors benchmark multimodal models on vision tasks (detection, segmentation...) using clever prompting tricks.
📄 Results: VLMs are solid generalists but still lag behind SOTA task-specific models — especially on geometric tasks vs. semantic ones.
paper: How Well Does GPT-4o Understand Vision? Evaluating Multimodal Foundation Models on Standard Computer Vision Tasks (2507.01955)
Authors benchmark multimodal models on vision tasks (detection, segmentation...) using clever prompting tricks.
📄 Results: VLMs are solid generalists but still lag behind SOTA task-specific models — especially on geometric tasks vs. semantic ones.
paper: How Well Does GPT-4o Understand Vision? Evaluating Multimodal Foundation Models on Standard Computer Vision Tasks (2507.01955)
sergiopaniego
posted
an
update
23 days ago
Post
258
You can already play with two of the latest most impressive models on HF via
@novita-ai
as Inference Provider 🚨
🌌 Kimi K2: 1T params model, MoE beast for coding, reasoning and agentic tasks
🔮 GLM-4.1V-9B-Thinking: VLM + deep reasoning model
Kimi K2: moonshotai/Kimi-K2-Instruct
GLM-4.1V-9B-Thinking: https://huggingface.co/THUDM/GLM-4.1V-9B-Thinking
🌌 Kimi K2: 1T params model, MoE beast for coding, reasoning and agentic tasks
🔮 GLM-4.1V-9B-Thinking: VLM + deep reasoning model
Kimi K2: moonshotai/Kimi-K2-Instruct
GLM-4.1V-9B-Thinking: https://huggingface.co/THUDM/GLM-4.1V-9B-Thinking
Post
2430
past week had huuuge releases 💗
here's our picks 🔥 find more models, datasets, demos here merve/releases-july-11-68750452c358c98b0fa663f7
> moonshotai/Kimi-K2-Instruct is the new sota LLM with 1T total 32B active parameters 🤯
> HuggingFaceTB/SmolLM3-3B is the new best LM for it's size, offers thinking mode 💭 as well as the dataset HuggingFaceTB/smoltalk2
> Alibaba-NLP/WebSailor-3B is the new agentic LLM for complex browsing
> Google DeepMind released medical vision LMs with an agentic doctor-patient app google/medgemma-release-680aade845f90bec6a3f60c4
> fal released a LoRA to improve details on face images fal/Realism-Detailer-Kontext-Dev-LoRA
here's our picks 🔥 find more models, datasets, demos here merve/releases-july-11-68750452c358c98b0fa663f7
> moonshotai/Kimi-K2-Instruct is the new sota LLM with 1T total 32B active parameters 🤯
> HuggingFaceTB/SmolLM3-3B is the new best LM for it's size, offers thinking mode 💭 as well as the dataset HuggingFaceTB/smoltalk2
> Alibaba-NLP/WebSailor-3B is the new agentic LLM for complex browsing
> Google DeepMind released medical vision LMs with an agentic doctor-patient app google/medgemma-release-680aade845f90bec6a3f60c4
> fal released a LoRA to improve details on face images fal/Realism-Detailer-Kontext-Dev-LoRA