research18h
Seeing Before Reasoning: Decoupling Perception and Reasoning for Shortcut-Resilient Multimodal On-Policy Self-Distillation
Researchers propose ViGOS, a visually grounded on-policy self-distillation framework for multimodal large language models (MLLMs). The method aims to prevent shortcuts in training where the model relies too heavily on text targets rather than image inputs. ViGOS guides the student model to use both visual and textual information effectively. This approach can improve the robustness of MLLMs in tasks that require multimodal understanding.
Key takeaways
- ViGOS framework proposed for visually grounded on-policy self-distillation in MLLMs.
- Method aims to prevent shortcuts relying on text targets over image inputs.
- Improves robustness in multimodal tasks requiring both visual and textual understanding.