research15h
Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models
Researchers introduce Act2Answer, a protocol for evaluating commonsense and world knowledge in Vision-Language-Action (VLA) models. The method adapts existing VLM knowledge benchmarks to assess VLA models' ability to answer questions through action. This helps distinguish between knowledge gaps and control generalization issues in VLA models. The approach provides a more nuanced understanding of VLA capabilities.
Key takeaways
- Act2Answer protocol evaluates VLA models' knowledge through action.
- Helps differentiate knowledge retention from control generalization.
- Adapted from existing VLM knowledge benchmarks.