Tag

#multimodal-models

Every item tagged multimodal-models, newest first.

3 items

Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers

You can now train and fine-tune multimodal embedding and reranker models using Sentence Transformers, which support text, images, and other modalities. This is achieved through a simple API that abstracts away the complexity of working with different data types. The Sentence Transformers library has seen significant growth, with over 100,000 model downloads and 4,000+ GitHub stars.

Key takeaways

Sentence Transformers supports multimodal models with text, images, and other modalities.
Over 100,000 model downloads and 4,000+ GitHub stars for the library.
Simple API for training and fine-tuning multimodal models.

HHugging Face Blog#multimodal-models #sentence-transformers #fine-tuning

researchJul 23

TimeScope: How Long Can Your Video Large Multimodal Model Go?

The TimeScope benchmark evaluates video large multimodal models on long-range temporal understanding. It tests models like InternLM-XTuner and LLaMA-3 on their ability to process and comprehend long video sequences. You can use TimeScope to assess and compare the performance of different video LMMs. The benchmark provides a standardized way to measure progress in this area.

Key takeaways

TimeScope benchmark tests video LMMs on long-range temporal understanding.
Evaluates models like InternLM-XTuner and LLaMA-3 on long video sequences.
Provides a standardized way to measure progress in video LMMs.

HHugging Face Blog#video-lmm #benchmarks #multimodal-models

researchMar 5

Introducing ConTextual: How well can your Multimodal model jointly reason over text and image in text-rich scenes?

The Hugging Face blog post introduces ConTextual, a benchmark for evaluating multimodal models on jointly reasoning over text and images in text-rich scenes. The benchmark aims to assess how well models can understand and generate text and image content together. You can use ConTextual to compare the performance of different multimodal models. The benchmark provides a new way to evaluate and improve multimodal models.

Key takeaways

ConTextual is a new benchmark for multimodal models.
Evaluates joint reasoning over text and images in text-rich scenes.
Assesses understanding and generation of text and image content.

HHugging Face Blog#multimodal-models #benchmarks #evaluation