1sec.ai

Tag

#multilingual-llms

Every item tagged multilingual-llms, newest first.

2 items

research18h

IndicContextEval: A Benchmark for Evaluating Context Utilisation in Audio Large Language Models Across 8 Indic Languages

Researchers introduce IndicContextEval, a 56-hour benchmark evaluating how well audio large language models utilise contextual inputs across 8 Indian languages. The benchmark tests models' ability to incorporate domain descriptions and entity lists into speech recognition. This work aims to assess whether models truly leverage context or rely on pre-trained knowledge. You can use this benchmark to develop and evaluate models that better understand contextual cues in multilingual speech.

Key takeaways
  • IndicContextEval is a 56-hour multilingual benchmark.
  • Evaluates context utilisation in audio LLMs across 8 Indian languages.
  • Tests models' ability to incorporate domain descriptions and entity lists.
aarXiv#multilingual-llms#speech-recognition#benchmarks
researchOct 1

🇨🇿 BenCzechMark - Can your LLM Understand Czech?

The author evaluated several LLMs on their ability to understand and generate Czech text. The best models showed promising results, but even the top performers struggled with nuances of the Czech language. You can explore the detailed benchmark results on the Hugging Face blog. This work highlights the challenges of multilingual support in LLMs.

Key takeaways
  • Czech language understanding is challenging for LLMs.
  • Top models still struggle with nuances.
  • Benchmark results available on Hugging Face blog.
HHugging Face Blog#multilingual-llms#benchmarks#czech-language