Logit-Contribution Scoring Identifies Non-Literal Retrieval Heads

Researchers introduced Logit-Contribution Scoring (LOCOS), a method that identifies attention heads responsible for non-literal context synthesis in large language models. LOCOS outperforms existing methods on retrieval benchmarks. This research advances the understanding of long-context model behavior.

1sec.ai