The State-Prediction Separation Hypothesis

The State-Prediction Separation Hypothesis proposes that separating state prediction from token prediction in Transformers can improve language modeling performance and efficiency. Experiments show that this approach offers better data and compute efficiencies, improving validation loss and outperforming standard Transformers by 2-3 percentage points on average on downstream tasks.

1sec.ai