RNNs vs Transformers vs SSMs: where should AI memory live for continual learning?
Researchers debate optimal memory placement for continual learning in AI models. The key comparison is not between RNNs, transformers, and SSMs architectures, but whether memory resides in a small recurrent state, a growing KV cache, or within the model network. RNNs store memory in a recurrent hidden state, which creates a bottleneck as state size grows linearly while parameter count grows quadratically. This limits RNN scalability.
Key takeaways
- RNNs store memory in a recurrent hidden state.
- RNNs have O(N) state but O(N^2) parameters.
- Memory placement is key for continual learning.
RNNs vs Transformers vs SSMs: where should AI memory live for continual learning?
Researchers debate optimal memory placement for continual learning in AI models. The key comparison is not between RNNs, transformers, and SSMs architectures, but whether memory resides in a small recurrent state, a growing KV cache, or within the model network. RNNs store memory in a recurrent hidden state, which creates a bottleneck as state size grows linearly while parameter count grows quadratically. This limits RNN scalability.
Key takeaways
- RNNs store memory in a recurrent hidden state.
- RNNs have O(N) state but O(N^2) parameters.
- Memory placement is key for continual learning.