Preprint finds periodic subject changes raise judged surprise in base language models
A new arXiv preprint reports that injecting a new subject every few hundred tokens can increase language-model outputs’ judged surprise and connection. The effect did not produce integrated documents or improve the best result in an online bin-packing test, exposing limits in how long generations are evaluated.