Game gets slow on Xbox One

The Game gets slower the more Enemys I fight in one Realm. I have a habit of not completing the Realm Quest to extract some extra Enemys for Knowledge gain and Singular hunting. But at some point the Game gets unbearingly slow that I fear it might crash, so I just complete the Quest and start a new Realm and everythings turns out fine. It might be my own fault, because I play against the Games intended design but maybe its not my fault.