The issue is with the WoW client itself. The event system is bugged and fires off events way more than it should, and each loading screen you experience has the potential to make it worse. Since ElvUI reacts to the events that are triggered, having them trigger more than necessary will lead to higher use of resources and eventually to fps drops.
It is an issue that has to be fixed in WoW itself and they are working on fixes for PTR 8.1.5. Hopefully it will all be fixed in the next patch when it goes live.
Until then, doing a full relog (log out, log in) will fix the broken events. If you do that right before a raid then you should be fine for a while. If you start noticing issues then do another full relog next time there is a small window of opportunity for it.
ElvUI: Request Feature
| Report Bug
My replies might be short sometimes due to time constraints. Nothing personal.