It should be removed automatically. It reads like you have another action bar addon on (bartender maybe) that is causing issues. Disable bartender and try.
You shouldn’t have two addons enabled that control the same element else issues like this can happen.
If it isn’t bartender, disable all addons except ElvUI. The quickest way to disable all other addons is to type in chat in game