If the items on the map aren’t needed for the current realm quest, they shouldn’t show up on the map. A little confusing until you get used to it.
And by map, I of course mean minimap… Whoops (Thanks Shibby)
You know what’s worse?
Those stuff you can’t even interact with on the mini map.
I’ve always found them a bit useful in placing where the buried treasures.