Pit Worm Harbinger's Hindrance vs Darkness Fate

I have a team with Darkness’ Fate (+7% per nature gem on my team). When the enemy team has a Pit Worm Harbinger (Enemies lose 20% speed at the start of their turn), my creatures lose all of their speed at the start of their turn. Well, it gets reduced to 5, but that’s still a noticeable drop from over 1million, and definitely more than 20%.