Haven't unlocked everyone but there has been a couple where I had to re-hallenge them. Granted, this was while I played against friends with characters I wasn't always used to.
Wouldn't say they're HARD so much as it can be HARDER than in past games (excluding World of Light).
The game's great btw.