I only lost a challenge fight once. I did have some epically close calls, but other than that, I managed to get everyone. I got most of them via World of Light, which you have as many tries as you like to get them. So yeah, not too crazy.
For novice or new players, I can see them having trouble getting them all.