Specs don't matter as much on handhelds because people play handheld games in shorter bursts, it's a smaller screen (thus more cartoony games seem to work better -- even on iOS look at Angry Birds and Plants Vs. Zombies), and people already get their fill of higher end graphics on their home console.
The other reason is handheld gaming is more driven by the kids market, a lot of adults simply don't bother to play handheld games. Kids are more forgiving of lower end graphics.
It's like when you eat at McDonalds/Wendys/Burger King or whatever, you don't expect the same experience as when you eat a full service restaurant with a waiter and apetizers/main course/wine/alcohol/desert, etc. etc.
They're both different experiences for different situations.
