I would blame this on third parties splitting up between three home consoles. Whether it was developers being unsure of whether to support ps3 or Wii or being paid to support 360, franchises that traditionally found strong success on a single platform were suddenly splitting their fanbases up between multiple platforms. Looking at JRPGs in particular, the genre has been spread pretty evenly across all three platforms, with some individual franchises actually being stretched across all three platforms (Tales, for example).
This lead to confusion among consumers, as well as a probable want to not buy more than their usual 1-2 consoles, so they mostly stuck with handhelds.
Of course, I'm sure the increase in support for handhelds by Square Enix alongside the Monster Hunter phenomenon have also played a part here. I'd say that and developers being too quick to jump into bed with Microsoft were probably the most pivotal factors overall. I mean, there was a good bit of time there where almost all the major games were either on handhelds or on Xbox.
