Whether it's on Sony, Microsoft, or Nintendo consoles, the games are merely decent. Hype is the entire reason the series is widely talked about now.
To answer the question though, I don't think their return to Sony has any factor whether they sell or not. There might be other Monster Hunter style games, but the games are in the same genre and nothing more. The games really aren't all that similar in gameplay, so if you liked Monster Hunter, there's no guarantee you'd like Ragnarok Odyssey or vice versa. A fan of one series could just as well like both and continue to purchase both.
With as many 3DS consoles sold, I'm pretty surprised though that Monster Hunter 3G didn't sell more. Most of us knew that it wasn't going to sell in the west. It caused a very slight sales spike but immediately dropped back down. But in Japan, even after being out for over a year, it's still yet to hit two million sales.
No one knows if a Vita version would sell well or not, though. There's no evidence to prove it would sell poorly other than the current Vita sales. No one really knows why 3G didn't sell as well as previous games, though some might argue that the 3DS sales are the reason Everything we say on either subject would be nothing more than speculation.