Sega couldnt even compete when it was just them and nintendo... Handheld market has sony and nintendo, plus all the tablet, smart phones, mp3 players, etc. i dont think it would be very smart at all to enter the handheld market. It is probably riskier than making a home console, to be honest...
As much as a liked sega consoles, it just doesnt make sense for them to start making hardware again...