If Sony does something like that and it has PS2 level graphics and is $200 it would sell really well I think.
You're describing the PSP! Only more expensive, and with a hard drive.
Re: the PSP's power, the tech writeups I've seen say it's slightly less powerful than the PS2, but because it doesn't have to display the same resolution, they're pretty much balanced. With the recent uncapping of its processor, the PSP might now have the edge. It's really an impressive little machine.
So a successor in 2-3 years with PS2-level graphics would obviously be a bit pointless, if not a step backwards. If you ask me, Sony would be making a mistake to put out a PSP2 at all so early. By all indications, handheld gamers don't look to graphics as a major selling point, and as a result, (successful) handhelds tend to have longer lifespans than home consoles. What's far more important is price. People make a lot of ridiculous statements like "the PSP is a failure" when in fact it's just now beginning to get a respectable install base, and Sony could ride it for another 5 years. They need to get the price down to $99 and really support it, rather than dump it and bring in a poorly conceived successor.
So whenever it does come out, what will the PSP2 be like? It has to look like a worthwhile upgrade, so graphics somewhere approaching PS3/360 levels. Or, if they've been paying attention, they'll come up with a gimmick that can compete more directly with the DS, in which case graphical power only 5 or 10 times the PS2 might be enough. It won't have a hard drive, because hard drives are on the brink of obsolescence already. Since we're probably looking at 2010 or later, it'll instead have high-density flash memory, maybe 20-50+ gigs. And motion controls sound like a really bad idea in a handheld (how do you see what you're doing?) but it's not out of the question. Maybe it could even work as that gimmick I was talking about -- what about games that rotate the image on the screen to stay level as you rotate the handheld?
As for the storage medium? I'd say
Mini BluRay is a pretty safe bet. These ones support 16.5 gigs per layer. 8 cm is a tiny bit large though, so for a handheld Sony might come up with a 6 cm variety.