Well, I don't see GPU replacing CPU anytime soon.
The raw computing power of GPUs is huge but that's very specialized. You can't execute complex logic on these. They have no branch predictors. Branch instructions are very expensive and they have wide registers. You still need a powerful CPU to compute complex logic like inner engine loops and gameplay tasks.
What you can do with GPUs is mostly rendering, skinning, physics, fluid simulation... That kind of mathematically intensive tasks. But that's it. They are even more specialized in stream processing than the SPUs actually. At least, you can do general computing with SPUs... It won't be very fast but that's acceptable... Line checks is probably the most obvious example of things you'll have a hard time doing on the GPU that you can quite easily do on the SPUs.
