That 16MB was still available to devs, just not for high access speed tasks. And the other 24MB was lightning fast compared to PS2's memory.
As for the difference being exaggerated, that may have held true for many multiplatform titles, but stuff like Rogue Squadron 2 and 3 was out of PS2's league.
Actually that 16MB was reserved for buffering audio and data from the mini DVD.
It was certainly lightning fast in terms of access times yet it did have a somewhat lower bandwidth compared to the PS2.
I'm not sure if graphics like Rogue Squadron 2 and 3's graphics are out of the PS2's league. I think it's met a match with MGS 3 ...
Although you may encounter some more rough textures on the PS2, however being able to spit out more triangles certainly helps obscures the lower resolution textures ...