Putting in higher detail doesn't just require more disc capacity, it also requires greater processing power. If you get too many polygons and high-res textures on screen, the system will slow down, simple as that.
The only thing dual layer discs really allow is more content. So for Metroid: Other M, it worked because of the great amount of FMVs they had. For Super Smash Brothers, which also used dual-layer discs, it had a massive amount of levels, character models, extra content, etc.
The reason why Nintendo doesn't usually take the Super Smash Brothers approach is because it sucks up a lot of their resources. Several development houses had to pull together to get Super Smash Brothers out the door. As a business, they would rather make smaller games that sell better (Wii Fit, Wii Sports Resort, etc.)
