Yes, but compared to XB360 Wii added little more RAM and while the driver itself for motion is surely small, some memory will be consumed by the game code that actually uses it giving a meaning in the game to the motion detected... In the end Wii has little more RAM but it uses it very well for things that people notice more than graphics, once these have reached an acceptable level and even exceeded SD TV's capabilities, increasing graphics performance would require a lot more RAM, not for the resolution itself, but particularly if increasing effects, textures size, antialiasing level, etc (as those 32MB of the first GeForce, for example, were enough for 1024x768@32bit, that's more than SD, but with very basic antialiasing and relatively small textures).
My point, to cut it short, is that having little more RAM it's perfectly natural it doesn't exceed XBox graphics capabilities in a dramatic way, as graphics are quite heavy on RAM, but Nintendo never meant to do it.
One consideration is that the Flipper handled texture compression on the GPU while the XBox GPU required any compressed textures to be decompressed before being passed into the GPU; on top of this the Flipper had 3MB of memory built into it, of which 1MB was devoted to frame-buffers, and (if compressed to the maximum allowed by the GPU) the 2MB of remaining memory was similar to a high speed 18MB texture cache. While this still may not sound like a lot compared to what the HD consoles have, this works out to 16 32 bit texels for every pixel at 480p.