Part of it is appealing to the core base while being marketable.
Metroid never got a bigger push in America besides during the Prime 1/Fusion release window. I do think that a side scroller like many purist wish can't really sell so the series best hope is either a faithful 3D third person shooter or a FPS like the Prime series.