Shmups are a huge, potentially audience-expanding genre.
2D Shmups, anyways.
Nintendo should take a 2-game 2D/3D approach to Star Fox like they do with Metroid and Mario. Make a 3D Star Fox game that goes back to the SNES/N64 roots. And then make the first ever 2D Star Fox.
3D isn't revolutionary anymore, and there's never been a 3D flight-based Shmup with the popularity of an Asteroids or Space Invaders or Galaga. Instead, FPS games are the Shmup of choice in 3D.
Star Fox is a good brand with cool characters that would work with different gameplay types.
And imagine the different gameplay styles and multi-player possibilities. You could have upward-scrolling and side-scrolling levels, flying through space, asteroid belts, or above a planet's surface, "ambush" levels with Asteroids-style controls, even on-foot shmup levels, and all of it with 4-player Fox+Falco+Peppy+Slippy. And then you could have split-screen modes, where two players are in space and two are on-foot, and they have to complete objectives for the other pair to continue. 4v4 Fox vs. Wolf online modes. Various ways of doing high-score or killcount competitions or leaderboards, or dogfights with SpaceWar/Asteroids style controls.
Why are there no games like this? Instead, all the 2D Shmups left are niche "bullet hells" like the OP said. Star Fox is a great existing brand to do this with instead of inventing a new brand.
"[Our former customers] are unable to find software which they WANT to play."
"The way to solve this problem lies in how to communicate what kind of games [they CAN play]."
Satoru Iwata, Nintendo President. Only slightly paraphrased.