It was inevitable and a part of me believes the publishers knew this (hence the attempt to get as many SKUs out as possible before the milk went dry), even if the developers, many of whom are musicians themselves, did not.
It's not that the games continually dropped in quality; if anything RB3 offers the most "realistic" simulated experience in a music game, but that may actually be part of the problem. The genre has simply evolved about as far as it can without turning into full blown "learn how to play covers of your favorite rock and pop songs with real but really inexpensive instruments" video game tutors.
As the games get more technical and specialized (catering to would be musicians), the niche shrinks considerably.
And while it's easy to blame oversaturation for the shrinking commercial viability for the genre, it may well be that the general public has simply lost interest, much like dance pad games and kareoke games.
In a way, as a part of the would be musician niche, music rhythm games eventually made me come to the conclusion that hours spent playing Rock Band would be better spent learning riffs, chords, scales and bass grooves, even if it takes longer to learn. But at least music games are still fun at parties because you don't have to be a musician to play.