It flopped because this is the first Rock Band where Harmonix was funding their own development - after their split with EA, HMX basically went Indie and on Rock Band 4, they couldn't afford good advertising. The set list is all filler songs that couldn't have cost them that much to license. There's no online play for the game yet either which is huge, only leaderboards. Rock Band Network isn't compatible yet either.
And finally, the instruments this time around launched with game breaking bugs on the hardware and software side. This is why harmonix is about to release updated hardware from a new vendor company for the game - madcatz instruments are just bad and have severe latency issues. My drum set broke probably my 10th time using them. That's never happened in the passed.
With Rock Band 4, the game is set to serve as a 'platform' that's not meant to have a sequel this generation. Harmonix has, true to their word been updating the game over time and we are slowly seeing the return of features, so I'm suspecting this game will have legs.
I also forgot to mention that there's no practice mode yet, a big deal imo
All in all, they just didn't have the proper resources to make a complete game out of the box. Everything went bad with them while under EA, and they probably just didn't want to deal with big publishers like that anymore. Harmonix is actually a really good company and they are very in touch with their base. We've had talks with them on the forums over at Scorehero years ago and they had official status titles. It sucks this is happening, but it is what it is