I think it being distinct at the time was a turn off and it was nowhere near the leap Ocarina of Time represented. It felt like a Zelda "Gaiden" type game at the time, not a "real" entry so many Zelda fans I knew back then foucsed on the infamous Spaceworld demo and the Gamecube.
As time has passed Majora's Mask stands out more and I think its darker story line is appreciated by older gamers. Couple this with upcoming remake and explains surge in popularity a bit.