Hockey just isn't that accessible to play or spectate compared to other sports. It's extremely expensive and time consuming for kids to get into, and a lot of people that haven't grown up watching it and playing it have a hard time following the puck on TV because it's so small and moves around so fast.
Having said that, I absolutely love hockey. There's nothing that compares to watching the NHL playoffs!