The golden flower is essential. In a lot of stages you can earn a lot of coins by doing loops. Going down a pipe will make all enemies respawn, so you can rack up coins until time starts to run out.
Level 2-4 seems to be the best place to collect coins, because you don't have to go through pipes all the time. After the checkpoint you can go down a pipe where Goombas keep coming out of pipes. Let the leftmost walk to the left as much as possible and then kill them one after another with a series of jumps without touching the ground (golden flower required, obviously). Repeat until time starts to run out. It's not hard to get 8,000+ coins everytime you play the level.
Level 2-4 is the level with my best high score (8,800 something). I collected over 250,000 coins by now. I have only played coin rush twice so far, because it's such a pointless mode the way it's set up. I think my record in this mode is only around 2,000 coins. I find it more interesting to increase my scores for each level in the regular mode, although even that is quite tiresome. Despite all the coins in this game and numerous tricks to get a few hundred coins within seconds, a total of 1m is still insane.
Well, I disagree about Coin Rush Mode. It has added a lot of replay value for me at least. I'm already 150,177 coins, even though I only finished World 1 & 2 mostly due to coin rush, but also since I like to play slowly and explore every single corner in each level (I've only had the game for like 4 days now). My record in coin rush is 10,116 coins.
However, My scores per level are not that high. My highest is a measly 1097 coins in level 1-A :(