I think when you're comparing the two it's tough to say, both are VERY popular games that a lot of people eagerly wait. I think that Brawl has a great chance of outselling Halo 3 not only because it's on a seemingly more popular console(okay I'm being generous..) but because Brawl can sell to casuals where Halo 3 only seems to sell to hardcore till it's price drop then spawns out to -some- casuals.
Throw in Brawls free online play(yeah there's friend codes but it's FREE) compared to paying(I know it's not much but for the price of live I could buy a new game.) it could also be a boost in Brawls sales(I'm gonna go ahead and assume nintendo will add online play)
Brawl has several advantages especially one, it fills a niche that no other game really fills. It's a fighter and they're fairly common but this ones different somehow and even people who may own only fighting games seem to buy the smash bros games. Where Halo 3 is on a system with several like minded FPS games, Gears of War, Lost Planet, Prey, Bioshock. Not everybody buys 30 games a year like gamers, some people only buy 3 or 4 a year and given the amount of high profile FPS games released it may tap into Halos numbers..
I think Halo 3 will have a bigger launch but I think Brawl will outsell it in it's lifetime by quite a large margin.