I've been wondering about this for a while..

The question is which one of those gives more profits to the big companies (Sony, M$ and Nintendo), if hardware sales(console, accesories) or royalties from third companies games ? Also, anyone knows what's the % that companies have to pay for publishing their games in consoles?

Royalties. Sony and Microsoft generally sell their consoles for a loss.

I can't answer your specific questions, but I can tell you that gaming companies make most of their money from software sales. It's why consoles are often sold at a loss.

And welcome to Vgchartz.

Which is a really boneheaded decision on their part

But there is always the exception which is Nintendo but that usually doesn't happen.

Historically hardware was usually sold at a profit, it's only since larger companies with more financial clout entered the market (Sony and Microsoft) that hardware selling at a loss became the norm.

In this generation, the average console manufacturer loses money on hardware, and DOESN'T sell enough software to make it profitable. 

Nintendo is the exception. 

I believe Sony gets 10$ from licencing fees per full price PS3 games from third party

This generation Microsoft and Sony sold their consoles at a loss which generated more losses then they could possibly ever recover on the hardware alone.

Nintendo sells their consoles with a small profit, but they also have extremely strong 1st party software. You don't want to include 1st party software in the equation, so I would say Nintendo possibly makes more on hardware and accessories than royalties. This is assuming they make $70-80 per console (with controllers before price cut) sold. That would be 8-10 3rd party games per system sold if we assume they make 8-10 in royalties. Due to the extreme popularity of 1st party software and an attach rate of only 8, you could venture to guess that hardware wins out, but only by a tiny bit. If 1st and 3rd party software were combined, software would easily win for Nintendo.

Nintendo never sells for a loss (hint hint MS and Sony, you should follow this amazing strategy a 5 year old could come up with). And they have the most powerful software in the business, 3rd party or otherwise, which also make boat loads of money.

Since Nintendo first party games are so successful (Wii Fit 22 million alone, thats about 2 billion, more than the highest grossing movie of all time). I'd say they make more money on software, but they do make loads on hardware too.