Saying that you're paying extra because the console is bigger makes absolutely no sense. Yes, a couple of materials are used more in a bigger console, but that's cheap stuff like copper wire and plastic (as opposed to the expensive hard drives and chips. A bigger console is cheaper because it was easier to design and manufacture.
I don't think that the hardware itself warranted the price difference, at least not completely, but that's not to say Microsoft is just taking the money and pocketing it. They also are doing a major server upgrade, so the X1 hardware itself isn't the only thing on Microsoft's budget.
$500 for the X1 is, all things considered, a fair price, but so is the PS4's $400. If you're after a console, you'll probably lean toward the PS4 purely because it's cheaper. If you're leaning toward multimedia, you should probably get a Wii U, which does TV integration, web browsing, and youtube all better than the X1 at $200 less (Skype is the major loss.)