It's really annoying, specially on winter mornings.
When two things, one cold and other hot, are put together they end up with the same temperature; however my house is warm yet the toilet is still cold.
Edit - Correct answer below:
High specific heat capacity (takes a lot of energy to warm up) OR high heat conductivity (saps heat from your body rapidly).
Your body's perception of heat is actually based on the rate of heat transfer so the second will register as cold even though the material may be at room temperature. This is why metals are cold to the touch.