The real reason it costs so much is greed.
Controllers have extremely high profit margins. Nintendo is just maximizing profit based on what they think they can get away with.
For example, a PS4 controller cost $18 to build in 2013.
Source:
http://allthingsd.com/20131119/teardown-shows-sonys-playstation-4-costs-381-to-build/