To be honest I don't see anything other than this becoming a reality in the next ~10 years or so and I think it would be great. When this technology is perfected (and let's not forget this is still in early development) would you really still rather scroll through menus using a controller rather than just your hands? You'll be able to access and navigate menus far more efficiently (even more so with speech recognition).
As for people claiming they don't want random movements to register I think you're underestimating the technology. It really wouldn't be difficult to avoid such problems. For example, to even open the menu then perhaps a specific gesture (such as a 360 hand wave ect) could be required so that random movements do not trigger any actions. Also the problem of 'clicking' or selecting an option really isn't difficult to solve. It could be tied into the voice recognition or even simply an 'OK' button could be displayed on the on screen display menu which you move your hand over once you've highlighted your choice. You also wouldn't have to cycle through channels one by one. The speed by which you move your hand would determine how quickly you move through the listings much like the touch screen technology does when you select songs on your Ipod.