I think I've seen this done already. It was Primesense (?) who use a time of flight camera to control various functions through your TV, and I believe it is compatible with 3D TV also. But the two units are seperate, perhaps integrating them wouldn't be too far off.
I can't wait for RGB-D cameras to become more common, because I have a million things I would like to do with them.







