Have you watched the free Xbox Speech tutor? I strongly recommend it. I'm still learning. I still have some learning to do. I'm really good with the old 360 system, and it's a little different on the Xbox One.
Also, as people said, make sure you calibrate it. It's really important. The 360 version I did a few times under different situations (large windows open/closed, etc.)
You have to say things as it expects them: "Xbox got Forza 5" If you say, 'Xbox, um what did I want to play, um, yea Forza 5" it's going to ignore you. It's "Xbox Goto (full name of app)."
If other people are talking, yes the speech can be effected. Surprise you are having issues with 'Xbox On.' It's works for me, and even turns on my TV and A/V Receiver. Try a little pause between the two words.
I can even say during a movie or game - "Xbox, volume down." and it adjust my receiver. It's great, as my old trusty Harmony Remote is starting to wear out. Plus I have fussy, fussy neighbors that freak out if they hear anything.
Good luck.
PS Just order a cable box (for free) so I can use some of the TV features.