I have issues with both also.
The D-pad is sunk in to the controller too much. Feels like your thumb fell into a ditch. Makes it hard to move in motions sometimes. It's great for clicking in directions though.
My issue with the top bumpers are different. I've always used my pointer fingers for both the bumpers and triggers. Ever since the first PlayStation. The Xbox 360 pad had the perfect size and distance with how the bumpers and triggers were made. Now on the Xbox One, the triggers have gotten bigger and are further out than the bumpers. So when I try to press the bumpers now, it feels awkward with the way I placed my fingers on the other pad. Feels like my finger is coming out of it's socket. Over the weekend I managed to come up with a new way to hold the controller without hurting myself. Instead of using the tips of my fingers to press the triggers and bumpers, I position my pointer fingers to cup the bumpers first, so I'm pressing it with the inside of my pointer finger, instead of the tip. Then when I need to press the triggers, I slide it down and press that with the inside cup also.