graphics do matter, its the difference between having a game running on your system or not.
If a next next next gen game launched for extremely high end PC's, I wouldnt be able to play it (luckily no-one makes those games)
If a game is made for PC/PS3/360, the wii and handhelds are not going to get them (not a respectable version anyway)
however if a game is made for low end hardware, all the more powerful devices can still play them.
it comes down to how much you want to spend and where the developers are and most people are spending on the high end consoles and thats where most the developers are too.
Graphics matter to a certain extent. I dont think it matters for XB1, its still getting the same games as the slightly more powerful ps4, just a lower resolution. wii U however wont be getting the same game so it should be worried.
for handhelds, theoretically vita can have any 3ds game but 3ds cant have every vita game.