This will sound like a cop-out answer, but it entirely depends on the specific piece of media and the execution.
Making a fim about feudal japan with exclusively japanense cast? Completely acceptable. Making the same film with colorblind casting? Wierd, but if you can pull it off, more power to you.
Making an adaption of Attack on Titan with an exclusively japanense cast? Yeah, it bothers me because it's unaccurate to the source material, where the cast is actually mostly white caucasians, with a few arabic and asian people trown in.
Making very clearly native american/canadian people white in the Avatar the last Airbender adaption, while making the analogue to tibetian person also white and the analogue to japanese people indian for some reason, on top of it being a total snoozefest? Pretty inexcusable.
It didn't bother me that everyone in Lotr is white, even though they certainly did have the leeway to cast some people of other ethnicities here and chose not to do so.
Likewise I'm also not bothered by black Hermione Granger. You can read the character that way, so it's fine.
Even a switch for artistic reasons, that technically goes against the source material but works well within the contrains of the adaptation is ok.
It's things that explicitly go against everythig the source material tries to do, that get me.
I still think it's beneficial to hold on, once in a while and question the artistic decisions we make, and if they are really sound that way, without being judgemental about it.