Seriously, in so many modern films and shows the characters are always mumbling and the background noise and music completely drowns out the dialogue to the point where I need subtitles just to avoid missing crucial lines.

I thought I was just getting hard of hearing in my old age, but when I watch something from say 20 years ago there's no such issue, it sounds fine.

Anyone else find this to be a pain in the arse or is it just me?

As someone who actually does this for living (I'm re-recording sound mixer), you're not wrong...per se. It is not as bad for theatrical movies, but it is somewhat present in TV shows.

Actors do mumble a lot more than they used to, which combined with the lot of TV directors/producers having delusion they're making theatrical release (which they're not) instead of something that's intended for TV (which has neither dynamic nor frequency response of theater), leads to exactly what you're complaining about. Add to that HDTV standards, that are way, way more tolerant to loudness dynamic range than SD standards of the past were and you have exactly what you're describing.

That said, as someone who both mixes and watches a lot of scripted shows, while I occasionally do tend to run across some that are not well done, I must say that I don't have a problem with intelligibility of dialogue, but I mostly watch shows that are on major streaming services, and they are (mostly) free of such problems.

TV and movies have gone through their "loudness" phase, like the music industry did. As people (and venues) gain access to more advanced (and louder) equipment that can pull off higher decibels without losing too much clarity, there's a built-in expectation of audiences wanting "more" out of said equipment. Sound can be a bit like graphics, in that it's difficult to mix something that works equally well on all types of setups and configurations. The fact also still remains that a lot of people still only use the TVs built-in speakers for shows and movies, and while there have been strides in this department in the past decade or so, it doesn't compare to even the cheaper soundbars with a small subwoofer. TCL have among the more impressive out-of-the box TV speakers I've tried, and often at a decent prices, which is great news for those who want decent quality sound and image without going bankrupt.

Personally, I had to made a lot of adjustments to my setup to make dialogue more distinct without ruining balance, but I've gotten it to a good level now. Most setups with a proper front center speaker have less issues. One of the coolest solutions I've seen and heard is Bravia Home Theater systems using a Bravia TV as the center speaker to underscore dialogue (mind you, only newer TV sets have this feature), while the speakers do the rest of the heavy lifting, it's a rather unique approach (but a high-end center speaker will work better). I have an 85" Bravia 9, so it can be used this way, but I don't have the Bravia Home Theater system, so that's a no-go for me (I have a Harman Kardon system).     

It really is a mess. When I'm watching a somewhat new movie, I am constantly adjusting the volume. I don't have such problems when watching something from the 90s.

Interesting to hear from someone who works on this stuff; thanks for the input.

Yeah it's a constant struggle with most new stuff for me; either the voices are too low, or if turn it up to hear them better then the music and sound effects are way too loud to the point of being uncomfortable.

Conversely, I'm currently watching an episode of Lost from like 2004 and it's crystal clear by comparison.

Yeah, that's dynamic range of HDTV, and especially streaming platforms like Netflix and Amazon have very lax sound dynamics range propositions. I've ran into few shows that are mixed like that, and if you're not listening them in home theater environment (or with some heavy auto level feature turned on), they are quite problematic. That said, this is rarely sound mixer's fault - we always warn person in charge what will happen, and they more often than not don't listen.

Back in SD days not only dynamics were much lower, but we were actually mixing for CRT speakers - that's why you can find dialogue from the olden era to sound significantly more boosted in 2-4KHz range than in HD era, since central speakers "don't like" that range (actually, listening surround mix on stereo speaker setup dips that range significantly compared to dialogue coming from center speaker in surround mix). So, what happens is if you mix only for surround, with center speaker in mind, once it's downmixed to stereo at your home it looses some of its intelligibility due to pure physics (audio masking of human head) - that is if your sound setup doesn't compensate for that, or if you are not using accompanying stereo mix that (in most cases) already has that compensation in itself.

Lost is sort of a middle ground between old and new - its dialogue is mostly treated as if it was for SD, and on occasions is painful to watch on modern audio home theater setups, since it's one of the noisiest dialogues of all mega-hit TV shows in last 25 years, due to production issues with locations they were shooting at and (at that time) lack of advanced background reduction hardware/plugins that are available now.

I should add, the films that prompted the thread was rewatching Oppenheimer and Dune Part II, and I think it didn't help that both Villeneuve and Nolan tend to favour directing actors to speak very softly.

Not sure if the same thing you are referencing, could be the "loudness" thing mentioned earlier, but christ shows drive me nuts. I go from barely able to hear talking to an action scene the blows my ear drums. I get there being a difference, but the difference is insane these days.

Those are theatrical releases, with big dynamic range. 

Process used to go like this - you mix for theater, then you tighten it a bit for DVD/BluRay version (aimed for home theaters) and then additionally for TV. Given that both Netlifx and Amazon (I can vouch for those, not sure about HBO and some others) have made their standards to be pretty much what DVD/BR home theater standards are, content delivered like that is fine in home theater environment - but if you watch them without any additional processing in environment that is not suited for that, you are probably in for a less than ideal experience when it comes to dialogue intelligibility and overall dynamics.

As someone who actually mixes under those standards, I'm not very fond of them - they are great from creative standpoint, cause you can have soft scenes being soft, and punchy scenes being loud, with full frequency range that SD era did not had, but that means what I said initially - directors/producers got into their heads that everything is made as if it's for theater (real or home) and most forgot about actual average living room TV experience, which is more restricted and often accompanied by additional background noise from house/outdoors. So as a viewer, I know what that means on the other side, if you mix that way - remote in your hand.

All in all, my advice, since this will not change, whatever you're watching from this type of content, if you're not doing it in home theater environment (with everything that entails), make sure to engage auto level and dialogue intelligibility  - sure, you won't be listening to "original vision", but you will reduce your annoyance level when it comes to sound for such content.

I noticed when I play certain movies on my TV with a sound bar, if I play the stereo audio track instead of the surround sound audio the dialogue can be a LOT clearer.

I always make sure to have two audio tracks when ripping my bluray/dvd's to my Plex server now rather than only having the surround track.