Inflation absolutely does matter. The further back you go, the more purchasing power a dollar has. It's just a simple matter of fact.
In 1960, a new car cost something like $2600, but a dollar had ten times the purchasing power in 1960 so that's $26,000 in today's dollars, about the starting price of most popular non-luxury sedans today (e.g., a Toyota Camry starts at just under $26k). In 1960, $26,000 would have gotten you a new Rolls Royce. A Big Mac was only 45 cents when it debuted in 1967, but now it's $4 and nobody bats an eye. Gasoline getting up to over $1/gallon in 1979 was beyond painful at the time, but would be beyond dirt cheap today. $400 for a game console today is affordable, but 30 years ago would be viewed as obscenely expensive (see the Neo-Geo).
Rarely do you see sticker prices remain relatively unchanged for multiple decades, but video games are one of those things. The only thing besides console games that I can think of from the early 90s that had roughly the same sticker price then as today is pizza.
If you want to bring up other economic metrics, well, average inflation-adjusted earnings for workers has remained relatively static since the early 80s. However, stagnant does not mean declining. Yes, some things have outpaced inflation, e.g., housing over the past 20 years or so. But many other consumer goods are technically more affordable today than they used to be. At least prior to the recent bout of higher-than-average inflation, some necessities like food, clothing, and electricity represent no more of a share of someone's income today than they did decades ago, perhaps even less according to some metrics. Major appliances and consumer electronics are far easier on the wallet today than they were when I was a kid, and in many cases you get far more bang for your buck than you used to (e.g., the cost per inch of screen for TVs has plummeted). To go back to video games, if someone working minimum wage wanted to buy a new copy of Street Fighter II for the SNES in 1992, they had to work 70% more hours than a minimum wage worker this year would have to work to buy a copy of God of War: Ragnarok for the PS5 when it releases.
TL;DR: You absolutely cannot ignore inflation. If video game prices kept up with inflation, then even ignoring cartridge-based games and sticking with disc-based titles they'd be at least $90-100 brand new today. Video games are more affordable than they've ever been, even if some publishers have upped the price for the first time in 17 years.