This.

Also, while I'd personally like better AI I feel a few things have conspired against it:

1 - reduction of importance of SP with combat centric AI coupled with huge increase in importance of online MP with no AI requirement whatsoever

2 - reduction almost to zero of use of 'Bots.  Killzone 2/3 and UT3 are the only recent games I know of that have Bots and fairly decent ones at that.  From a combat perspective I'd say the best AI was never in SP but actually in 'Bots

3 - increase in sales of titles with 'appearance of life' AI such as Oblivion, Assassin's Creed, etc. where the focus from SP is less on combat and more on having a sense of reality around NPCs shopping, eating, etc.

4 - lack of more innovative titles that could use better AI constructively - for example I do think AI in RPGs could do with some serious hikes and I also think it would help the gameplay

Therefore I think that for many titles there seems, from a developer standpoint, little justification to invest heavily in pushing the envelope with AI.  As Twesterm notes clever coding aligned to careful level design - FEAR being a great example of this - can give the impression of much smarter AI that it actually is, as I think for a lot of developers that's seen as enough now.  The focus is on fps, graphics, netcode, etc. for a lot of the combat titles while - so far- other genres that could push AI in a different way also seem to be failing to.

I agree with OP, AI is terrible in most games and relies more on cheap developer tricks and exploitation of scripted events to gain the upper hand rather than intuition and drive and any real thought behind their actions.

Its a shame, its high time they bumped up AI to match visuals, storytelling and sound in games today.

Why focus on AI, when the main point of your game will be Multiplayer? Saves you time and money if you use already old AI.... (Unless its a SP only game)