I don't necessarily think core gamers in Japan have lost interest in mature titles, although the total number of core console gamer has been probably dwindling due to various reasons such as the declining birthrate, the popularity of handheld, PC gaming, the diversification of the sources of entertainment, etc.
There are conventional theories about the PS3 failure in Japan --- the high pricing of hard and software, the unavailability of mega titles, the release delay, the difficulty and high cost of game development for 3rd-parties (making them focus more on sequels rather than entirely new stuff), the problem with backward compatibility etc.
If there is another reason, it must be that Sony has lost much of its brand image among gamers and game-retailers in Japan. The name "Sony" became lame to them.
Sony used to be a symbol of "cool" until the early 2000s. The name was almost an equivalent of taking risks and doing new things. The success of PS2 was a big plus for improving their image among young audience, while Panasonic and Nintendo failed with their 3DO, N64, and GC.
Due to the success of PS2, however, Sony became a bit arrogant and started to do what Nintendo did in the 1990s all over again - antagonizing retailers by controlling retail prices and restricting the distribution of used titles, etc. In 1999, Kutaragi started to say really dumb and aggravating things to the mass media. His classless verbal attacks against Nintendo were quoted over and over by the media -- lowering the Sony image. Just like Panasonic did with 3DO, Kutaragi called PS3 "a home information appliance" instead of "game machine", which was seen as lame by most gamers. Furthermore, he said "It's a PSP's spec" to people who reported a button problem with PSP, which dissapointed a lot of PSP supporters. The XCP problem with their music CD lowered the corporate image among young audience, although it is not about game per se.
By 2005, people seriously started to question if Sony was still "cool".
What's worst was when Sony employees started to post praise for their product on various Internet forums and blogs without revealing their identity, pretending like innocent customers. They also fiercely attacked people on various blog and forums who criticized Sony. Viral marketing I suppose.
These activities by Sony were revealed later when administrators of these websites and forums identified the ip addresses of these posters, which were from Sony.co.jp. This was widely publicized and destroyed their brand image among gamers.
For a lot of gamers and game-retailers, the Sony's once-progressive image was lost. To them, Sony = Lame. This is presumably an auxiliary reason why Sony is failing but it certainly didn't help.
It's a bit sad because I liked what Sony used to be and what they represented. I really hope things will change better for them.