I would actually say RE started going downhill because of RE4. Not that it was a bad game (neither was 5 really, though 4 was better than 5), but it wasn't the same genre anymore and, well, I think there were better action games at the time (at least in my memory).
Not that every game has to stay in the same genre per se, but RE4 wasn't even remotely scary and had little to no puzzles. It was mainly shoot, shoot, find treasure. Only thing that made it better than 5 was the fact that you had a decision on keeping treasure and combining it with other stuff.







