There are different factors depending on the game type, I think.
With a game that is focused on a particular story, for instance, such as aliens invading earth, when you get to the end of that story, there is a feeling that creating a new conflict just to keep the series going is "milking". How well this is done, though, is very important. If it's logical, if it makes sense, then it's usually forgiven, at least by the fans of that series. If it feels contrived and tacked on, then obviously it's more of a negative.
With other types of games, those that center around the characters or the gameplay, I don't think it really matters how many titles are in the series. Something like Super Mario Brothers or Disgaea would be examples of that.
When you take popular characters and basically re-skin completely different games around them, however, that is clearly milking those characters. A little bit of that is fine, especially with mascot types, but when you churn these types of games out on a continuous basis, then you have to wonder where the creativity is going.
The time consideration is probably a new one, at least with AAA games. It used to be that you had to wait at least a couple of years between releases in a franchise but now they're coming out every year. You have two teams working on staggered releases for the same franchise at the same time. It's getting out of hand, I think, and it's burning people out on those series quickly. Publishers seem to have a "let's milk it dry while it's popular" attitude, which is only good for the short-term.
Still, if the quality remains high, I won't really complain much. It's when the quality starts nose-diving that you really have to get annoyed with the publisher/developer.