I think it depends on the franchise.

It might be different to the Yearly release ones, to Generation releases one.

Basically, what Resident Evil did. Losing the fixed perspective and tank controls wouldn't of been bad. But converting the series to action bullshit was a line they should of never crossed. If the main Pokemon games dropped turn base. That's when the franchise would screw itself.

Enough to stay in their own genre at least ...

Ive played every single FF game. Even most of the spib offs. And I do not like 13. I hated 13. HATED IT.  And thanks to the bitter taste 13 left in my mouth. I didn't even give 14 a second look and I loved 11. 

I hated 13 so much that I didnt even bother finishing it. 

As long as it makes each game on a particular franchise its own unique experience and it doesn't feel forced to be there for the sake of it, formulas should always toyed around to keep things interesting, games like Kingdom Hearts, 2D and 3D Marios before NSMBW and Galaxy 2 respectibly, and Zelda are excellent examples of how you should go about making sequels.

A sequel that reinvents absolutely everything from the previous game has no bussiness being a sequel rather than being its own game, and sequels that change absolutely nothing from the previous installment but instead improve on everything have no bussiness being sequels instead of DLC.

It can turn 360 degrees if you want

I think that if only one game after fifteen different entries has managed you to hate it, it definitively proves that there's nothing wrong with a franchise trying to reinvent itself after every entry. Sure, they failed you with XIII, but after so many games where you came out at least somewhat satisifed, I think the franchise certainly deserves the right to make something wrong at one point. And I hear nothing but praise about FF XIV after they fixed the disastrous original release, so it kinda shows my point to the OP. 

Just look at 3D Mario. That's the upper limit of how much games in the same series should change, IMO. Introduce new mechanics and ideas, improve visuals, but don't fudge with the core mechanics of the game; there's just no need. There are plenty of ways top keep things fresh without a compete overhaul of the things that made your franchise great in the first place.

Hmm, that's a good question. I suppose if the game still has its core element, then it can be stretched out pretty far. Taking Mario Sunshine into consideration as an example, it was quite different from 64, with the FLUDD thing and an emphasis on cleaning. However, there was still that core element of platforming, and the FLUDD enhanced it with hovering, high jumping, and that... high speed running technique. Quite different, but still a core Mario game.

I suppose you can say it has to be fun as well. Mario's gotten out of his element with RPGs, and those still worked, also still having that core element of making Mario still.... Mario! It had those Mario staples, and was fun!

I suppose something like making Zelda a puzzle game like Tetris wouldn't work as it's too far. But yeah... as long as it still has the core thing of said IP and is enjoyable.

It depends on the game, some already have a basically good and complete formula that shouldn't be changed but having a spin off or change can be good for even those series.. It's really case on case