So the good definitely outweighs the bad.
According to my own research, you should avoid the first entirely. Caligula 2 is sequel BUT it is really treated as a "Redo". The inside joke is that we the player, our regret, is buying the first one so we entered Redo to relive Caligula in the better game :). I found that very cheeky. Even the developer knows the first game was not good, but I still want to try.. just later. Start with 2.