I am unsure about polygamy because it depends on how it is defined. There are couples who agree that it's okay that one or both of them can have sex with different people, and that's okay in my book because those people are honest with each other. So if all parties involved in polygamy consent, then I couldn't be bothered to object to such a lifechoice because I don't see how that would harm anyone.
You're thinking of polyamory. Polygamy refers to one man marrying multiple women, i.e. a structurally unequal marital relationship.