The whole secret to the trick is the state of the person's body being lifted.
The first time they try to lift the person, the person's body is in a relaxed state so their weight will be shifting around when being lifted (arms bending/moving, legs bending/moving, persons chest moving, body twisting, etc, even if it appears the person isn't moving much when people are trying to lift them their body's weight will indeed be shifting around). This causes the efforts of the people that are trying to lift him to work against each other and themselves as they all are constantly trying to adjust for a constantly shifting weight.
The second time they try to lift the person, the person's body is in a rigid state (on purpose by someone who knows how the trick works, or from a reflex reaction from an attempt at trying to lift their body a few seconds/minutes earlier, or from a chant or prayer that instructs the person to remain rigid). Because of the rigid state their body it's weight doesn't shift around so the four people can very quickly adjust their efforts so they're working in unison and can lift the body easily (At first the people lifting will instinctively use their full strength because of their failed first attempt and they'll be startled at how quickly/easily the person lifts up and how they seem lighter because they're reducing the effort from full strength to something a lot less since they're all sharing the load almost equally)
P.S. That is why variations of this like "Light as a feather, stiff as a board" require the person to keep their body rigid/stiff, otherwise it won't work.
