Because we (the West) recognised the PRC as the legitimate government of China. For example, its permanent Security Council seat.
As to why territory is the way it is, these things are not based on fairness or "rights" to some land. It's based on military superiority and the determination of the people. Mainland Chinese people see PRC as government -> PRC is the rightful owner of mainland China.
We gave it back not because of the contract so much as because the UK no longer wished to have an empire. And the PRC made more sense to give it back 'to'.







