In your PS3 select
Settings > Display Settings > Video Output Settings
Select the HDMI option, select the Custom option, you're brought to next step where you're shown the video modes.
You can test the selected mode from the list by pressing the square button, if it only shows a black screen after 30 seconds of test you'll be brought back to the safe lower resolution mode. Enable the modes that work, then go to next step and press X button to save. Your PS3 should use the highest resolution among the ones you enabled.
PS: as it was said in the post before this, if anything goes wrong and you're stuck with a mode that your TV can't display, you can keep the power button pressed for a few seconds when turning on the PS3. It will go into a safe mode in low resolution where you can set the video once again.