You should hate yourself instead.
Jokes aside, the game is that, seak and beat colossi.
There is a story going on, it develops mostly during the end of game, tho.
The colossi gets a lot "harder", tho, not in the sense of pure difficulty but on working out how to beat them.
The more you realize the colossi are the actual levels and the map is there just to make you wonder what you have been doing (and seek upgrades and collectibles, as yes, there is stuff to look on while exploring), the more you'll realize the actual design of everything being as it is.
Shadow of the Colossus is a masterpiece, one of the most powerful videogames ever, you should stick with it.