Agreed. Although the driving becomes less annoying as soon as you get the hang of it, the missions also get more difficult, which together with clunky shooter controls and very unforgiven (barely non-existing) checkpoints deliver a good portion of frustration also. It does have its good moments and I managed to finish it, but it doesn't qualify as a good game by any means for me.
The 3rd one plays much more like the 2nd one, but it's difficulty is more forgiving and it has more variety in its missions. It's still worse than the 1st one but I would give it a try even if you give up Jak 2.