I decided to do my first play through on Veteran. I found it to be just challenging enough to still be fun... up until the last level. That is where I thought it just became INSANELY hard. Even here, not so bad until the last stand before Radec and Radec himself. The bit on the balcony. DAAAAMN that part was frustrating.
I finished it on Trooper difficulty afterwards and that was a cakewalk.
I still have to try Elite out.