I just finished the main quest around Alltrades Abbey and now I can switch vocations, but is it really worth it? My party is a minstrel, mage, priest, and warrior, all either level 20 or 21. Do I need to switch to avoid getting waxed for the rest of the game, or should I be good, or is it really up to me? I really haven't had much trouble getting through the main quest so far, but surely they didn't just throw this option in here for no reason, right? Any tips would be appreciated.

yes. abilities carry over.

you can get all the str bonuses and attk for your physical char and the magical bonus for your caster to make them tougher.

Although, prepare to grind metal slimes if you decide to do that.

Grinding each class to 15ish early on shouldn't long and you'll have enough points to dump in skills you like.

It's also important that you max out 1 weapon skill.

Caster should go for wand or bow.

Pure casters would go for wand while the hybrid ones would go with bow since there's a bow skill that allows you to steal MP based on how much damage you dealt.

Fource is very useful. Learn to use it and you can beat the bosses quite easily.

Minstrel is relatively weak but you should get 1 char to level 40 so you can do the quest that will get you a very very useful scroll.

Level 100 Shield quest is also extremely useful. You should max out shield for all your chars regardless.

Next important piece would be Armamentalist lvl40 quest.

You can probably beat the game with the initial vocations, but unlocking and using the advanced vocations is probably best. It'll take some time to retrain your party members, but it'll be worth it.

So there are quests that can only be unlocked and completed once you have a certain number of a certain ability point? Interesting. I have a lot of work to do it seems.

For weapon quests.  You get 1 at lvl30 and another lvl100.

For class quests.  You get 1 at lvl15 and another at lvl40.

Some of the quest are ridicuously easy and some are painful. A few of them requires you to do a bit of thinking to complete.

The class quests are usually better. You should at the very least do the lvl15 class quest if you're going to use that class a lot. Most likely the rewarded item will be useful until close to the end game. Gold is not that plentiful unless you exploit it.

Boosting Tension if also a good strat to beat bosses. You can 100% in 2 rounds if you combine Egg On with Psyhe Up.

Lastly, get 1 of your char to learn the "Eye for Trouble" skill and scan all enemies. It helps you find items you needed to craft and str/weaknesses of the monster.

Yes, changing vocations is worth it, as you can get stronger, similar classes (gladiator instead of Warrior, sage instead of mage), and all-round classes (ranger, Paladin). 

To get these, you need to talk to certain people (most in Altrades Abbey, but keep an eye out!), complete the quest to gain the vocation. Then, return at Lv15 to get another quest, which nets you some nifty armour. Return AGAIN at Lv40, to do another, much more difficult quest, to gain a class-specific scroll, which do nifty things mid-battle, such as auto-stealing items, new abilities, carrying over tension etc. It's well worth investing the time, though grinding in high-level grottos with metal king slimes is a must, or on Slime Hill (just above Angel Falls)