Well the way I would do my podcasts is there were 3 regulars and every now and then "special guests" for opinions. Setup was very easy. Had a lineup of things to talk about that was distributed before hand, followed the schedule, and everything went smooth. Of course it is easy to get off topic and go overboard on some discussions but I was always very lenient in that respect. Had one podcast that lasted 3 hours haha.
But a few recommendations, is you need a solid host. Host, like any roundtable, is definitely very important. Someone who keeps the discussion flowing, humorous, and doesn't allow discussions to get stuck or biased. I was alwasy very good about that in my podcasts haha which is why I hosted. Second, never have more than 5 people tops. You get more than that and it'll get hectic or very timely. That 3 hour one I had, had almost 7 people in the podcast haha. Finally, have a strict schedule that is followed loosely. Meaning have topics setup with questions, news, and discussions ready to go but make sure the conversation flows naturally rather than forced.
But with podcasts, trust me, the hard part is not in actually doing the podcast. The hard parts are editing and uploading. They are tedious and brutal. Not to mention setup beforehand especially when you have people that span across countries. But I was always able to manage it for the most part. But lag can sometimes be a factor but hey whatever.