Some people already said it, but an RPG is a game which is entirely built around some sort of progression system (through levelling up, equiping items, unlocking skills or whatever). That’s its staple, what really defines it as a genre from a design perspective. The rest is there (or should be there) just to enhance the former, although some features are traditionally more common than others.

A narrative, for example, gives that progression system a context and helps making it more meaningful, and when done really well it can even add to it, by making the character(s) evolve in more refined ways than just tweaking numbers or options. But there can be good RPGs without a story.

Combat is usually the way the progression system is realized, because it’s the most straightforward "excuse" to make the player grow (generally by obtaining experience or loot when defeating enemies). But there isn’t a particular style of combat that an RPG needs to have in order to be an RPG —choosing one over another merely results in a different subgenre, like action RPG or tactical RPG. And, of course, it’s entirely possible to make an RPG whose character(s) evolve through means other than combat, although it’s very infrequent because it’s harder to think of something that ticks that box while at the same time works well as an interesting and satisfactory mechanic (and thus it’s more risky).

In other words, you can take away the narrative, the combat and all the things that are usually taken for granted in any of the multiple RPG subgenres, and if the game still keeps a progression system as the core of its gameplay loop, then you still have an RPG. But take away the progression system from a game like (for instance) Chrono Trigger, while leaving intact all the other elements, and what you have right there is no longer an RPG —it’s an adventure (in this case). And, if a game has some sort of progression system, but it's not the core of its gameplay, then that game is only partially an RPG: enough to call it that or say that it has RPG elements, but not entirely.

So, taking that into account, what you can do to enhance an RPG is designing the progression system before anything else. That means that you must be able to test it and refine it in isolation, ideally polishing all the details in the process (not necessarily in an engine, it could be in a different platform or tool, or using any other method that is accurate enough). The result should be a robust progression system that feels rewarding and gratifying, and is in itself an interesting game loop. In order to do so in a competent way, of course, you need to know very well what you’re doing, so acquiring a deep knowledge in the matter should be the first step if you really want to go that route. If you can do it, however, it’s quite probable that, while designing the progression system, you come up with some interesting ideas about how the rest of the game should be, and those ideas will most likely intertwine in a very natural way with the progression system that you just created (since they were born from it), making the whole game way more cohesive as a result.

As for what can make the game worse, a difficulty curve is quite tricky to execute correctly and can hamper the overall experience if done badly. That’s true for all genres, but it’s arguably worse in RPGs because of grinding: a particularly sharp spike in difficulty in other games can feel unfair or frustrating for a while, which is generally bad, but that same sharp spike in an RPG has the potential to lead to hours of tedious and boring gameplay, and this description is perhaps the worst thing that anyone can say about any game loop. Besides, the progression can easily start to feel artificial if you’re forced to do the same repetitive tasks over and over just to move forward, and since the whole core of an RPG is the progression system, a badly designed difficulty with lots of mandatory grinding can make the whole game fall apart very quickly, causing it to feel like a chore. Grinding, of course, can be an integral part of the progression system, but, if that’s the case, it needs to be done in a way that feels natural and interesting, organically implemented, and not just some sort of secondary effect caused by an invisible wall between the player and the fun. And, if you want to create challenges in some parts of the game, but don’t want the players to just grind their path to success, you have to not only make sure that the players have the necessary tools to beat the challenge at that point without any backtraking or side-questing, but also ensure that they actually know that they can do it and that that is in fact the intended way to play that part of the game (whether they then decide to do it that way or not). And transmitting that information to the players is way harder that what it may look like, specially in a genre where many people are pre-conditioned to grind even before they start playing. (Note: making parts of the game way too easy is just as bad as making then way too hard, and designing a way too linear difficulty curve is also equally bad.)

In any case, I strongly advice you against making an RPG, unless you don’t have any intention of developing more games after that and you are just doing this as a long-lasting hobby on your free time. The reason for this is that the most likely scenario (EXTREMELY likely) is that you won’t get close to finishing it. And, even if you do it, many many years will have passed and the game will not be anywhere near the quality that you expected it to be at the beginning. So, as a piece of advice, do this only if it’s a pure hobby and you don’t mind finishing it or don’t have any concrete expectations. If that’s your case, then I strongly encourage you to keep going (it's really fun!) =) and you can completely ignore the following spoiler box.

Anyway, if one day you publish this RPG (or a different, perhaps smaller, game), I'll be curious to play it if you share it.

I see where your heart is, but I think you're really off the target here.

Ultima is attempt to make video game "D&D" (it didn't have license). And fairly successful attempt at it. It has everything - character creation (even with with free point buy of 6 "D&D" attributes, something that D&D introduced only in 3rd edition), choice of race and class, overworld map, random encounters, NPCs, dungeons (that are always in the same place, but are procedurally generated for each playthrough), open world and non linear story. You get to sail on seas (once you buy a boat) and you even get to go into space (yeah, it's that weird).

It is not a question of if, but by how much. And Ultima set the standards not just for RPGs to come, but for other games as well (Miyamotio said himself that he played Ultima and that's what made him introduce overworld into Zelda, whose initial design didn't have it at all). Ultima is arguably THE most influential VG IP of all times, especially in the long run (latest example being BG3 and Swen Vincke, Larian's CEO salivating over Ultima VII).

I can see how a lot of people had no idea what Ultima is, especially if they had only access to consoles in 80s, or that "RPGs" (as I said, we called them FRPs back then) were a thing...but that's just how it is. By the time DQ came out, Ultima was on its 4th installment, dealing with completely novel concepts of exploring virtues and morality, instead of trying to defeat Big Bad Evil Guy.

I'd say that resolution mechanic goes hand in hand with progression system - personally, this is where I think it should start.

Something like D20+modifiers vs DC of D&D 3e/4e/5e has different probabilities (and progression) than something like 3d6 under of GURPS, d100 under of RuneQuest/Call of Cthulhu/BRP/Mythras, d6 dice pool (with 6 being success) of Mutant Year Zero engine, highest of d4 and your skill of Savage Worlds, 2d20 under of Modiphius's Fallout or Star Trek and so on, especially since some resolution mechanics don't give just binary Success/Fail, but level of success as well.

I'm not a biologist.

In an RPG that has dice as the main way to determine a lot of what happens, I’d say dice are in fact an integral part of the progression system, so I agree with you. Which dice are used, how many of them can be rolled, how many times, what the threshold for success is… These are all things that directly affect how the game and the character(s) progress.

And they also naturally add randomness, which should be included to a certain degree in all RPGs one way or another, because it reduces predictability. Determining which aspects of the core loop are affected by randomness, and to what extent, plays a very important role in how the whole progression is carried through and how interesting it is for the player(s): too little randomness and the game can feel dull; too much and it can become arbitrary and frustrating. And non-binary resolution adds a nice extra layer of depth to that randomness when done well. =)

For the OP: I focused on video games in my previous post, but designing a tabletop game is a really nice way to start making a playable experience, specially (but not only) if your primary interest is just designing gameplay mechanics (I have a chess-like game "created" this way): no coding, no worrying about sound and music, no need for fancy graphics if you’re just casually testing with friends or family… Just pure, raw game design (and at most narrative if you wish). And this actually makes it a lot more viable to create an RPG game without/before all the fuss that comes from developing it. So a tabletop RPG may not fit perfectly with the idea that you have in mind, but, at the very least, you could extrapolate a lot of the stuff you'd learn.

Yeah, going with tabletop design is probably easiest way to test and develop everything, if it's classic dice based RPG and, especially if it's narrative based, actually make a story through gameplay (Disco Elysium came out of tabletop campaign, for example).

Personally, I see action-RPGs as quite different genre. Let's say bow skill - chance to hit in RPG is based on your character skill and various modifiers (range, cover, wind, target speed, target size...). In Action-RPG, your chance to hit should rely in large part on player skill, while still being sufficiently influenced by character skill (i.e., the higher the skill the less swinging of bow, or shot spread being smaller, or something like that). Action-adventures with pseudo RPG mechanisms of modern era wouldn't have skill influence directly accuracy of your shot, instead giving you some equivalent of feat as you progress (and as such are more of a feat trees than skill trees), AC:Odyssey being good example of this.

So I 'd say, choosing which type of RPG one wants to make is quite important early decision as well.

Expanding on this, I can remember tests from 2000 calling Diablo 2 not a "real" RPG due to it's hack & slash gameplay. So this wasn't just on consoles, it also happened on PC to a more limited degree.

Sorry for the late reply, I couldn’t post before.

TL;DR

I understand why you see action-RPG as a different genre and I only partially agree with you (although action-adventure games with RPG mechanics are definitely different). Also, I mostly disagree that the type of RPG, or the genre in general, should be an early decision to make, specially in the case of inexperienced developers, because that can lead to mindless imitation and lack of real design, resulting in mediocre products. What it’s definitely important is to have a base concept or experience that you want to trasmit (and then find the genre that best fit it).

Apart from these points, my post is mostly trying to expand on my first answer to the questions posted by the OP.

——————————

This may be a subtle difference, but I think what really matters when starting to plan the development of a game, rather than the (sub)genre, is the concept you want to express or the kind of experience you want the player to have when playing your game. And that concept or experience may be intrinsically more suitable for certain genres, but not necessarily so. In the end, a genre is much more like a vehicle than a destination, and what you actually need is to choose the right vehicle that gets you to the right destination: if you want to visit some nearby friend, you might go walking, driving a car, riding a bike..., but never pick a plane or ship; if what you want is to spend some holiday time in an exotic island, however, a plane or ship are the way to go; and, if you want to go to space, you have no choice but to use a spacescraft.

So you definitely must know where you’re going and why you’re going to that place, but you don’t have to immediately decide how to get there, because just knowing the 'where' and the 'why' can naturally lead you to choose one vehicle or another. Consequently, I’d recommend not to give too much importance to the (sub)genre itself and just do what better fits the concept or experience that you want to transmit. In fact, I’d say that starting a video game by choosing the genre can turn into a very bad idea in the case of inexperienced developers, because there’s a high risk of falling into the trap of merely imitating what they’ve seen in other games instead of actually understanding why those work in the first place, resulting in a mediocre list of mindlessly ticked boxes.

About the action-RPG not being a true RPG, it depends: if the skills, stats, equipment, etc. of a character are decisive and definitely necessary to progress, and the action part is just there to (try to) enhance it or make it more dynamic (or to sell more), then I would still consider that an RPG, just combined with some action elements. On the other hand, if the skill of the player is what mostly determines the outcome of the combat and the RPG stuff is just there to offer some variety or provide a certain degree of customization, then that in my opinion is an action(-adventure) game with RPG elements. In the end, the lines that separate different genres or subgenres can become quite blurry, and many games usually mix them in a way that makes it hard to discern where one ends and the other begins. But, as I said previously, if you design the progression system before adding any other elements, you will probably end up with an RPG regardless of whether the combat style is action-oriented or not.

I’ll go a little more in-depth into that and give a simplified example of the thought process of designing a progression system (in the hope that it is somewhat valuable for the OP). So let’s say that we want to create a game and we already got to the conclusion that, among several options that we previously pondered with careful consideration, a level up system is what better fits our game. In that case, we’d probably want to decide if there is a limit level that the character(s) can reach and, if so, which that ceiling is. Choosing one number or another has important consequences, because a small one means that we have less room to make interesting things happen (and therefore break the sense of progression), but a large one could make our game feel sluggish or turn it into a grinding fest (and therefore break the sense of progression). We might then want to pick a value that is neither too small nor too large, but... which value is that? 50? 70? 100? Hard to say without context, so how much experience do our characters need to be maxed out? And how smooth or steep is the overall experience curve? Every new level should (generally) be harder to reach than the previous one, but if we make the difference between them either too small or too big, the sense of progression could be hampered one way or another.

Apart from that, how many characters are there in our game anyway? Only one? A limited, predetermined number of them? A big roster to choose from? Are they all mandatory, optional or a mix of both? For those optional, how would picking one or another, or when, affect the whole progression of the game? And, for those mandatory, are they all available from the beginning or spread out throughout the game? The latter case has the potential to be a lot more rewarding, but also trickier to pull off, specially if they don't share the same experience curve. Which, by the way, do they? Do all characters level up at the same rate? If they do, the progression could become a bit monotonous, so how do we spice things up a bit? And, if they don't, that means that their progress is different, so how do we make sure that all of them feel equally gratifying and that their unmatching progressions complement each other nicely? And what happens when they level up, anyway? A mere raise in stats is not interesting enough (unless we find a way to make it immediately and sufficiently noticeable every single time it happens), but more substantial upgrades may be less satisfying to obtain and not feel as much of a reward if they are given too often. So how do we keep the illusion of progress intact even at the times when there is nothing particularly remarkable going on, or what can we do to eliminate or reduce those times? In general, how is the whole level up system paced?

On a similar note, how often should we test the players? If we don’t do it enough, the sense of progression can wither away, but if we do it too frequently we can break the pace. Also, is the difficulty of those tests dynamically linked to the level of the player or is it fixed? If it’s dynamic, to what extent and how do we make it feel natural? Is the difficulty in the rest of the game dynamic too? How do we maintain the sense of progression if the whole game improves along with the player? What’s more, why do we even need an explicit level up system if we’re just going to change the level of everything in order to adapt it to that of the player? On the other hand, if the difficulty of our game is fixed, how do we make sure that as many players as possible experience the game in the way we intended? If it’s easy to fall too much behind the level of our tests or blow past it, then our level up system needs to be better balanced.

And so on.

I need to point out that many of the features that we take for granted when playing a video game shouldn’t be taken for granted if we aim to create one: even the stuff that seems the most basic and obvious should be scrutinized in the context of the specific game that we’re developing. The questions above are just a sample; when actually making a game as complex as an RPG, many more should arise, as the more general ones usually lead to several others that are more specific. And each and every one of those questions should be followed by the most important one, which is "why?" What does that specific feature actually bring to our game and what does it add to the concept or experience that we want to transmit? If we can’t justify its presence with at least one compelling reason, or that reason is simply that there are other similar games which do the same thing in the same way, then it’s quite likely that that feature should be reworked for our purposes or that it shouldn’t be there at all.

Also, please notice how none of the questions above tackles any mechanic other that the level up system. Sure, we’ve mentioned characters, but only in relation to that system and the way their presence or time of inclusion may affect it. And we’ve also talked about testing the progression, but we’ve never jumped into any specifics: we may or may not design a combat system to achieve that, and, if we go that route, at this stage we still don’t know (or care) if the combat will be action-oriented or turn-based. And, of course, that doesn’t affect the genre at all: it’s pretty clear so far that we’re talking about an RPG game, regardless of the combat style.

That clarity might get somewhat lost when we finally get to materialize the progression system or when we start designing other gameplay elements and integrating them into what we already have. That’s natural, and in every development cycle there needs to be some flexibility, because changes are always going to happen and the end result is almost never going to be what we planned at the beginning (sometimes for worse, some others for better), but it's also one of the reasons why the introduction of an action-oriented combat may often dilute the "purity" of the RPG experience.

The other reason is, of course, that turn-based combat is a more natural fit for "pure" RPG experiences due to how it works in synergy with them. However, action-oriented can also be integrated nicely without damaging the RPG roots. Auto-aiming on distant fights and "stickiness" on melee combat, for example, are two features that can be turned into character stats more or less easily if we wish to, and they could work well with the more traditional ones: more precise auto-aiming or a higher degree of "stickiness" could be compensated with something like lower attack, making different characters fulfill different roles. However, the values of those stats would still increase with every level, so even though there would be skill required to beat the enemies, the character progression would still be clearly noticeable during the fights and the player would need to adapt to those changes. (Whether doing this is a good idea or not depends on the particular game that we’re making, what it tries to achieve, how it is balanced, the way it is designed... But it's just an example anyway.)

In any case, what is (arguably) true is that turn-based combat is harder to design properly than its action-oriented counterpart: in not so few RPG games, it’s often the case that for every character there’s a strategy that we could consider more efficient than the rest, which means that it makes us win faster and more easily. That strategy is also usually more boring, but when players have to fight over and over again in order to progress in the game, at some point they will want to get through those portions as fast as possible regardless of how fun or boring it is, so if they notice that there is one action that they can repeatedly perform to win faster, they will just do it, because, even though there may be other more clever and interesting tactics that they could come up with if they tried, they are just not worth their time. So, in the end, if the combat in your game can be largely reduced to just smashing one key or button over and over, why not making it action-oriented instead?

As a clarification, action-oriented combat can have the same problem, but it’s easier to mask thanks to its more dynamic nature and the higher amount of visual flare that usually comes with it. In any case, that "single strategy" issue is just a part of one of the of the many aspects (balance) that must be tackled when designing the combat system (some others may include choosing between random or visible encounters, creating the different types of enemies and the combat synergies between them, deciding which powers or skills our character(s) will learn and in which order, etc.).

As an extra note about modern games and the presence of RPG elements in (*exaggeration*) every game in existance, the "problem" is that the roots of the genre are so versatile that they can fit almost everywhere with just a few tweaks and, at the same time, they have some intrinsic benefits that make them quite desirable for developers. In particular, having some sort of progression system in place, even if secondary, is a very straightforward way to continuously provide the players with a list of short-term goals that they can look forward to: better equipment, improved skills, higher stats... All of that, when done well, acts as a reward and serves as a motivation to keep playing either a little longer or in a different way (or both), while also giving the players a certain sense of accomplishment every time they reach their next target. It’s basically a regular dose of dopamine, and it helps increasing player engagement, which is something that companies value a lot these days (specially if the game has, or is planned to have, paid aditional content). Those RPG-like elements can, of course, be better or worse integrated in the whole experience, but that’s a totally different story.

I have to be honest, I don't design video games, so this is all theoretical - the only time I was involved professionally in VG design was some 2 decades ago, the project was way overambitious and, expectedly, failed as early as prototype stage. I can't say I miss it much (I work in the film industry, so it's never boring), and given that THE game I would like to make still cannot be made (though it's getting much closer), I "limited" myself to tabletop RPG design, which honestly is what I prefer anyway. So most of how I look at VRPGs still comes from TTRPG point of view.

That said, I do consider VRPGs to be, for the most part, an attempt at converting TTRPGs to video game format, and not completely its own thing. So making an VRPG, at least systemically, would be, from my perspective, not that much different than creating an TTRPG.

The reason why I consider action-RPGs as a different genre is that they can have all the systems of TTRPG, but in addition have a whole another layer that needs to be taken into account, that being action gameplay and, from my perspective, "proper" action RPG would translate attributes and skills of an RPG into actual action gameplay.

In my previous Bow skill example, a TTRPG/CRPG would take that skill, and, after applying all modifiers, trigger resolution mechanisms and see if your character hits or not. Ideally, in action-RPG (again, from my perspective), you as a player would aim, and it would be up to you to aim as best as you can, but Bow skill will influence how easy it is for you to do that, or maybe how much spread your shot has, once an arrow has been released (there's always that weird middle ground where TES:Arena/Daggerfall/Morrowind fall in, where actual skill roll determines a hit chance, although you, as a player, are controlling actions of your character).

Another example could be Strength and Size attributes. In RuneQuest/BRP/Mythras, your melee damage modifier comes from a combination of two (in addition to weapon damage). Ideally, in action-RPG (again, from my perspective), those attributes would be translated to actual physics based properties (Exanima and Scum try to do that, to some degree). So your character Strength and Size would be tangible physical values in those world, which would directly influence how hard he hits, or his Agility would directly influence how quickly he performs actions, Speed how fast he moves and so on. Of course, there aren't many games that try to do this and most are satisfied with having fairly fixed action gameplay, with various RPG stats affecting numbers and not actual moment to moment gameplay itself.

So, I guess what I'm trying to say is, while RPG and action-RPG could be pretty much designed the same way on macro level (world, plot, even some mechanics), when it comes to gameplay part, I do see them as quite different beasts. Personally, I would never try to make an action-RPG, cause that would mean intimate knowledge of how to implement action mechanics in conjunction with RPG mechanics, and, to me, that seems much more difficult to do (at least properly) than designing turn-based RPG, for which there are plethora of ways to do and reference, either in TTRPG or VRPG area. Hence, why I said, choice of genre should be fairly early decision as well.

An rpg is hard to define these days. Becasue all the elements like "upgrading" are utlizied in most games these days. I think the definition of an RPG is based on who ever is playing the game and what there experience is with playing an RPG was. For example some people might consider Call fo Duty an rpg because you "level up".