I got my PSP when it was still relevant and I got a 4GB memory card for $8. Hardly costly at all.
I have about 18 PSP games. Needed a 32GB card to fit all the PS3 Style game installs onto it. That card cost me $30 in 2019. I know a 4GB card would barely hold a few game installs, and not installing the game would make it run slower and/or take up battery life. I forget.