It would have sold 20-25M. To do more it would have needed far more, proprietary memory sticks was just one of the problems, the others were:
1) price, particularly once Ninty was forced to make a huge early cut on 3DS, exceptionally (for Ninty policy) selling it at a loss for a while
2) Sony giving up too early on believing in it and supporting it
3) bad marketing
4) losing MH franchise, one of the most important for PSP, and not replacing it with games as much important
Solving all these problems it could have sold 40-50M, not more, as it had the bad luck of coming in a period of portable decline that only Ninty with the hybrid was later able to solve.