The Neo-Geo was successful for many years. What ultimately did SNK in was the rampant Neo Geo piracy. Apparently there were lots of bootleg Neo Geo carts floating around (which is odd because Nintendo once chose carts over CDs to prevent piracy).
The Neo-Geo was great for what it was and I'd say it was pretty successful considering that SNK manufactured the hardware up until 1997 (7 years) and released Neo Geo games up until 2005 (15 years after launch). The Neo-Geo was never meant for mass consumption. It was a high-end home arcade machine with arcade perfect games targetted towards a niche audience (with lots of disposable income. lol). It's similar to how a Ferrari doesn't sell anywhere near as much as a Honda Civic but yet no one would dare claim that the Ferrari is a financial failure. The Ferrari by nature is intentionally exclusionary. They market to a specific niche and they make it work for them. The Neo-Geo is similar. It was basically the Ferrari or home arcade gaming.
Nowadays, arcade perfect ports on consoles isn't a big deal anymore since it can be done now at a low cost with Next Gen consoles or the PC. But back then, Neo-Geo had it's niche.