AI doomer/optimist?

[haxxiy]

Zkuq said:

LLMs probably won't destroy us, but it's hard to be 100% sure. It looks like they're capable of transforming at least some fields, but the real question is whether they can do that in an econimically sustainable manner. Economic viability will determine the eventual outcome. I don't think LLMs can change everything though, they're too error-prone, and the technology just isn't on solid ground - we're already pushing it way past what it's designed to do. It's incredible how far we can make it go, but it's still fundamentally flawed for general use.

As for AGI, that will be a much more interesting story. It doesn't look like it's going to happen any time soon, or at least that's what I was saying for the longest time. LLMs might be able to accelerate the development of AGI though, but we'll see how it goes.

Of course all this also hinges on the availability of computing power. It's not clear how much we'll end up needing, or how much AGI will end up needing.

I'd argue this year showed that 10x scaling from 2025 (after two years of similar-sized models being RL-fried to a crisp since the hardware for making them larger was unavailable) was enough to elicit high levels of common sense, which is key to applying technical skills and setting priorities in a way that lets agents do valuable work. The same scaling also allowed LLMs to go from doing well at IMO to solving Millennium Prize problems.

Since there's at least another 100x of scaling until 2032, it's almost certain to me now that all engineering that humans can do is going to be automated just with the current methods, including the creation of post-training/RL tasks and environments to plug AI capability gaps as they get discovered.

The question is whether continuous learning is included in that or it remains inaccessible to LLMs and needs a much more impressive conceptual breakthrough.