Elon Musk to start an AI Game Studio

[sc94597]

Mnementh said:

sc94597 said:

Yes, scaling by adding more data likely won't work anymore, BUT we are starting to see scaling using search-refinement in test-time compute. That is what OpenAI is doing, quite roughly in this first iteration, with their o1 model series. 

I've fed o1-preview some graduate level algorithm design exams, and it consistently is able to get in the 90%-100% range on them. A few of these algorithm designs are not something it would be trained on. That is enough for me to think it is capable of doing some rudimentary reasoning in a way previous LLM's (which would get 50-60% on these exams, when I fed it to them) could not. We're practically at Level 2 Reasoners in OpenAI's roadmap. 

Also there is still not a fully realized multi-modal model. The current "multi-modal" models are really a stitching together of various specialized models. So advancement can be done in that direction as well. And of course LLM's aren't the be-all end all of AI research. LeCun for example, is working on his JEPA architecture. 

So there is still a lot that can be done to advance things, even without going into topics like neuro-symbolic hybrids (which AlphaGo, AlphaGeometry, AlphaProteo technically are), and neuromorphic computing in general. 

I don't think we'll have another AI winter. The difference between now and previous eras is that previous AI winters were mostly caused by funding waves drying up in academia and the fact that we were hardware-limited. AI research is now an industry in itself (which it really wasn't in the past) and that is likely not going to end. There might be bear markets for AI development, but they're going to follow the regular business cycle, and advancements will still happen during them. 

Thanks for the Manna link, I'll definitely check it out. 

So I started to skim the links you provided, thank you for that. Especially neuro-symbolic machines align with my thinking. At university (or it is called college in the US I guess) I learned Prolog in my computer science. It was the classical approach of symbolic reasoning, an earlier product of AI research. But it was tedious, as you had to manually model every knowledge and you could run into edge problems pretty fast.

But with the success of LLMs recently and seeing their particular limitations my thought was: what if an LLM at training is instructed to translate their summation of the training data into a symbolic model akin to Prolog (sorry, Prolog is the one I know, there are probably better symbolic languages out there). This has many advantages: a symbolic world model works better to hold context than textual reference, so it could help an LLM to keep focused on the task instead of "forgetting" earlier parts of the conversation. A world model is also helping to keep an general understanding of things. Also while a big LLM can build the world model inside the neural net it comes at high computational cost and massive training data. Symbolic representation of a world model can be cheaper and computational faster.

LLMs are powerful because they operate on text. We have text/writing since a few thousand years and have developed it to be used in many contexts as a powerful tool. For instance modern math is interfaced with specialized text (mathematical symbols) which LLMs obviously can work with. We have textual notations for many games like chess or go. And most programming languages operate with text. All this LLMs can interface with and operate on. And yes, this includes symbolic reasoning, as prolog shows. So yes, combining an LLM with a symbolic reasoning system with a textual interface, which an LLM can query while processing requests could be an improvement I think.

Prolog is still the common general-purpose logic programming language (although it has many dialects now.) I took a Knowledge Representation and Reasoning course earlier this year, and we used Prolog. We also used a language called Clingo for answer-set programming. 

To an extent LLM's (especially now that they're incorporating search in the form of "reasoning tokens") are already neuro-symbolic hybrid systems. We've also seen the success of neuro-symbolic systems in the Alpha series of models, where rule-sets are made explicit and then the reinforcement learning is used to optimally traverse those rule-sets. These models are super-human in their narrow fields. So it is definitely a powerful combo. 

Last edited by sc94597 - 5 hours ago