My experience of working with people like Terence Tao, and being nowhere near their standard, is that they are looking for any kind of creativity. Everything is accepted, and it doesn't have to be "at their level".

Having read what he's saying there, and with my experience, I think your characterisation is inaccurate.

And having been at the talk he gave for the IMO earlier this year he is impressed with some of the interactions, it's just that he feels that any kind of "creative spark" is still missing.

Right, Terrance was hoping it would have something new to think about it, some new perspective, right or wrong. GPTs have the ability to process insane amounts of information across all branches of math, science and art. This ability eclipses that of the most motivated intellectuals such as Terrance. It is thus a little disappointing that it was unable to find anything in its vast knowledgebase to apply a new lens to the problem.

I wonder how much of this is an intrinsic limitation of LLMs, and how much that interdisciplinary thinking and mashing together of problem domains is missing in the training data. It's a pretty rare thing, and the only times these analogies and linkages become noted is when they happen to work out (and they don't seem so far out of left field anymore once this happens).

This is what synthetic data might support— based on mashing together the implications of massive numbers of concepts…

I wonder what the creative spark even is in the context of an autoregressive transformer.

Perhaps it’s an ability to confabulate facts into the context window which are not present in the training data but which are, in the context of maths, viable hypotheses? Every LLM can generate bullshit, but maybe we just need the right bullshit?

That's interesting. I am young so I don't know what actually is creativity. Could you explain that part for me?

> Creativity is defined as the tendency to generate or recognize ideas, alternatives, or possibilities that may be useful in solving problems, communicating with others, and entertaining ourselves and others

Basically can you provide a new perspective on solving a problem that hasn't been considered or a new way of looking at an existing idea in a new way to unlock a path.

That is such a good quote to remember. Thanks for your answer! It actually opened my understanding a lot. I searched your quote and found some other good ones.

You’re asking the right questions, IMO! Chomsky has spent his life trying to answer this question in different forms, and has ultimately arrived to the conclusion that we live in a “Pre-Gallilean” era of cognitive science, where the only answers available to us are developed through the use of intuitive interpretation (like they used to do with ‘the heavens’/space) instead of empirical contradiction (aka science).

He does have some answers, such as “human creativity is the ability to create an infinite range of outputs from a finite range of inputs that nonetheless pertain to our motivations/context in some useful way”, but that’s obviously not a very satisfying answer. It tells us a little — I think Tao is gesturing to exactly this when he complains that GPTo1 can only apply and combine mathematical approaches within a sort of closed domain rather than propose radically new ones - but it’s not helpful for an everyday understanding of creativity. IMO :)

In his words, from Language and Mind:

  "Roughly, where we deal with cognitive structures, either in a mature state of knowledge and belief or in the initial state, we face problems, but not mysteries. When we ask how humans make use of these cognitive structures, how and why they make choices and behave as they do, although there is much that we can say as human beings with intuition and insight, there is little, I believe, that we can say as scientists… 
  What I have called elsewhere 'the creative aspect of language use' remains as much a mystery to us as it was to the Cartesians who discussed it, in part, in the context of the problem of 'other minds."

If this sounds intriguing to you/anyone, I highly recommend his (in)famous debate with Foucault, which is available for free on YouTube. It’s a bit wandering, but about halfway through they discuss creativity in depth, contrasting Foucault’s vaguely postmodern view-that human creativity is mostly constrained by societal circumstances-with Chomsky’s view, that human creativity is mostly constrained by the natural structures of our cognitive system(s).

https://youtu.be/3wfNl2L0Gf8?si=WA3DpnaFEvd3QqCt

Wow. Thank you for your thoughtful answer! > “human creativity is the ability to create an infinite range of outputs from a finite range of inputs that nonetheless pertain to our motivations/context in some useful way” Do you think this can be used as a metric? Like the more useful answer we can come up with, the more "creative" we are. The constraint of outputs and "some useful way" is such a good insight.

> whoever came up with this was able identify what can be traded off to make the previously impossible, possible, unlocking new scales and possibilities.

In fairness, that is an extremely standard trick so it's reasonably unlikely that the author came up with it themselves.

But the first time someone came up with that idea it was an act of creativity.

Yeah I agree. Though it is hard to tell if LLMs are capable of "easy" creativity like that though because anything that easy has already been done many times in its training set.

You've have to invent some new domain I guess and see if it could be creative within that domain. Difficult to think of a good test though.

Thank you for your answer! TIL a new word: qualia I like your two examples. They have different perspectives. Though I must say the "qualia to manifestation" is still a bit abstract for me now. I'll keep them in mind.

> ‘Able to make the same creative mathematical leaps as Terence Tao’ seems like a pretty high bar to be setting for AI.

There's no need to try to infer this kind of high bar, because what he says is actually very specific and concrete: "Here the result was mildly disappointing ... Essentially the model proposed the same strategy that was already identified in the most recent work on the problem (and which I restated in the blog post), but did not offer any creative variants of that strategy." Crucially the blog post in question was part of his input to ChatGPT.

Otherwise, he's been clear that while he anticipates a future where it is more useful, at present he only uses AI/ChatGPT for bibliography formatting and for writing out simple "Hello World" style code. (He is, after all, a mathematician and not a coder.) I've seen various claims online that he's using ChatGPT all the time to help with his research and, beyond the coding usage, that just seems to not be true.

(However, it's fair to say that "able to help Terence Tao with research" is actually a high bar.)

This has been observed by more people than just Terence Tao. Try using chatgpt to program something of higher complexity than tutorial code or write a basic blog post, it lacks creativity and the code is poorly designed.

Even for like basic Rust programs it ties itself into countless borrow checker issues and cannot get out of it, both the OpenAI as well as Sonnet (Anthropic).

It doesn't really get logic still, but it does small edits well when the code is very clear.

I think this will always remain a problem. Because it can never shut up, it keeps making stuff up and "hallucinate" (works normally, just incorrectly) to dig itself further and further into a hole.

Autocomplete on steroids is what peak AI will look like till the time we can crack consciousness and AGI (which the modern versions are nothing even close to).

To me it's quite good for Rust as it usually suggests me some (unknown to me) borrow checker API that actually helps me either solve the issue at hand or at least point me in the right direction to figure what is actually wrong with my code and it doesn't compile

Funny it just wrote this for me:

  fn parse_image_url(html: &str) -> Option<String> {
      let re = regex::Regex::new(r#"src="([^"]+)""#).ok()?;
      re.captures(html)
          .and_then(|caps| caps.get(1))
          .map(|m| m.as_str().to_string())
  }

Prompt was "write a function which extracts image urls from a given block of html"

Of the 40 or so definitions of consciousness, I have no reason to think any is either necessary nor sufficient for any of the things we want AI to do that Transformer models can't (or can but badly and we want them to do better).

This is precisely the first thing I thought

If arguably the person with the highest IQ currently living, is impressed but still not fully satisfied that a computer doesn’t give Nobel prize winning mathematical reasoning I think that’s a massive metric itself

So what then should the first year maths PhD think? I believe Tao obliquely addresses this with his previous post with effectively “o1 is almost as good as a grad student”

> If arguably the person with the highest IQ currently living, is impressed but still not fully satisfied that a computer doesn’t give Nobel prize winning mathematical reasoning

No offense, but every part of this characterization is really unserious! He says "the model proposed the same strategy that was already identified in the most recent work on the problem (and which I restated in the blog post), but did not offer any creative variants of that strategy." That's very different from what you're suggesting.

The way you're talking, it sounds like it'd be actually impossible for him to meaningfully say anything negative about AI. Presumably, if he was directly critical of it, it would only be because his standards as the world's smartest genius must simply be too high!

In reality, he's very optimistic about it in the future but doesn't find it useful now except for basic coding and bibliography formatting. It's fascinating to see how this very concrete and easily understood sentiment is routinely warped by the Tao-as-IQ-genius mythos.

I’m not sure i understand your complaint

Are you arguing that I’m making an appeal to authority fallacy?