Transcript:
Alex Lupsasca: I met Mark Chen, who works here at OpenAI, he's chief research officer, and he gave me a challenge.
Alex Lupsasca: He was very proud.
Alex Lupsasca: He said, you know, why don't you just give it a hard problem?
Alex Lupsasca: And I thought, h, ha, you want a hard problem, okay.
Alex Lupsasca: And so I gave it this question.
Alex Lupsasca: Right.
Alex Lupsasca: So I had just found these new symmetries of black holes, which is something that doesn't happen that often.
Alex Lupsasca: And I'd written up a paper that came out in June on the archive, and I was very happy about that.
Alex Lupsasca: And I thought, okay, well let's see how GPT Pro handles this new question.
Alex Lupsasca: And so I gave it the equation.
Alex Lupsasca: And I didn't say that it has some symmetries.
Alex Lupsasca: I didn't give it a leading question.
Alex Lupsasca: I just said, what are the symmetries?
Alex Lupsasca: And it thought for five minutes and it said, no symmetries.
Alex Lupsasca: And I go, ha.
Alex Lupsasca: It's not there yet.
Alex Lupsasca: Still better than the AI.
Alex Lupsasca: And and Mark Chen is visibly crestfallen.
Alex Lupsasca: He goes, okay, well just um just give it an easier question then.
Alex Lupsasca: And so I think, okay, I'm gonna give it the warm-up baby version of the problem, which is find the symmetries of this equation, not in the full black hole space-time, which is complicated, but in the flat space limit with where the space-time is empty.
Alex Lupsasca: And
Alex Lupsasca: Hit enter, it thinks for you know nine minutes, and it comes back with this beautiful answer.
Alex Lupsasca: Oh, this equation has conformal symmetry, which is the the correct thing, and here are the three generators, and it was very beautiful.
Alex Lupsasca: And you know this version of the equation it
Alex Lupsasca: probably has been studied, I'm sure has been studied many times over the decades.
Alex Lupsasca: So I don't know what he did exactly, but he came up with the answer.
Alex Lupsasca: And I thought, okay, this is very good.
Alex Lupsasca: This is a great outcome.
Alex Lupsasca: And then Mark said, okay, well, but now that it's been primed on the warm-up example, try again in this instance of chat the harder problem.
Alex Lupsasca: And I thought, okay, let's go.
Alex Lupsasca: And so we give it the heart problem again.
Alex Lupsasca: Hit enter and it thinks and it thinks and that was the first time I saw it thing for so long.
Alex Lupsasca: I think it took 18 minutes and it comes out with this beautiful answer that was completely correct.
Alex Lupsasca: And that blew my mind because I had been working on this for a very long time.
Alex Lupsasca: And I would say that that calculation is at the edge of my abilities.
Alex Lupsasca: I think it's something that, you know, very few people could have done the way I did it.
Alex Lupsasca: And so I was really shocked because you know you spend years of your life training to be best in class or something and finding symmetries of black holes and these kinds of equations.
Alex Lupsasca: That's that's my jam.
Alex Lupsasca: And I thought, okay, so I guess that just happened.
Alex Lupsasca: And it really sent my mind reeling.
Alex Lupsasca: And I was a little bit shell-shocked for a few days, and then I just couldn't stop thinking about it.
Alex Lupsasca: And after that I realized, okay, I have to become involved in this because to see this capability emerge into the world.
Alex Lupsasca: Like right now and not to not be involved with this just seemed crazy to me.
Kevin Weil: I was gonna I actually think you made you made a really important point in the middle of that around the the fact that you gave it the hard question.
Kevin Weil: It didn't get it right.
Kevin Weil: You gave it an easier question, it got that right, and then you were able to give it a harder question, it got there is still, you know, as excited as we clearly are about the future here, there's also a very real sense, like you when you're giving
Kevin Weil: GPT-5 or any of these AI models a problem that's on the frontier, that's at the limit of their capabilities, they tend to still be wrong a lot.
Kevin Weil: Right.
Kevin Weil: Kind of like any human would be at operating at the level of at the frontier of their capabilities.
Kevin Weil: And it takes you know, it isn't just
Kevin Weil: Automatic yet.
Kevin Weil: Hopefully in the future it will be, you know, enter in any hard question and the model answers it.
Kevin Weil: But today there's a lot of back and forth.
Kevin Weil: And the people that are best, the the researchers that are best at getting the most out of the models
Kevin Weil: have a sort of patience to go back and forth with them.
Kevin Weil: I think that's natural.
Kevin Weil: It's probably the way that you would work with any any you know any two people operating at at about the limit of their capabilities.
Kevin Weil: But I think it's important especially for folks listening to this who are doing research with the models to know that it's not it isn't just one shot and it always works.
Kevin Weil: It there really is a a back and forth and sort of a patience that it takes.