8:49
One of the biggest surprises of the last few years has been the ability of coding agents to generalize across domains. And now, researchers on the front lines are showing that this includes controlling robots. This prompted MIT professor Philip Isola to suggest in a recent viral essay that we may be entering an era of robotics, where general- purpose model agents could make various robots more capable. So today, François and I invited the founders of Waddle Labs and RoboCurve, two startup groups working at the very frontier of creating more capable robots using LLM. Maybe you guys want to briefly introduce yourselves and talk a little bit about what each of your companies focuses on.
I'm Jamie. I'm from Waddle Labs with Vincent. We are working on creating LLMs that drive robots, and we are doing this through two things. Creating a system that allows LLMs to do this very efficiently, collecting data, and then using that data to train better LLMs. Hi, I'm Jay, co-founder of RoboCurve. We are an e-house company for physical AI. We measure everything, any robot, any model, including LLM, as well as classic approaches such as VLA, visual language action models, and world action models. And
we evaluate all kinds of environments, such as hands, grips, arms, humanoids, four-legged animals, all kinds of things. So over the last few weeks, videos from both of you have gone pretty viral on Twitter, and I think what inspired me is that it seems like both of you are actually quoted in Philip Isola's essay on robot-using agents. The video showed how LLMs can unscrew lids, communicate between multiple robots, and perform tasks such as opening a handle. So I thought it might be went over some of the research that led to this point with you. And then maybe we could also show some demonstrations of what it can actually do. Why do n't we talk about some early research on transfer between language models and robot control? So, I want to
talk about some of the early work on pre-learning language models for decision making, as well as the RT2 paper that started VLA. So maybe, Jay, would you like to tell us a little bit about some of this early work related to the possibility of using language models for any kind of robot control? One of the earliest successful approaches to using AI on robots is the RT2 paper, where they use a pre-trained language model on web text and images and use this to control robots. What's interesting is that this is a fine- tuned version of a language model, and instead of outputting English, for example, they just output what we call the end effector pose, which is coordinates that can then be converted into common commands that can control robots. In some
cases, this is, in a way, very similar to what we see now with large language models, only instead of fine -tuning the models are good enough to just do it "out of the box." This is actually very similar to the reflection I have in my head, something like a COT moment. And so, like when we did GSM8K, and you said Susie had $8, she spent five, how much does she have now? Hmm, it should have outputted about four hash marks and then the answer. And then EOS. There was no possibility for him to have a chain of thought. So then we let him actually have something like: okay, let me take 8 minus 5 is 3 and so,
hash-hash-hash-hash three. And so you allowed him to do that thinking before he actually took action or responded. Hmm, and similar to this, now the VLA, it's essentially like RT2, it's essentially supposed to output the action. There's nothing I can't allocate more computation to a more complex task. And now I have this chain of thought of code that I can do. And I can say, " Okay, even if you used an example where you don't actually output the code, you just give it to Astra and let it think, think, think, and then output the action." Similarly, now, for example, and then with the code, is it a code with higher Kolmogorov complexity or lower Kolmogorov complexity, the length of the code to just use the code and really compact
code, just like here's the code. Just go ahead. I completely agree with you. It seems to me that in the end, it's a matter of smaller scale, isn't it? If you give an agent, an AI model, more autonomy, and if you untangle it a little more and give it more resources, it can actually do a lot of things that we fine- tune it to do. I'm going to take a look here too. I think the really interesting thing about this lesson is that, like VLAs by design, architecturally, they're also built on language models, right? They can potentially reason, they can write code. So perhaps the best lesson here is not necessarily what architecture you build, but what data is most useful. Yes, people have been stressing over this VLA data button for years, and we're
seeing very, very slow progress. So the lesson here is that maybe we have a data modality that we know works really well, and this idea of transfer between different modalities that I'm really excited about, according to Hamming. For example, how do you take something that is traditionally not distributed, like robotics, and make it something that is distributed? Perhaps the way to use the lesson better is to say: let's choose data for which we know this modality is quite blessed. There's a lot of data there. These language models understand it well and then use it as a way to unlock many other areas. Yes, and also, the reason why RT2 is such a good model is because it uses a language model that affects the modality of all the data that the
language model is trained on. So, all these web images and web tags actually improve VLA compared to simply training a specific basic robotics model without any prior training. So you get the advantage in the case of RT2, these early VLA approaches, from prior training. Um, I think, what's the limitation in, I think, the fine-tuning approach that can now be circumvented if you can write code directly? You mean the difference in data. What exactly is this difference in data that you are talking about? There are a few things that the models have gotten much better at since RT2. Um, for example, they are much better at using tools. So, you know, what data.
Now these models can. uh, well, I mean, they write code much better. So now they can.uh write complex policies as code. They're also much better at using tools to explore.er, the types of environments they're in like what hands they have access to . And a lot of that comes from contextual learning. The key difference between such VLAs and/or GPT-6 or LLMs is not necessarily in the architecture, but in the training approach. We want to be better, as a lesson, right? We want to benefit from all kinds of data. We want to incorporate computer usage data into our robot models. We want to incorporate coding data into our robot models. But when you do that, you just get what
we consider to be universal LLM agents. And then, for me, it's like: okay, why not just build a really good basic LLM and use it to control robots, rather than training a model specifically designed for robotics, which is more dependent on just data about robots With that in mind, maybe one of the things that I think, if we could turn back the clock a couple of years, might be a good sign that we're moving in the right direction is the study of code as politics, right? Do you want to talk a little bit about what exactly it means when the research community refers to code as policy? And I think it's worth considering this in the context of the release of this article, which is the coding agents that were just starting to
work. I think this was the era when people were adding comments and auto- filling blocks of Python code, which we now consider agent history, but that was about 2 years ago. So how does code work as a policy and how did it inspire you to what you now see as possible? I'll probably go back to Voyager, where Voyager was one of the first.What does encoding agent mean? A coding agent requires good use of tools and creating tools on the fly. This is called code. Okay, you have Python modules built in. Let's say I only have built-in Python modules. These are tools, and I have sort, and I have if, and I have for, and I have while, and I have all these things, and I need to use these tools to put together a new tool called francois.py, right? And
now it's a new tool and I can use it. It was Voyager. And yeah, I do n't know, I wouldn't say Voyager was the first to do that. Voyager was the most popular one to do this for Minecraft. And then they created tools that they could call on the fly to help them play the game better, and compressed the thinking and experience into a new tool that they could use later. And then, I think because everyone in 2024 had this understanding, they thought, "Okay, if we just put in, you know, Darian and Sam both understood that if I put all my heart, soul, resources, computing, and intelligence into improving coding, we can take off." Then I can automate the machine learning engineer, then I can take off, and I'll have everything. I will solve all problems."
And that's what everyone did, and I don't think they had that understanding. I don't think they were insightful enough to know, "Oh, if I did this, we'd have LLMs who were good enough to write policies for robotics, and I could replace all robotics with real coding agents, and I could replace artists with, like, coding agents who write JavaScript to create beautiful, uh, art." I'm not sure they had that understanding, to be honest. Tell us a little bit about what some of these early Coda policy methods might have done in general? So even, you know, this is way ahead of the fact that we have encoding agents that are widely available and people intuitively understand
how they work. This is way ahead of people using RL techniques to actually train these coding agents to be really good at calling tools. What some of these early ones did, maybe even before Voyager policy, like Coda, which I think the article came out about in late 2022, around the time ChatGPT came out. What were their capabilities compared to what you can see now? Yes, those Coda policy documents, especially from the Google DeepMind team, were incredible because they created features like picking up an object, raising it, or moving into a certain pose. They provided this list of functions to the coding agent, literally
in the form of Python functions. And then the coding agent will be able to write code that includes these functions, to then control the robot to perform very complex tasks. And what was really amazing or what was amazing about this was that the encoding agents could do it all at once. They didn't need any additional data about the robot to work with this code because they were already trained on such a large amount of coding data. They already have an idea of what to do first, what to do next, to, for example, move a block into a bowl. And I think it's this ability for disposable use, this ability for contextual research, that has really motivated a lot of subsequent work to continue research, including us, to continue research into how we can
apply LLM to robotics. I know, Francois, you have this, um, structure that we talked about, different ways of teaching. There will be learning embedded in the weights, versus learning in context, etc. Do you want to talk about that quickly, and then maybe we can think about how all the research of the last few years fits into that framework, especially the direction it seems to be going in? I mean, I've been working on this experiment, and I haven't actually formalized it into a paper yet, but it's basically something like what's the most efficient way from the perspective of our intelligence to sample learning, let's say in this context
the outcome of a state action or the reward for a state action, to reflect policy. There are ICLs. Yes. That is, it is in the context of the training that we are describing here. Learning where I just add it. This is how most people use LLM. They just say, " Oh no, don't do it like that." Do it like this." And I answer: "Okay. " And then it's still in context, and then I remember that I can iterate. And it really only works if you have trained the LLM or at least post- trained the LLM so that it can learn and improve, and that it is a self-improving mapping, as
described in the literature from long ago, which actually allowed for multi-turnover in the training set. If you don't have that, it doesn't learn in context. But even then, there is a limit to how well ICL can work. So I did this experiment where you take the LLM that we were studying and I set the task to, let's say, GSMA K for simplicity. And then I ICL and I want to measure on the valve set how much it improves based on each sample. And it improves significantly very cheaply. It's not expensive. I don't need it, I don't have SGD, so I don't have any failures. And so I can adapt and improve my valve set very quickly. For example, example, example. The first is non-monotonic improvement. What is wild. So, if it gets worse, then it gets better, then it
gets worse, then it gets better quite aggressively. Secondly, it breaks down very quickly. And so after about 20, 30, maybe 40 examples, it actually becomes saturated, and further putting examples back into context doesn't improve it. So you're kind of limited by the model's ability to intelligently use all of its context. That's right. From the moment after training. How many turns did she actually take to be able to improve and self-reflect? And then the most important thing that it definitely, if not definitely, cannot do is after it reaches the length of the context window of the model it trained on. If she
trained at 100,000, that actually means you have a context window of about 50,000. Once you go above 50,000, your results don't improve anymore. In fact, you're just making things worse because the model can't take everything into account. You gave a very good example that I hadn't thought of, like if I could use rag or load my actions into active memory. And it's very similar to Prime Agent Continuous Harness, like the idea that I can pull things out on the fly, like examples like that, then it feels like I have an exam or a textbook exam. I'll do better if I have a textbook to look up. And that, you know, would definitely work, and it still works, and it also, um, scales, I would imagine, a lot better. And then the last two paradigms would be for Laura with rank one
or two, or some other. Laura with rank 10 or rank 100, and then all the way up to full SFTRL. And if you're Tesla, and you have all the data, infinite data, and you're learning to build a driverless car using ICL, what do you do? Are you kidding? It's clear that we wouldn't do that, right? Um, and if you're something like. But, um, it's amazing how well in low- data modes, um, you can do with ICL, and then even cooler we can compress ICL into tools, use them into tools, or compress them into training. And this is the hierarchy that we have n't really understood, and that you're probably at the forefront of. So, do you want to talk a little bit about that, because I think it's a central part of how you think about building a model?
There are a few things here that are quite interesting, I think, to uncover. So, first of all, we thought about this idea of rigidity almost as a form, um, as a domain specificity, um. For example, when you deploy a robot in a new environment, maybe a wet lab, and it just needs to You can learn a lot in context, but one way to consolidate that context for future agents is to kind of package each of these skills that you've learned into specific, ahem, applications, for example. For example, writing down these skills and recording these memories is a form of consolidation. It's like a form of distillation of past experience for use by your future agents. So that's pretty interesting.
Another rather interesting thing is that, um, there seems to be some connection to the broader literature on metalearning, um, throughout the history of machine learning. For example, you have a larger model of sorts that then programs a smaller model to perform certain tasks. And one of the really interesting things is that yes, these smaller models will learn in context. They are like wrapped in a harness, and they perform the task. You can make them maybe smaller, you can make them run faster if a larger model can do that specialization of your system well enough. So, this is what we are very interested in right now. And one final note, I think the really interesting theoretical question is how far And I think that this is indeed a solvable issue.
For example, there are papers showing that these models can also approximate gradient descent during contextual learning. So this boundary between weight space and symbolic space, I think, is a little blurry. The next batch of YC is now accepting applications. Do you have a startup? Apply at ycombinator.com/apply. It's never too early, and filling out the application will take your idea to the next level. Okay, back to the video. So what exactly are we observing here? Er, we watch Ashra maneuver her hands to lift the block off the table and place it in the bowl. So here, Ashra is just using this camera input, and presumably she knows about the robot she's controlling, and she's going to write code that
controls the robot at its individual joints. Yes, this is done on all cameras. She has all the feeds from the cameras. Instead of code, it's more like a tool call. So she sends robot teams to control and place the block into the ball. So in this situation we're using Astra directly, but if we were using, say, Weddle straps, what's the difference between using Astra directly to manage this and doing it through Weddle's API? Sometimes direct control, such as directly controlling Astra for the pose the robot should adopt, is not the optimal tool to use. If this is a recurring task, you don't need Astra in a loop. Maybe you need to write code that can be executed repeatedly very
quickly. Or, if this is a task you've done a similar task before, you should be able to call a skill that has compiled and use that skill to complete the task faster and handle edge cases better. So here we see that it seems to be coming closer to this guy. Oh, very impressive. Yes, you can look, you can see that the latency is slow because we are limited by the Astra latency. But if you look at the trends, the latency of these models is improving very quickly. So, one thing we've seen is that for the Fable class LLM, their latency is improving by about half a month. Which is very, very fast. Very quickly. Yes. If trends continue, we could have real-time control by the end of the year. So, now that we've seen how this task went from start to finish,
why don't we figure out what it really meant? So here in the loop there is an encoding agent that executes it. What steps would an encoding agent take? And let's compare the direct encoding agent version and the water harness version in the loop version. Yes. So what just happened is that, uh, in a few moves, Astra got, uh, the images from the cameras and outputted the end effector pose that the robot was supposed to go into. And she does this over several turns to complete the task. This is less about the basic rules, but more about tool challenges. The on-screen part is quite repetitive. So you can automate a lot of this with code. Uh, and also actually approaching the bottle, lifting it. Like a lot of these things, uh, they're pretty, uh,
deterministic once you've done the task enough times. Uh, the interesting part is where you embed variation into the code as a policy graph. Uh, and so we have a few points of variation. This can occur, for example, when you detect an object. You may be using VLM, which is related to calling the tool. Uh, or when something goes wrong, like, how do you check that it went wrong? How do you then, uh, potentially do something different depending on failure? As in such more flexible answers, we tend to place the VLM inside the loop to ensure that, you know, even though the encoding graph is deterministic to some extent, there are points of variation that allow for generalization.
The biggest insight I think gained in the worldview of how to do machine learning and how to get the rest of the way to AGI came from many conversations with François Chollet. And in 2020, maybe even in 2018, when he was researching the measure of intelligence, he talked a lot about transduction versus just plain wrong versus programmatic induction. And transduction just means that I'm studying the theta function, which maps from x to y. And why is that wrong? It's just slow and inefficient from an information perspective. So to go from x to y, I need a lot of pairs of x and y. And if I have a very small number, then you're going to need a lot of inductive bias, and you're actually going to need a good generator to map x to y. So now my theta
dependency takes three n-n pairs of x and y and outputs a function that maps x to y. And that's what code is. And so, if you were to interview me about coding on a whiteboard, you would say, "Okay, here's a, uh, coding challenge, here are some examples. Okay, cool, write a function. And then we generate F, which will map X onto Y. And that's pretty interesting, is n't it? Because I think that was a significant part of the traditional program synthesis literature. And I feel like a lot of the reason why these methods didn't work so well was because of this inductive bias, as you said, right? You shift the complexity of the problem from finding this mapping to finding the right inductive biases to map it to this smaller code, to then perform the
mapping. But finding this set of inductive biases is extremely difficult. So maybe that's what Astra is buying us. Even in cognitive science, there's a giant shift in the literature from, you know, specific neurosymbolic methods to just using Astra to write code for display. And so maybe that's a cool way to think about it. Neurosymbolic. This is neurosymbolic. We have neurons, and they emit symbols, which are code. I think this is actually a pretty good transition to the last topic, which is around, um, inspired by this, um, article, also by Filippo Okola, um, which he calls the Platonic Representation Hypothesis, right? Where is Platonic because, um, it's a
reference to " Plato's Cave," and, you know, the idea that, you know, there you see shadows of, um, different realities. And I think here, um, the point he's making is that, um, there's a lot of evidence that, um, language models and different representations of different data actually learn these, um, distance mappings between, um, these kinds of things under different learning policies that, um, overlap. And so the idea is that as you train these systems on increasingly large amounts of data, they converge to, um, a kind of consistent representation of the world. And perhaps this means that we expect language models to get better and better over time. This would make them useful for new tasks, such as controlling robots. But maybe I.I
imagine that this representation hypothesis is actually quite central to all of your. guys' worldview or your view of your companies. So maybe some of you guys can tell me a little bit about what you think about this. And also maybe we can use this to make some assumptions about why we think models like Astra are much better at managing robots than previous models and where you know, make some predictions about where we might be headed from here. So, I think one way to put this into context for the discussion of robotics models versus language models is that very, very strong language models will have a very similar representation of the world to very strong robotics models. And if that's the case, then if you have a really strong, uh,
language model, you also have a really strong robotics model Uh, using, if we believe in the Platonic, uh, hypotheses of representation. And uh, if that's true, then the beta lesson is that this is the best manifestation of the beta lesson, right? Because you only need one really strong model regardless of architecture, and it will outperform any specific models, uh, that are a little weaker. It's interesting that this may not be entirely intuitive, and I would perhaps be interested in hearing your thoughts. That's why these newer models, particularly the Astra, seem to be much better at spatial intelligence. You know, we've all seen online
demonstrations of how they operate Blender and create beautiful 3D images. Now, it seems like, well, you know, U Show J in the benchmark, which is something like a significant improvement in the step function for certain tasks. Where do you think, before and after training these models, what changed in the open AI approach that made them so much better compared to models even 6 months ago? For example, we still benefit from a simple lesson, and we're still extremely good at coding, and we still solve math problems, but we haven't yet fully developed the spatial intelligence required here. I think Astra does an incredibly good job with visuals. It was probably pre-trained on a much larger amount of data used by the computer than ever before. So, it was pre-
trained on data such as CAD data. And all of this data probably teaches the model a similar, well, understanding of the physical world as my.a lot of robot data can. So, computer usage data is an interesting point because I think it's not really intuitive why computer usage data is useful for understanding the physical world, and what it's, you know, generally like a computer that you click on. Tell me more about why you think this is a big discovery, because I totally agree that they were learning from a lot more data about Right. Hmm, I immediately think that if you were just training a model to use a computer, it wouldn't be applicable to
robotics. But if you're inputting computer usage data into a large model like Astra, hmm, and by computer usage data I mean how you drag your cursor around the screen to rotate around some CAD object to design in Blender. This tells you, um, how to think about spaces. That at least tells you about top-down, left-to-right, all those concepts that you need to control a robot, right? Hmm, and I think that's why this data is so helpful in making these LLMs much better for robot use I think on the side, like a lot of people in the robotics community are also trying more and more different kinds of data, like self-centered videos.
Going from just teleoperation to, you know, a broader range of this data, because it's not just robot data that can teach a model to use a robot. So, if we take this to the extreme, why not incorporate all kinds of data, coding computer use, egocentrism into the same model? I think that's how we achieve the most capable agent, such as using a robot. Yeah, it's so funny that, you know, if you go back to Xerox Park in the 1980s or somewhere, we literally made the graphical user interface more like the physical world so that we could interact with it. And as a result, we created an environment that actually helped roboticists learn to use the physical world.
Right? We have file systems, we have files, we have folders, we have GUIs for SolidWorks and Autodesk to rotate things to make them a simulation similar to the physical world. And then we couldn't get robots. to work in the physical world, so we just learned about it. And now it works on a physical level. That's right. There were some incredible articles, I think from Princeton, as well as Cloud Robotics, that touch on this topic. We simply design the right system where the tools interacting with the robot look like the tools using the computer, for example, the agent drags a cursor to control where the robot moves, this improves the LLM's ability to perform these physical tasks.
Where do you think, given any reasonable assumptions about where the underlying models will continue to improve, and your own investment in evaluating those models or building systems around them, what do you think will be the opportunities that we might consider difficult to implement now, but that will become increasingly possible or even trivial within a few months? For example, 6 months ago, even the demos I saw on your Twitter, I think, would have been awesome if presented with an LLM. There's a certain consensus at Frontier Labs, and also at the. Robotics Foundation, uh, robotics companies, that we're going to have general purpose robots within the next 2 years or even sooner. And this is something that society
probably doesn't know about or is n't even ready for. And when we talk about general- purpose robots, we mean something like if you give it any instructions in natural language, it can do what a competent teenager could do with their bare hands. It's kind of like the GPT moment in chat, but for robotics in terms of capabilities, where it can generalize to invisible tasks and invisible environments. For you guys, uh, for the guys at Wild Labs, what does this mean for what you're creating? I think we will need to take steps to achieve this in about 2 years. I think there are so many issues that are very visible. For example, how many people on the Internet talk about delay. This is a big problem. If Astra just stays informed and thinks about
every step, it will be very slow. This will not be economically beneficial. So how do you, uh, you know, consolidate a kind of first pass of Astra, which just does contextual learning, into something like a faster skill or policy that you can.then run multiple times with incredibly high throughput the mappings for humans will become more and more similar, where the optimal might be to return a new SAR (state, action, reward) to the context, that's very fast, but then there has to be something like going to sleep for a while, like almost everything that is intelligent sleeps. For example, tell me an intelligent system that doesn't sleep, right? In a sense. And then during sleep, compression occurs. A strange thing that happens to your hippocampus, and
its short waves spread across both lobes, and there's a strange, you know, transition from memories that were compressed throughout the day to learning weights. And so, it might be right to do something like Dagger, the data aggregation system in our classic RL, where you collect a bunch of data and, and you reflect on it to some extent, and then you use it to update your weight file, and then you have this relation from those experiences back into the updated weight file, which may not be. Astra, but maybe it's your own models. Sounds a lot like Dream Coder. Mmm, ask. Mmm. But a lot of these specific tools, I think people have built before, like Dream Coder, right? He had this library of skills, and then during the sleep phase he
essentially refactored everything, uh, into a more compact representation, etc. I mean, we see something like this happening, just not as tightly as it used to be in the software realm, but now it's maybe how you want to refactor tracing. Maybe you want to, uh, refactor skills. A lot of this robotic code is like political documents, right? They have this growing library of skills, and it's a guess as to how you cut them down, organize them, and so on. So, yeah, uh, I think as the water moves forward, like how you manage the growing contact of skills, uh, deployment data, it's all going to be pretty interesting.
I think that with that in mind, I think it was a great discussion. Thank you so much Vincent, Hanmeng, and Jay for being here. I'm really. I'm excited about all the incredible advances in robotics that I think we're going to see over the next few years, and I think you're absolutely right, Jay, that I don't think the broader community fully realizes how much is ahead, and it's going to be a really incredible few years. Thank you very much.
