27:15
What if an AI could see into its own future? Not perfectly, no, just a little. DeepSeek did something absolutely amazing in their new research work. You see, when you ask an AI how to rewrite a difficult email, it answers you one word or one token at a time. That can get pretty slow. So, why not put out multiple words, like a whole sentence, at the same time? That would be much faster. And that would be a game-changer because it would boost many slow AI systems into usability, maybe one right on your phone in your pocket. Well, yes, except that it doesn't work. A big, smart AI is like a senior editor, brilliant but expensive. If you ask it to write five words one by one, it still has to think five times, slow,
expensive. So, what do you do instead? The solution is unexpectedly brilliant. What is that? Well, you hire a junior writer. The junior writer writes the next few words quickly. Of course, these words are not always accurate. So, you ask the senior editor to verify them. It looks and says, "Yes, yes, yes, no, no." And everything after the first no is thrown away. If the junior writer guessed well, we get our five words for cheap quickly. Hooray! We call this speculative decoding. But, not so fast. There is a problem. The writer is not that experienced. It messes up. For instance, it forgets things, it starts a phrase and finishes another. And I don't want this little writer to get fired.
So, here is DeepSeek's incredible new work called DeepSpark with three new tricks to make this relationship work a bit better. One, DeepSeek adds just a tiny bit of memory to the junior writer so it doesn't forget everything. With this, a drafted word can influence the next one. Doesn't need a huge brain. No, just enough for these few words not to fall apart. Two, you have to realize that some words are just doomed. If we ask, "Hey little AI, what planet do we live on?" And if the answer starts with lasagna, I think you can throw the rest away without looking. Although, mhm it would be great. A yum. So, here DeepSpark estimates which words are unlikely to survive and saves the senior
editor some time on these. And three, DeepSpark also asks, "Is checking this extra word worth precious GPU time right now?" Well, sometimes yes. If we talk about code or math, the next word may be very predictable. The writer is excellent there. But, mhm if the prompt is open-ended, like writing a funny wedding speech, then oof, [clears throat] there are a million good and bad answers. In those cases, the draft gets risky much faster. Maybe the first few words are fine, but the later ones are much more likely to fall apart. "Ladies and gentlemen, today we are gathered here to celebrate the ancient potato." No.
Just no. And DeepSpark predicts that in advance. That is amazing. So, what does all this magic get us? Now, hold on to your papers, fellow scholars, because they report a 60 to 85% speedup on their own flash and pro models. And this is measured against MTP-1, their old production multi-token prediction baseline. Wow. But, careful, fellow scholars, in the paper you also get a number that says 661% throughput. What about that? Now, that is not the normal everyday speed up. That only happens in corner cases where the old MTP-1 system is running out of room.
Just trying to help you avoid some pitfalls. In practical cases, we get 60 to 85% and that is amazing. And here is the best part. In principle, it can be implemented into many AI assistants, but not magically from the outside. No, you need a matching draft model, access to the target model's probabilities, and a serving system that can do this efficiently. So, once again, not a new smarter AI model. Therefore, you don't see many headlines about it. No, instead, this makes a busy brain even faster and we get all this for free. Open science and I think that is amazing. What a time to be alive. Now, not even this technique is perfect. One, it is not a magic switch you can put into any closed API. We just noted that.
But, it gives you an AI that kind of sees a bit into its future. And two, the gains depend on the workload. Code and math, great. Open-ended chat, not so much. This is where that lasagna strikes back. I use Lambda to reproduce AI research papers often in minutes. It's also great to train your own models or fine-tune an existing one. Run inference or text-to-image or video, easy-peasy. Running a deep seek chatbot or agent, super fast, super reliable. Lambda gives you powerful Nvidia GPUs to run your own experiments. I test ideas from the papers I cover and moments later, results. Love it. Seriously, try it out now at lambda.ai/papers.
