10:43
It's mania. It's absolute mania. Okay. Your local AI builds a playable Breakout clone. Great. Whatever. You compress the model, it runs faster, it's running at three bit you know, three bit quants, whatever, it's still running and you still build Breakout. Did you lose anything important? Well, we built the tools to investigate that because we are in peak AI mania right now. we got to quantizing It did not all run according to plan when we put the tool together. I really want to dial back the mania and I'm glad you got something useful out of a three bit quant, you know, building a Breakout clone, but that's no way to go through life.
Let's discuss. Depending on who you ask, AI is either the beginning of a new industrial revolution or tulip mania so extreme that it may cause a planetary recession. Those possibilities are extreme and they aren't mutually exclusive. A companies it can be wildly overvalued and their promises can be ridiculous. You know, meanwhile, many of you I'm I'm just over here trying to eke out a living trying to use the new cool thing to solve problems or help build you know, something that I couldn't before, you know, the computer janitor stuff. That's that's sort of what I'm working on. That's my interest here. That's my interest to build tools that free
builders to be able to build. And if your personal growth and discovery journey involves building a Breakout clone, then more power to you. Let's get it done. I'll be glad to help you. It's going to be awesome. So, local AI always goes to tokens per second and benchmarking when we look at it. It's like, "I want to run this thing. I'm going to run it on my 3090." Okay, that's cool. Those measurements do have their place and I want to run faster, too. I want it to run really, really fast locally. One of my own tests is actually asking the model to a Breakout clone. And now even heavily quantized models can do it. One, two, three bits, you know, paddle ball, bricks, it works. You can add a twist to it, a lot of fun. One of
my prompts is over 6,000 tokens long, but it doesn't really give you a lot of insight into how well the model followed the constraints, how it was working, you know, the harness that the AI is in is doing a lot of the heavy lifting in terms of making the AI have a good outcome for the task. It really doesn't tell us how much confidence we should have in the whole thing because the harness is pushing the AI to build how much of it was really the AI, how much of that outcome um was the model being smart. It's all a little bit too hand-wavy. And this is where I want more qualitative analysis. We need to understand what's happening under the hood or at least have better visibility into what the quantization is doing
because what you and I are experiencing with our own hardware is way different than these insanely expensive hardware in the cloud. So, behold, PowerTool. It's loaded with the Quinception data set and you can play with this right now. Check out the thread on the Level One Techs forum. You can jump wildly ahead. You don't have to listen to me ramble. All you're looking at is a recorded session well, multiple recorded sessions with LLMs and different configurations of the model um with predictions at the same point. Uh Quinn is the model in this case. Now, the storage requirements for this are enormous because you're saving
everything at every step. And this would not have been possible had Kioxia not loaned me a bunch of flash storage. PowerTool is the first step. It gives us a way to examine the black box at a point where numerical differences become behavioral differences. Point-blank at the logic level. Now, if you are thinking, "Oh, of course the AI it running in different scenarios is it's random. It's going to give you different answers. For this comparison, we're choosing the highest scoring next token rather than randomly sampling from the possibilities, so it's different. And each configuration gets exactly the same recorded text leading up to that decision. Um so, you know, choosing the highest score
doesn't guarantee that different hardware or even slightly different software will calculate identical scores. I mean, floating-point arithmetic has finite precision, and even just changing the order of uh addition operand or like what you just doing floating-point addition um can alter the result because of rounding. Uh so, if you have something that's close, maybe different ones win. That's one of the possible source of divergence. Implementation bugs are another possible source of divergence. So, AI is random it's kind of a naive statement there. Where those predictions diverge in PowerTool, remember, this is predicted next token after all, you can inspect the captured continuations and see what happens next in this
visualization tool. And some of what is going on here is very, very surprising. Qualitative analysis, remember. Understand what we're doing is capturing not just the divergence, but also the hundred or so tokens following the divergence. Uh we've captured uh results on the GB300, the MSI WS300 workstation that I was able to borrow. Check that out. Check out that video. It's the very latest Blackwell Ultra hardware from Nvidia's data center family, as well as RTX Blackwell the RTX Pro 6000, which is called SM120, and DGX Spark, which, you know, GB10, and that's called SM121. Those Blackwell labels cover a lot of different hardware uh and hardware capabilities and software execution
paths. Now, there are some details in the forum thread, as you might imagine, but in the FP4 execution path was wildly different on GB10, DGX Spark. Oh, and there's a one result in the forum write-up I specially want you to look at. On the GB300, our mixed NVFP4 checkpoint changed more next to token choices than the other weight formats we tested. But, in a deliberately selected set of difficult continuations, the ordinary FP8 cache configuration broke more tool call structures. So, FP8 for your KV cache, maybe don't do that. We had six selected starting points where BF16 reference produced structurally valid output. With the FP8 cache, five became structurally invalid, and with mixed NVFP4 weights,
one became invalid and two were still incomplete when we reached that 100 token divergence limit. Those are selected problem cases, not failure rates for general use, of course. And incomplete doesn't necessarily mean broken, but this is why counting only the changed tokens is not enough. We also have to inspect what the changes actually did, cuz, you know, at the end, you could end up with a working breakout clone, but I want to sort of look at things as they start to go wrong. And even before all that, one of the divergences we encountered involved running across one GPU versus two. Tensor parallelism one versus tensor parallelism two resulted in different outcomes. We changed nothing. We changed nothing at all with the configuration, but how the
workload was divided up, and that resulted in a divergence. Now, is this an NCCL bug, or is it just unlucky rounding behavior? It's probably the latter, but the way because of the way the floating point works in general, but you don't know, and it is a divergence. But I just want to show you how to look at this for yourself. I want you to be aware of it, so that you can ask for yourself. You are the builder. If you've made it this far in the video, you can be the builder with these tools, because, you know, you're listening and you're thinking about these kinds of things.
Your mind is in the right spot. If we're going to use these systems to build things that are load-bearing and you don't need janitorial cleanup after the mess it creates, we are the people that demand the clarity and understanding. We We are the people that use phrases like this falls apart under scrutiny. These are the thought processes to fight hallucinations and rein in the mania, I think. And that's why it's so critically important. It's inherently extremely unfair that home users, I mean, you don't not really home, but what people that have access to this crazy enterprise hardware are third-class citizens in the quality game. And we need better visibility into the quality aspects so that we can fix it ourselves
cuz no one else is going to fix it for us. It's going to be up to us. We got to do it. So, here's what I want us to demand alongside the tokens per second benchmarks. A reproducible comparison chart uh against a stated reference. Tell us what the checkpoint was. Tell us the hardware. Tell us what the software and the kernels and whatever actually ran on that. Show us the distribution drift, the KLD top one disagreement under a documented test procedure. Um then show us the representative continuations and what happened when the resulting code or tool call was tested. That applies to quantized weights, inference engines, performance optimizations.
It's faster is not it's not an evaluation. It's faster at what cost to the quality? May maybe none. Maybe you figured it out. Like NVFB before, as much as it cuts, still maintains surprisingly high quality, but it's also a madhouse of divergence in our testing here. So, PowerTool gives us some more to start. Open it. Follow a divergence and bring your own expertise. Tell us what you're working on and whether the changes were merely wording or broken tool call or something else that we need to be tracking and thinking about it. If we want AI to be load-bearing, this is part of the work. I mean, sure the harness the AI is using can give us guardrails and help fix around some of the problems. I mean, Nvidia has said that
in their own press stuff. But, some of these problems arise from a quantized model and harness is not the only place to fix that. And it may not even be the best place to fix that when these problems arise from the quant. So, check out the forum thread and what's going on in the community and power tool and change the way that you think about this because quality really does matter. I'm Level this Level One. I'm going to go hang out in the forum. If you want to chat or I missed anything and keep trying to work on this. I'm signing out and I'll see you there.
