Kioxia CM9 Enterprise Storage Meets Nvidia DGX Spark in a Custom Build

Kioxia CM9 Enterprise Storage Meets Nvidia DGX Spark in a Custom Build

A look at the 16TB Kioxia CM9, cutting-edge enterprise storage that remains out of reach for most buyers due to its cost. The video explores pairing this drive with an Nvidia DGX Spark, which uses a smaller 2242 M.2 format, and shows how a 3D printed adapter helps bridge the gap between enterprise hardware and compact systems.

Supercharging Our Nvidia Spark With Kioxia Enterprise Storage! | Transcript:

This 16-TB CM9 from Kioxia. This is the cutting-edge, bleeding-edge in storage right now, enterprise storage. I know, even enterprises can't buy enterprise storage right now, unless you've got, you know, an eight- or a nine-figure IT budget, and then, yeah, it's no problem. But, innovation continues to happen, and production and things are ramping up at CM9, new generation, new performance. Have a full benchmark on it, that's great. 16-TB DGX Spark. This isn't even the right format, this is E3. Sparks have a 2242 M.2. What is going on? What's this 3D printed skirt?

Couple other things, but it is doable. You can adapt E3 storage to your DGX Spark. Let's take a closer look. If you're new here, welcome. A thing that I like to do on this channel is like PCIe lanes. Everything is just fundamentally PCIe lanes. This M.2, this is four PCIe lanes. So, all you need to do is adapt those four PCIe lanes into whatever you want to use. In our case, we're going to go from M.2 to E3. Fortunately, AliExpress has an adapter. You can order an adapter that's kind of cursed, but you can go from E3 to M.2, and that's exactly what we've done,

and there's a 3D printable file that uh Gigabuster has put together to let you slide this in, slide it in the side or not, but um yeah, the Kioxia is full performance on this platform, because of course it is. Now, it needs a little extra power. This can draw up to 25 W. DGX is not designed to do that. Our adapter board uh lets you supply power externally, so that's what I've done here with this uh this setup. And that works fine. Our E3 drive shows up just like it's a regular m.2 as far as the DGX OS is concerned. It is an m.2 just an absurdly large 16 terabyte m.2 that can sustain 15 gigabytes per second. That's part of our franking testing with the DGX Spark. It was never really meant

to do this. You'll get PCIe errors depending on the adapter that you use. And so these errors are a little bit worrying. Even though these errors say that they're corrected, they indicate that there's a cabling or signal integrity problem. In total for this project, we ended up testing both the CM9 and the older CM7. And after iterating through those issues with a signal integrity and everything else, our flagship CM9 was up and running in our DGX Spark. 16 terabyte DGX Spark? Oh, yeah. Uh the other warning or caution that I will give you is PCIe Gen 5 is really not tolerant of uh this kind of chicanery with long wires. You'll need re-drivers or re-timers or an adapter that includes

some electronics to clean things up. Now, I tried a couple of different ones from AliExpress and I have two that look identical. One that works and one that doesn't at PCIe Gen 5 speeds. Both of them work at PCIe Gen 4 speeds. Both of them work at PCIe Gen 3 speeds. But you want Gen 5, right? If you're going to do this, you're going to get something that's 15 gigabytes per second, probably. The uh perhaps one of the dumbest design decisions for the DGX Spark is this 2242 format. 2280 is like 2280 m.2. 80 mm is what everybody's used to. But these are 2242 and I don't know why Nvidia would have chosen 2242. I don't even think it's reasonable that it's like, "Oh, this is too small and we need

a really small SSD." I don't even think that really flies. I don't even really like the thermal solution for this. It's a big thick thermal pad that dumps a bunch of heat into the PCB through some surface mount components. Like it doesn't seem like a super amazing optimized design in uh in the original DGX Spark. It's like a 4 terabyte 2280 m.2 even before the shortage crunch was still absurdly expensive and a 2242 version of that, a 4 terabyte m.2, is even more expensive than that. And now it's basically unobtainium because it's so crazy expensive. But you really do need a lot of local storage if you're using one of these with a bunch of models and you're crunching numbers and you're actually

trying to do fine-tuning or something like that. And yeah, you've got the 200 gigabit QSFP interface in the back, which theoretically could do 10 to 20 GB/s through the network interface there through the InfiniBand stuff. Um and so you could use that for both your storage network as well as your inference network, especially if you get two of these connected like I suppose. But I know that Nvidia and like their Thor and uh past Jetson stuff have experimented with actual PCIe slots, so why? Why make us suffer needlessly? All right, so this is our other adapter. This is m.2 to a full-size x16. That's It's x16 physical, but it's only x4 electrical. And the problem with PCIe Gen 5 is this length

of wire. PCIe Gen 5 is extremely unforgiving for this length of wire. And almost anything that you plug into this slot is almost certainly guaranteed to get PCIe AER errors or PCIe errors in general. This is a PLX bridge card. And yeah, I know Broadcom has acquired PLX a long time ago, but Broadcom still uses PLX in the chipset. So stop being a terrible person and being pedantic. PLX isn't a company. Yeah, it's not. It's a chipset number. Except for when it's PEX. Sometimes it's PX PEX except for PLX. And the answer is I don't recommend this. This is version 6.6. This is a 16 CZ version 6.6 JAHL JAH link. This still doesn't work. Gen 4

or Gen 5. The wire is just too long. And it's probably because the signal-to-noise ratio is tuned for this m.2 slot to be as little as possible. So I did a lot of experimenting with PCIe bridges as well. So you know, Broadcom chipset there's not necessarily Broadcom chipset. This is not a re-driver or re-timer. This is a literal like bridge chipset. And it seems like it wants to work in a couple of scenarios, but mostly it doesn't work reliably. Not for the sort of fun things that you would recommend. It is very cursed. So, if you're going to do this, understand that it is a very cursed setup. So, I have here my M.2 to X16 adapter, which is only X4 electrical. And then I have the Broadcom

uh PLX chipset bridge. This is a PCIe Gen4 bridge. And then I've got it, you know, a breakout cables to PCIe slots. And then I've got a you know, another Kioxia CM7 SSD attached. But, signal integrity here, little bit of a problem. And what exactly the Spark supports in terms of PCIe bridges is a little nebulous. You've only got these four PCIe lanes to work with. It's definitely not fabulous. It was never designed to do this. I was actually kind of surprised that I could get 16 TB E3s to work. And there's there's a power constraint there. And you can run these actually at a lower speed. Like, you can use some programming commands to tell

these to run at a lower wattage um for reduced performance. But, I also wouldn't recommend that. Just, you know, set up an external power supply and you're basically good to go as far as usage and everything else goes. But, yeah. This was a fun experiment. I just wanted to fart around and you know, here we are. This is what we do at the forum at Level One Techs. In case you were wondering, could you connect a PCIe bridge with more PCIe slots and have it be enumerated correctly by the platform? The answer is about 70% yes. It doesn't work in every scenario, but it does work in a lot of scenarios, especially if you use a Broadcom chipset.

Your mileage may vary. But, you probably shouldn't spend money on this. But, it is an absurd amount of fun adding a 16 TB drive to uh DGX Spark. I mean, what can I say? I'm Wendell I'm signing out and you can find me in the Level One forums.

More Tech Transcript