13:02
Often - I think about this a lot. I think YouTube spends too much time chasing other platforms and competition when they should just be doubling down on what makes YouTube, YouTube. So anybody paying attention has probably noticed by now there's been a lot of big changes to YouTube over the past couple years. Some user-facing, some creator-facing, and some of them are great, some of them not so much. But this latest one, in my opinion, is their craziest, riskiest, and possibly worst idea yet. So I'd like to explain why straight to you, YouTube. Because this past week YouTube announced that it would be rolling out a feature called Video A/B testing. And look, they talked about a lot of stuff.
They had a whole Made on YouTube event that they've held annually for a few years, and it's great! They tell us about a bunch of the changes coming to the site proactively. They revealed some other stuff like custom homepages, and new live stream features, and AI Shorts editing tools. Some features I may not ever use, some features I might. But this video A/B testing, I just think is a straight up bad idea. Now, you've been able to A/B test thumbnails and titles for a little while now. Actually, that started last year. So YouTube shows one thumbnail to one part of the audience, and the other thumbnail to the other part to see which one performs better,
and then lets you pick the winner. And, you know, it took me a little while to warm up to that, but I've done a handful of tests since then and I've learned some things about what you guys like to click. You can also A/B test titles. I've tried that once or twice too. but A/B testing the video itself. seems like that's a little too far for me. But look, there's a lot of smart people at YouTube. This clearly made it through a lot of levels. It got all the way to the top. It got greenlit. So, let me just show you how it was presented.
Here is a classic problem we've all had. You shot two different opening hooks because you weren't sure which one was going to perform better. Now with Video A/B testing, you can just upload your cuts, Shorts or long-form. YouTube tests them with real viewers and helps you select the video cut that holds audience attention best. So it looks a lot like that thumbnail feature that we've had. But at the end it didn't show this. But at the end, once the test is over, it'll, it'll show a chart of the three different retention graphs from all three versions of the video, and let the uploader choose the winner based on which one holds
viewer attention the longest. So I saw this and immediately had a bunch of questions that were not answered by that. Like, first of all, as a viewer, if I'm watching a video, do I know if that video is part of an A/B test or not? Probably not. If I'm reading the comments on a video, how do I know which version of the upload they watched? And if someone's comment has a timestamp too, does that break if I'm watching the other version of that video? And how long after publish can creators be A/B testing a video? And if I send a video to someone, can I actually end up watching a different video than the one I watched? And how different are the tested videos allowed to be from each other?
Like, I think this, you know, "I shot two different cuts of the same intro and I'm not sure which one to use." I think that's like the most generous, innocent possible use of this feature, right? Sounds like probably how it was pitched. Like, MrBeast was probably like, "Hey, YouTube, I'm trying to get 500 million views on my next video here, like, this would be the perfect tool for retention maxing." But what I've actually found over the past year of thumbnail testing is that if I'm testing two different thumbnails that have a very small difference between them, like say, a slightly different size text or the arrow placement or something,
or even arrow versus no arrow, like, the results are usually not statistically far enough apart to actually matter. Turns out I don't really learn much from small differences. But when the differences are way bigger, like two completely different concepts for a thumbnail, that's when the performance differences are actually big enough to learn something. So now applying the same logic to this, you know, a small difference of like two or three shots in a cold open or an intro sequence, I don't think is going to make that big of a statistical difference. So how big of a difference are the videos allowed to have? In their own example in the presentation,
these videos are pretty drastically different lengths, if you ask me. 17 versus 19 minutes. And also, this has got to break timestamps, right? It has to. So I got on a call with YouTube engineers, to their credit, and I asked them all these questions. And now I have for you some of the answers. So first of all, it's sort of a not-so-secret that YouTube has had this video replacement ability behind the scenes for a little while now. And it's extremely limited use. Like, almost no one gets to use it. But if you have a contact with YouTube who will let you use it, you have to have a very specific reason for using it,
like one specific factual correction or something. The videos are otherwise identical, but you need one thing to be changed to upload a new file to the same URL. You might remember Apple uploaded some commercial a few years ago that had a glitch in it and everyone was talking about it, and then mysteriously the next day it was gone. That type of thing. The video you're using as a replacement has to be like the exact same length as the file it's replacing, minus that one change. Because changing a video is precious. So the main thing I learned from talking to YouTube engineers is that for rolling
out this video A/B testing to everyone early next year is that they plan on building essentially a semantic analysis model that can comb through the entirety of both video files that you upload and determine their similarity. And then only videos with a high enough similarity score according to that model, will be allowed to run as tests. So theoretically, completely different videos can't be run as a test. And I hope, I hope. this works. I don't know how confident I am in it yet, just because whenever I ask Gemini to look through YouTube videos for things, it's not very good at it.
It usually just defaults to the transcript. But at least it's something, right? There's at least some sort of attempt to. to limit how different the videos can be, as determined by this bot looking at the story of the video. And they'll also only allow A/B testing at the launch of a video. So you can't just go running through old videos, A/B testing and replacing them with different cuts. That's just gonna be for new videos for a limited time at their launch. But then the rest of my questions kinda don't really seem to have answers yet. Like, I'm about 110% sure that comments with timestamps will be confusing and kind of broken. And also the uploader will not know which version of the video the commenters are
talking about or which one they watch. The viewers won't know that they're watching a video being A/B tested. And it's also possible that people. might watch a cut of the video that loses the test, and so then they might come back to watch it later. and it's a different video. And I think YouTube will never really tell exactly how it works, but it'll always be a question to us, like how well the similarity test score thing actually performs. Like if, if one sentence in the video is different, but that one sentence. changes the conclusion of the video in some way, will that flag it? Like if, if I have two different cuts of the same video, but one of them is like out of order from the other,
does that change it? We just will kind of have to wait and see. I think this really comes down to is there's this sort of unspoken rule. of community. on YouTube. that like we're all. having the same. experience. together, we're all watching the same video. That is what makes it a cultural phenomenon as we all saw the same thing, even if it's slightly different packaging that got us there. So letting people put up two or three different versions of the same video at the same time just kind of fundamentally breaks that. It just doesn't feel right. But also, like I said at the beginning, I think this is part of a trend.
Like YouTube spends too much time in my opinion, like responding to things that their competitors are doing that they really don't. I don't think they should. Like years ago, getting rid of the public video dislike counter was framed as. protecting creators. But the reality is Facebook and Instagram and TikTok, they don't show dislikes either. So YouTube got rid of them too. Even though the like-to-dislike ratio was a uniquely useful YouTube thing. If you've been following the Waveform podcast, and surely by now you've heard us talk about the new way. that they're counting. views here on YouTube.
I don't know if everyone's heard about this, but they essentially now are counting views. much more liberally as literally the first frame of a video loads. that counts as a play. Even though before, YouTube had a fairly sophisticated way of trying to measure what a real view is and that made less views more valuable. That meant like we saw a million views on TikTok and we knew that wasn't the same thing as a million views on YouTube. We always knew that those things were different. but again, not anymore. And you know, there's all sorts of TV stuff that they're always responding to.
They seem obsessed with getting like late-night hosts on YouTube, all of this stuff. But you know, at the end of the day, I do think YouTube is still. gonna do this video A/B testing thing. So let me take this opportunity to talk to my fellow creators who are also looking at this feature. So, dear. YouTubers, so I, I don't think most of us will ever. need to use. this feature. In fact, I would go as far as saying most of us probably shouldn't. You know, this is being framed as a tool to help you improve your videos. But. I think for 99.9% of creators, well, to be literal, it's not actually helping you make better videos.
It's helping you find which version of your video has the best retention, which doesn't necessarily mean it's a better video. Hot take. And there are so many examples of creative decisions. that I have made and that so many others have made in the video creation process that are definitely not the best retention-optimized decisions. But they're fun. and creative. and interesting and new. and entertaining and characterful. So there's always going to be like the top 0.1% of channels that are like all about mass appeal. And that's fine. But last time I checked, a lot of what makes YouTube fun is the stuff that is.
more niche down, that is not mass appeal by definition. It's the cool, small hobby you found or it's the fun, unique thing that you just discover that only a few people are talking about that, that stuff, that's what makes it like fun. I think if a lot of those channels and videos suddenly became like super, you know, optimized retention-maxing and everything, like, those would get a lot less fun to watch. And so imagine if all the feedback anyone ever listened to was just to have the best retention. It's like, as you optimize. over and over again, you just. you start to trim out all of the other creative stuff that happens to not be the most efficient way to tell a story. So I guess what I'm trying to say is. the skill of being
a video creator on YouTube is. not always necessarily about, like, the maxed out, most efficient, optimized way to do everything, but it's to find creative ways to share something. or teach something or explore or explain something or review something. So I just think making. and finishing. and publishing a bunch of videos. is a better use of a creator's time than making one thing. And then, you know, a whole bunch of different drafts of titles. and thumbnails, and different cuts of the video to try to optimize for, like, the last. few percent, to squeeze out a few more views from the average view duration being longer. That's all.
I think just stand by your creative choices. You made this video, this is the video you chose to make. And now you get to share it, and you put it out, and you learn from it. And maybe some go well, some don't go well. You learn from the wins and the failures, and then you make the next thing, and then you learn from that, and then you make the next thing. That's the creative process. Have conviction in what you want to share with the world. But, you know, hey, maybe. maybe this is just the. old school.
YouTuber in me. Old man yells at cloud. It's very possible. I just. I feel like the. video A/B testing, though, is something a lot of us can agree we're not super interested in. But. let me know what you think. Either way. thanks for watching. Catch you in the next one. Peace.
