10:01
TechCrunch's premier podcast about the business of startups. I'm Rebecca Balon, and in this episode we invite industry experts to take a deep dive into current tech trends. Last week, outdated web tooling software nearly brought down a large portion of the internet. Banks, hospitals, retailers, and government websites were forced to go offline to patch two PHP vulnerabilities known as PHP 2 Shell. We talked about this a lot in Friday's edition, and our cybersecurity editor Zach Whittaker covered the topic in detail, but today we have someone who was at the very epicenter of the events.
Tomer Schwartz, CTO of Risk Management and Application Security at Wiz. Tomer, welcome to our studio. Thank you. Thank you for inviting me. I am very happy to talk to you. I apologize if my voice sounds a little sickly. I'm not feeling very well today, so you may hear me sniffling, but I apologize for that. So, to start: last Wednesday, in case anyone missed this, I don't know, maybe you were isolated from the world, the PHP Foundation released patches for two critical vulnerabilities in PHP. This is the language that WordPress, Drupal, Joomla, and MediaWiki run on. This is approximately 70% of the entire Internet, which depends on this technology.
It turned out that the bug allowed an attacker to remotely take over an unpatched server without a password or clicks, and security experts call this " unauthenticated remote code execution," which is almost a worst-case scenario. You know, we talked about this a lot. We have formed our own understanding of the situation, but I want to hear your opinion, Tomer, because you were at the very center of events. Can you clarify the situation? What of what has already been reported do people still not understand or Well, I think people generally get the point, and I'm proud that we were able to convey the importance of these findings. And, as you rightly noted,
this is a critical situation, isn't it? We are talking about a potential vulnerability that could allow an attacker to take complete control of the target system. If it is an external web server, it can use this to hijack the application, access data, other users' data, and accounts. This applies to all PHP deployments over the last 12 years. So, virtually every deployed server was affected until Tuesday or Wednesday when we released the patch. Another thing is that this applies to default configurations . So this is a very common setup, covering 70% of the web. That's why there was so much hype around the publication and the fact that people need to install patches. So did installing patches cause people to take down their websites? For example, there
wasn't a fix yet at the time, and so they had to disable them until it came out? Well, there are a lot of stories about those 48 hours before the release date, right? We had a very clear plan and schedule according to which we planned to release information about vulnerabilities along with patches. We also coordinated with many different vendors and open source projects to ensure they had releases available. But, unfortunately, information about the existence of the vulnerability was leaked. And at that point we had to decide what to do with the initial graph, given that if people know roughly where it is, they can suddenly expand their research, find similar vulnerabilities, or run AI models to look for the same thing. In the end, we
decided to accelerate the schedule, which meant that even though the PHP team was already working on it and had a great few weeks developing and testing these patches, it actually meant that some third-party projects that needed to create new releases didn't have enough time to get them ready for the new release date. So what actually happened was that a lot of different websites, government agencies, and companies were worried: once the patches were released, attackers would be able to reverse engineer and find vulnerabilities, which is what ultimately happened. And during that time, when they knew that some patch releases were not yet available, they decided to shut down those sites.
Yes, it's scary. Okay, so chronologically you already knew about this, right? So this Kalif is a research firm, right? It seems they found bugs and they were validated within the Open AI and Wiz initiative. You were planning to post something about this, weren't you? You had a schedule for releasing patches or updates, and then the information leaked onto social media, causing real chaos. Do I understand the situation correctly? Yes, many different people participated in this study. And I have to say, they all worked brilliantly in figuring this out. It's interesting to watch how different companies and groups of researchers find similar things in parallel. As far as I understand
the timeline, Calif found the first CVE and reported it to Open AI. They realized that they had found very similar variations of the same mistake. Together they reported this to PHP. Meanwhile, in parallel, we didn't even know what was happening on the other side, we were just exploring PHP. At the same time, we reported both CVEs to the PHP team, and it was the PHP team that ultimately decided to say, "Look, this is such a critical finding that it's going to take us a long time to figure out the best way to fix it." Why don't we all join the same Slack channel and start working on this together? I believe that this was absolutely the right
decision. We managed to do a great job together, and honestly, I was lucky to be in touch with the PHP, Open AI, and Wiz teams every day, working together on the patch. Yes, it must have been a high- adrenaline time for you. That's right. I want to clarify one point you made: that many different organizations found this bug at the same time. And you're claiming that this is because everyone is using the same AI tools to find vulnerabilities? Well, I can't be responsible for everyone, can I? But I can say that we use our own special system that we built ourselves. We call this project Atlas, and it is based on different AI models.
It uses the capabilities of DeepMind, Open AI, and Anthropic. And this actually allows us to scale a lot of different ideas and test different potential attack paths at the same time. This gives us a very broad opportunity to generate many research ideas and find many potential problems. After that, we try to select the most important ones and work on them. At the same time, I know something about this from our work with OpenAI and Calyx. They are doing very similar things, trying to use their own models and new ideas to scale research using AI. And in many cases they find very similar errors to the ones we see. But to what extent is bug-finding really the work of AI, and to what extent is it
enhanced human labor, right? For example, Calyx found bugs during a security audit, and our experts say that AI helped with the exploitation, but 90% of it was human work. Calyx's conclusion was that no single model is simply capable of detecting all errors, at least not yet. Well, I partially agree. There are still elements of art and science in this matter: how to route all these models and how to align them to find such critical errors. But actually, I think we could find this without AI. So the opportunity for the team of 10 of the most talented people who currently lead the Wiz vulnerability research team: if we got them all together and said, "Hey, maybe try to find some RCE bugs in PHP." Maybe it would take them a few months of
work, maybe even 12 months. And in the end, who knows? Maybe they would have found the same result. But the coolest thing is that we were able to gather all these people and say to them, "Hey, why don't you try 20 different ideas for different purposes , just run them on Atlas and find a bunch of potential vulnerabilities, maybe even in a weekend?" And by the time come back next week to see everything, we've already got a model running that's tested a lot of different options that a human researcher might not have found, right? It's pretty cool to realize that now we can, you know, multiply the work of a researcher by 100 or 1,000 times, and they can choose the best research ideas, the best solutions, and understand where it's worth digging deeper.
, thanks to the capabilities of modern AI models, there may already be a ready-made exploit that we just need to test and understand what the vulnerability is. Mhm. Yeah, I mean, it's great for defenders, but it's also great for attackers, right? And that's why we've seen such a strong push to fix this bug, as attackers are using AI for mass attacks to, you know, increase their own capabilities 100 times. Can you tell us a little more about Atlas? So, Atlas is your tool platform. You said you use OpenAI and Anthropic models. Are you using the latest versions of it, for example, Mythos? Do you already have access to it? What models do you use and Well, we
change them all the time, of course. I mean, given the rate at which AI models are getting better, we have to update them constantly. Furthermore, there are different stages at which certain models prove to be better than others. Can you give an example? Well, let's talk about binary exploitation vs. code understanding, shall we? These are very different tasks. If I take a very large codebase and look for, "Hey, what interactions are we seeing here between components and what do they suggest about how other components use them?" Maybe there's a model that's really good at processing large amounts of data and compressing it into a set of ideas that we can use later. But if all we need is to get to a certain part of the codebase
and figure out how to activate it remotely, then that's a whole different task. Therefore, we are constantly trying to evaluate different models. We actually have a public benchmark that we release. Fortunately, we also have access to unreleased models through collaboration with the laboratories we work with. So, we are trying to make the best combination of the best solutions to find the optimal approach to this problem. What share of this mix is allocated to open models? After all, I know a lot of people I've talked to, including Clem Delang from Hugging Face, who say that open models are great for cybersecurity because you don't run into any limitations when red
teaming or finding vulnerabilities, unlike some of the advanced models. Do you observe the same thing Well, open-source models are good and getting better, but honestly, we're lucky to be part of Google now, so we have access to pretty good models at unique prices, I would say. But beyond that, I think when you really think about scaling research at large scale, and maybe some of the startups are listening to us on the podcast right now or are thinking about it too. We have conducted many different experiments with open-scale models and find them to be extremely effective. We had several iterations when we used them. It's just that the current model that we use, the current toolkit that we apply, doesn't use any models with open weights, no.
We do not use any models with open scales. Interesting. Not now, but the situation may change in a month or two. It's quite interesting to watch how quickly the world of AI is developing. Yes, definitely. And because things move so fast, that became part of the problem with the rush to release a fix as soon as this bug was known. Did you learn anything from this? Did Wiz learn anything from this, because I know you had a timeline for when you wanted to publicly announce it. Have you learned anything about how to do it quietly and quickly? What was it like from the inside? Did this ruin your weekend? What happened? There are always lessons to be learned from this, and the coordination of
disclosure is something that, you know, is fundamentally flawed. We want to involve more and more people in a narrow circle to make sure that developers are aware of the vulnerability and have time to prepare. But on the other hand, we cannot announce this to the world without our enemies finding out about it. Um, and this is not a model we invented, obviously. This has been a problem in cybersecurity for decades. The question of how to do it well and how to do it in a very short time frame, um, I think, is a big question right now that the industry has to address given the imminent " wavepocalypse" that we're witnessing right now. So, I think it was a pretty quick
turnaround time compared to all the things that had to be done. From discovery to fix, I think it took about 2 months, which is pretty amazing considering the fact that it was a very unique architectural flaw in PHP that had been lurking there for 12 years, right? We had a very hard time finding a solution that was robust enough, covered every aspect of this unique vulnerability, and was compatible with some PHP extensions. There were many different constraints within which we had to work. And given the fact that we couldn't get a lot of people into that narrow circle, I think we got through it, you know, again, relatively pretty quickly. The other side of this is what happened at the end of that cycle, when we realized that this is something
that could literally " put down" the internet. So we started working with cloud providers, with web firewall providers, with many different vendors related to this, who have to be on top of it to make sure they release updates in a timely manner, have detection rules so that we have full visibility and potential, not full, but partial protection against what's going to happen. And when you realize how big this industry is, you realize that maybe that small circle suddenly becomes hundreds of people. And maybe they're talking to some of their colleagues because they have to involve them in order to be able to work. So I don't blame anyone for this leak. It's simply a function of how coordinated disclosure occurs. And, fortunately, we were
almost prepared for such a situation. We had a kind of plan "B" on which we could act. And I think we made the right decisions at every moment. So, to take a step back and explain coordinated disclosure for those who may not be familiar with it. So, researchers privately notify software developers, giving them time to prepare a fix before the vulnerability becomes public, right? So the PHP Foundation could give large companies and hosting providers a slight advantage in time. Maybe someone has already, you know, implemented the fix or started doing it before everyone knew about it. I don't know. So, the intention was not to give certain companies preferential treatment and say, " Hey, fix this because you're more important than others." Uh, that
was not our goal, and we made that very clear. I think what we meant was that for those nodes that have a huge impact on the part of the internet and industry that relies on PHP, we should at least make sure they're ready for release, right? So what I mean is: let's talk about how distribution works in open source great team that maintains the PHP codebase. A team of volunteers, right? Well, I can't say for sure. I think they're volunteers, but, you know, we should ask the PHP Foundation about how it works.
I think so, I think they are volunteers. But, essentially, we worked with them day and night, right? For weeks, they were on Slack at all hours of the day. Even on weekends, trying to move things forward and make sure we have a comprehensive solution. But when they finish their work, maybe a patch comes out that everyone is happy with, and it protects the PHP code base from such vulnerabilities, then they have to prepare an official release. But this will be a binary release on the php.net website. This does not necessarily mean that if you are using Ubuntu, Red Hat, or the standard AWS Linux image, you will automatically have a binary distribution with this
patch. So to get all these binary releases out at the same time, we have to give these teams some time to prepare and tell them, "Hey, here's the patch." Here's what you need to do a few days before. Make sure you have a plan and that the revised versions are available to your customers as well. And make sure you keep them to yourself until we release this at the designated time. Uh, now, at the same time, we realized that with this level of exposure, we want to, uh, reach out to, you know, some cloud providers, maybe some web firewall providers, to tell them, "Hey, here's a set of detection rules that we can work on together to see if attackers are exploiting this vulnerability right now." And
this was important to me for two reasons. First, you know, as soon as patches are released, as we discussed earlier, attackers can take them, reverse engineer them, figure out where the vulnerability is, and create an exploit. Another thing is that while working on this coordinated disclosure, we wanted to make sure that there were no attacks in the real environment. And it was critical for us to know if we needed to press the pedal even harder. Maybe we need to release patches without coordinating with anyone. Because if people are currently at risk, we want to make sure a fix is available. So we deployed what's called "silent" detection or monitoring, or simply getting a better level of visibility into what's
going on around us. And we've worked with many vendors who have this unique level of visibility to have threat data to make decisions as we go. Oho. Yes, it seems like this was a situation where everyone had to get to work. So, you know, Cloudflare said it had seen millions of exploit attempts. And Amazon says it saw the exploit about a week before the patches were released. I want to understand a little bit, who is still vulnerable, what still remains vulnerable? What work still needs to be done? But before we do that, I wanted to ask about one thing, because I read that the bugs were noticed in 2024, but they weren't considered
exploitable at the time. So what has changed? You know, they're over ten years old. So how can a mistake go unnoticed for so long? Well, there are many parts of many codebases that, you know, those who are very familiar with them might say, "Well, you know, this area is a little more sensitive than others, is n't it?" But there's a huge difference between a simple mistake, a quality control issue if you will, and something that can be exploited by attackers. These, these, these are completely different situations. I believe that even though information about similar problems appeared back in 2020 and even earlier. And
when we spoke to the PHP team, they said they see this particular area as somewhat problematic. No one could create an exploit for this, right? No one has reported the existence of a fully working exploit that allows these bugs to be remotely and securely activated in a way that attackers can exploit. I think what happened is this: again, it all comes down to AI. Now that we can spend not only our time and energy, but also countless tokens and cognitive effort, we can switch between different ideas, which gives us the opportunity to scale and try to create exploits in a completely different way. So I think what's
happened is that we, as defenders, suddenly have such great tools to try all this and close the basic issues in all the potentially vulnerable areas that we can discover a whole bunch of new vulnerabilities, right? And that's why we keep saying that this isn't the only critical vulnerability that will be published in the coming months, right? I think the industry has gotten used to an annual cycle where we have one major headline a year about the most critical vulnerability found during that period. I think this pace is changing very quickly. Yeah, I mean, we've heard about it, right? And this feels like a wake-up call about how safe the internet can be. Like, what happens next? This is certainly the first of many. Is there anything that, you
know, well, before I ask, what can we do? Regarding PHP in particular, who still hasn't installed the patches? You know, Zach was saying that small IT teams, lone developers, anything that uses PHP, like routers, storage devices, firewalls, development tools , are typically still not up to date. Then there are legacy systems that cannot be upgraded. So, do you know who is at risk? Who should take a closer look Well, anyone who hasn't installed the update in the last few days is currently at risk. We monitor this situation daily both among Wiz customers and through broader internet scanning. We see that most workloads are still not updated. It takes a lot of time to completely update a live website: understanding the correct way to do it, identifying the owner,
and making sure all releases are actually installed. I worry that we will see this "tail" of non-updated systems for quite some time. So, anyone who hasn't updated, any site that doesn't have updates. There are many small businesses that use WordPress, Wix, GoDaddy, or other hosting platforms to build their own websites. Is this a problem only for PHP-based sites? Can these people rely on these companies, hosting platforms, and website builders to fix this for them? There is, for example, Replit, which creates a bunch of sites for people, Lovable, etc. I wonder if every person with their own small
website should panic, or can they trust the businesses that help them build these sites? I think these big companies are well aware of this and have various ways to determine if their customers are currently using PHP. So you've seen a lot of notifications about this. I believe that these centralized businesses are taking this very responsibly and acting proactively. In many cases, they can install the updates themselves, behind the scenes. If not, they notify their customers to warn them. But I encourage you to think about all those small businesses that may have built a WordPress site in the past and may not have used these more centralized platforms. I think they are highly likely to be vulnerable. Another thing: if you're not sure, just ask. You
can contact the platform you are using. You can even contact us at php2shell.com. We try to put clear recommendations there with all the technical details. And if something isn't covered there, there's a way to contact us to see if we can help. Good. Since this will be an ongoing problem, unfortunately, such critical vulnerabilities will continue to be found, or at least we believe that they will. What is your advice for companies with 10 people and no security team? What should they do this week or this quarter? This is a valid question, as we must prepare for times when critical vulnerabilities are constantly being discovered. So I advise new startups and businesses to think about how to make their architecture as resilient to change as possible. If
we talk about businesses built before the AI era that have manual update cycles, I don't think they will scale given the number of vulnerabilities we face If similar vulnerabilities are discovered every month or even every week, we need to rethink our update strategy and entire IT architecture from scratch: can we shut down systems, can we update them automatically, is everything written in the code to run continuous deployment cycles and make changes silently-these are great practices, especially while you're still young. It's much harder to change something when you already have a thousand employees and a bunch of outdated systems. So I think that the small companies that are emerging now are in a pretty advantageous position. But we
need to think about this, not just hope that nothing will happen. Do you think that for early- stage startups, one of the first employees hired or even co-founders should be a security specialist? It's hard for me to say because I'm like that myself. But I think it depends on the business. I think there are areas where it really makes sense. If a significant part of your business is based on trust, then it makes sense to have such expertise on staff. If it's not the co-founder, then maybe one of the first hires should ensure you're building everything right. Otherwise, it will always be something that prevents you from accelerating and moving faster. So, if you don't think about it from the beginning, and let's say you're dealing with critical personal data
that is the foundation of your business, you should think about it from the very first days. Otherwise, again, you will find yourself in a situation where every following week will be like putting out a fire, and it will be very difficult for you to scale. Tomer, you did n't just work at Wiz, right? You had two startups of your own, right? Both were cybersecurity startups. So maybe you could briefly talk about what those companies were, how they ended up, and what advice you could give to the many cybersecurity startups emerging today. Yes, I've been through this before. I was the CTO and co-founder of Dazz, a company that Wiz acquired, and that's how I ended up here. Before that, I was the CTO and co-founder of Armis, which has
now been acquired by ServiceNow. I think the cybersecurity industry is very interesting for startups right now because the usual scenario we had before the advent of AI has completely changed. And it changed for the same reasons we talked about today, right? I think AI is changing everything, not just in cybersecurity, but in the underlying IT infrastructure, and also in how we perceive how organizations do a lot of different things. We talked about the upgrades, for example, but it also changes how they approach internal applications, and it changes the approach to how to use AI as a replacement for human labor. There are so many different changes happening right now that we don't yet have a clear plan on which to build cybersecurity solutions. So for
me, these are very interesting times to be working in the cybersecurity industry. I think for those startups that are already in existence, if you haven't already thought about how to update yourself with the changes that are happening, now is probably the best time to do so. Um, but one more thing for those who are planning to enter the cybersecurity industry as a startup: I think you should think about the changes that are happening and different ways to reach a broader market segment, and also think about how Certainly. Hmm, that's great advice. Well, we're almost out of time, but I wanted to ask again about that website- can you repeat where people can go if they have questions?
Yes, this is phptoshell.com. Quite simple. Okay, great. Um, where can our listeners contact you online? Either there, or on Twitter or LinkedIn, you can find me. I'm Tom Schwartz. It's a little hard to spell, but I'm sure that name is in the podcast description. Yes. Well, thank you again for joining our show. To our listeners: you can also find me on Twitter and LinkedIn. You can find Equity on Twitter and Threads under the nickname Equity Pod. See you next time. Equity hosts- TechCrunch senior reporters, producer-Theresa Lo Concannon, editor- Kell. Subscribe on YouTube or wherever you listen to podcasts, and find out about upcoming events at techcrunch.com/events. Thank you
for listening, see you next time.
