3:14
Hello. How are you? This is CS50 and this is our class on artificial intelligence or AI, especially for all the family members who are joining us in the audience for the first time. In fact, for the students, how about a round of applause for all the family members who came to join you today? Very good. It's great to see everyone and, as CS50 students already know, it's a tradition in programming circles to have a rubber ducky on your desk. In fact, a few weeks ago we gave one to each CS50 student, and the idea is to have someone or something to talk to when there is a mistake in the code or when you are confused trying to solve a problem. And the idea is that, in the absence of a friend, family member, or
teaching assistant to ask, you must put into words your confusion and doubts about that inanimate object on your desk. And in that process of verbalizing your own confusion and explaining it to yourself, the proverbial lightbulb often turns on in your head. And that's it, problem solved. Now, as CS50 students also know, we have virtualized that rubber ducky in recent years, most recently in the form of this guy here. So in the CS50 students' programming environment, a tool called Visual Studio Code available at CS50.dev, they have a virtual rubber ducky at their disposal at all times. Initially, the first version of this rubber ducky was a chat window that looked like this. And if the students had any questions, they could simply type
something like, "I hope you can help me solve a problem," in the chat window. For several years, all the CS50 duck did was respond with one, two, or three quacks. We have empirical evidence that suggests that this alone was enough to answer the students' questions, because it was precisely when writing their questions that they realized it. Ah, I'm doing something silly. And they discovered the answer for themselves. But of course, now that we live in the era of ChatGPT, Claude, Gemini and all these other AI-based tools, perhaps it wasn't a surprise that in 2023 this same duck started responding to students in English. And that is now the tool they have available, which in practice is intended to be a
less assistance-oriented version of ChatGPT, one that does not reveal the answers directly, but tries to guide users towards solutions, just as a good teacher or tutor would. Therefore, today's class is about exactly that and the fundamentals that make the existence of that rubber ducky and all the AI with which we are becoming increasingly familiar possible. That is, generative artificial intelligence, which uses this technology known as AI to generate something, whether it be images , sounds, videos or texts. And actually, what we're thinking of doing to involve everyone from the beginning is: if you have a cell phone nearby, would you like to scan this QR code here? This will lead them to a voting session, where they can submit their
answers. CS50 mentor Kelly will kindly join me here on stage to help me operate the keyboard. And what we're going to do now is play a little game to see how good we humans are at distinguishing AI from reality. Therefore, we are going to use some data from The New York Times, which a few years ago published some examples of what AI is and what it is not. And we're going to see exactly how good this technology has become. So, here we have two photos on the screen. At one point, your phone will ask you if you were able to scan that code. Which one of these is AI? The one on the left or the one on the right? So, I hope that here on your phone, if you want to move forward and swipe to the next screen, let's activate the
survey here. At some point, you should see a message on your cell phone inviting you to select left or right. And feel free to raise your hand if you're not seeing this. But it seems the answers are coming in and, at the risk of giving away a spoiler, it seems that more than 70% of you believe the answer is on the right. And Kelly, maybe we can go back to the two photos. In this specific case, yes, it was indeed the one on the right. Maybe it seemed a little too good, or maybe a little too unreal. Maybe. Let's see, perhaps, some more examples. So, the same QR code. There is no need to scan again. Let's open these two examples now. Two photos, same question. Which of these is AI?
Left or right? Left or not? Very good. Let's take a look at the chart to see what responses are coming in. In this case, the difference was slightly smaller, but most of you believe the answer is indeed on the left, although 5% of you sincerely admit you are not sure. But Kelly, if you want to go back to the photos, the answer this time was actually a trick question. Actually, both were AI, which perhaps demonstrates how good this technology is becoming. None of these faces exist in the real world. They were synthesized from a lot of training data. So, two photographs that look like human beings, but that don't actually exist. How about one more, this time focused on the text, which will, of course, be the
underlying focus of our duck? Was this written by a fourth-grade student or by the new chatbot? Here are two final examples. It's the same code as before, so there's no need to take another look. And here are the texts. Essay one. I like to bring a tasty sandwich and a cold juice for lunch. And sometimes I even bring a tasty piece of fruit or a packet of crunchy potato chips. While we eat, we chat, laugh, and catch up on each other's day. Period, period. Spot. Essay two, my mom prepares me a sandwich, a drink, a fruit and a snack. When I enter the cafeteria, I find an empty table, sit down, and eat my lunch. My friends come and sit with me.
Period, period, period. The question now is finally: Which of these is AI? One or two? Rehearsal one or two? The bars here are fighting each other. It seems that most of you chose essay one. Let's return to the text. And one of you who is here, one of you who said rehearsal one. Because? If anyone wants to raise their hand quickly, because that's what rehearsal one is about. OK. So, essay two is more like what you would write. May I ask what grade you're in? A fifth- grade student? So, is this a new fifth-grade student or not? The answer here, actually, is that essay one represents AI, since essay two is more like what a fourth-grade student would be. or, if I may, I would write in fifth grade. And I dare say there may be some
revealing signs. I'm not sure if a typical fourth or fifth grade student would understand what the classmates were saying with the slang we see in the essay; Suffice it to say that this game is not something we can play in the coming years, because it will become too difficult to understand, whether it's generated by AI or not. Therefore, among our goals for today is to really give you a better idea not only of how technologies like this duck and these games we play here with images and text work, but also what the underlying principles of artificial intelligence are, which, frankly, have been with us for decades, have been developing for a long time, and have really reached their peak in recent years thanks to
advances in research, thanks to the growing cloud computing, thanks to the increase in memory, disk space, and information. The gigantic volume of data that we have at our disposal and that can be used to train all these technologies presented here. So, this duck is built on a rather complicated architecture that looks more or less like this, where here is a student using one of the CS50 tools. Here's a site CS50 students are familiar with called CS50.ai, where we, the team, write a lot of code that actually communicates with what we call APIs (application programming interfaces), third-party services from companies like Microsoft and OpenAI, who
have really done the heavy lifting of developing these models, as well as some local ingredients, some special touch that we at CS50 add to the mix to make it specific: the Duck's responses to CS50 itself. But what we've essentially been doing is, um, something you may be partially familiar with. Prompt engineering has begun to appear, for better or for worse, on, um, LinkedIn profiles everywhere. And prompt engineering isn't really so much a form of engineering, but rather a way of asking good questions and being detailed in them, providing context for the underlying AI, so that the answer is highly likely to be exactly what you want to get.
Therefore, there are two terms in this world of prompt engineering that are worth knowing. Therefore, CS50 used both concepts to implement this duck. We, for example, write what we call human-written prompts, usually in English, which in a way guide the underlying AI technology to have a certain personality or a specific domain of specialization. For example, in CS50 we wrote a system prompt that basically looks like this. Actually, nowadays it has many lines, but the essence is this: you are a friendly and helpful CS50 teaching assistant. You too are a rubber ducky, and that's enough to transform an AI into a rubber ducky. It turns out the answer is: only answer students' questions about CS50 in the area of computer science. Do not
answer questions about unrelated topics. Do not provide complete answers to the exercise sets, as this would violate academic honesty. Answer this question with two points. And after this preamble, if you prefer, also known as the prompt. Basically, we copy and paste any question that a student has written, also known as a prompt book. And that is why, in our case, the duck behaves like a duck and not like a cat, a dog or a PhD, but as something adapted to the specific objectives we have from a pedagogical point of view in this course. In fact, those of you who are CS50 students will remember that a few weeks ago, in week zero, when we first introduced the course to the class, we created some code that same day that ended up
looking something like this. And I'm going to explain it briefly, line by line. But now, having just studied a bit of Python in CS50, this code that I quickly created in the first class, perhaps makes a little more sense. In that first class, we imported OpenAI's own library, code that an external company wrote to allow us to implement code on top of theirs. We created a variable called client in week zero and that gave us access to the OpenAI client, which is software they developed for us. Then, in week zero, we defined a user prompt that arose from the user using the input function that CS50 students are already familiar with. And then we defined that system prompt that day when I said, "Limit your response to one
sentence, pretend you're a cat. I think that was the personality of the day." And then we use a slightly more complex code here. But essentially we created a variable called response, which was intended to represent the response from the OpenAI server. We use the client to create, which is a function or method that OpenAI provides and that allows us to pass three arguments: the user input, that is, the user prompt; our instructions, that is, the system prompt; and finally, the specific model or version of AI that we wanted to use. And the last thing we did that day was print response. Output text. And that's how we were able to answer questions like: "What is CS50 or similar?". We've seen all this before, but that week we didn't discuss exactly how it
worked, or what else we could actually do with it. And in fact, what I thought we would do today is unveil a layer that we haven't addressed in the course so far. And, in fact, they still can't use this resource until the end of the class. In CS50, when they reach the final projects, they are welcome and encouraged to use VS Code in this specific way. So here's VS Code again. For those who are not familiar, this is the programming environment we use here with the students. And let me open a snippet of code that was assigned to the students a few weeks ago, namely a spell checker that they had to implement in C. So I came ahead of time with a folder called Speller and inside that folder I had some code that day and all the
students had that week called dictionary.c. In this file, which may not be familiar to many of you if you have not yet completed weeks 0 to 7, we have included some spaces reserved for students. In summary, the students had to answer some questions, that is, write code to perform this task. This task, this task and one more. There were four functions or blank spaces that the students needed to fill in with code, and I dare say it took most students 5 hours, 10 hours , 15 hours, something in that wide range. Let me show you now how, using AI, you aspiring programmers will soon be able to start writing code much faster, not just by choosing a different language, but by using these AI-based technologies
that go beyond Duck itself. So what I did here on the right side of VS Code was to activate a feature that CS50 disables for all students from the start of the course called Copilots. This is very similar in essence to the products of Google, Antropic, and other companies as well. But this one comes from Microsoft and, in turn, from GitHub. It also offers a sort of chat window here, and that's just one of its features. For example, if I wanted to implement the verification function to begin with, I could simply ask it to do so. Implementation. What if we use a hash table in C? I'm going to press enter. Now it's going to work. It is using as a reference, that is, as context, the very file I opened,
which is dictionary.c. Here, the Copilot, in general, like many AI tools, is familiar with CS50 itself, because it has been freely available as open educational material for years. What you see here is essentially the reasoning process, although this is a bit exaggerated. Actually, this isn't thinking. Try to identify patterns in the problem I want to solve using all the training data you've seen before and come up with a pretty good answer. So, for today's purposes, I'm going to ignore the GPT chat-style explanation of what to do that appeared on the right. But what is more interesting to observe here is on the left. If I scroll down now, what's highlighted in green is all the suggested code for implementing this
verification feature here. Well, maybe it's not exactly the way you implemented it yourself, but I dare say this gives clues to exactly what you probably did when it came to implementing a hash, a hash table. And I can actually keep all this code if I like the result. Let's assume everything is correct here. You may now want to implement the loading function. So, what if we implement the load function now? Enter. It's that simple. And what data is being used? Well, some different things, says one reference. Therefore, it is in fact using this file. But there are also what we call comments in the code, which all students are already familiar with. Those gray commands give clues in English about what
this function should do. There is implicit information about what the inputs to these functions, also known as arguments, should be, and what the outputs should be. Therefore, the underlying AI called copilot, in this case, presents a reasonable number of accurate results. Clues and, like a good teaching assistant or a good software engineer, that context is already enough to figure out how to fill those gaps. And so here too, if I scroll down now, we'll see in green some code suggestions that could also solve this same problem, the load function. And I dare say I spoke for far fewer minutes than the CS50 students spent programming the solution from scratch for this problem here. Then I'm going to click on keep. I'm going to
start from the premise that it's correct, but that's actually quite a bold assumption. And for those wondering why we're learning all this if I could just ask him to do my homework for me in English? I mean, there's a lot to say about motor memory that I hope you feel you've been developing over the past few weeks. The reality is that, if you don't have a trained eye for what you see, there's no way to solve a problem here, explain it to someone else, make specific adjustments, or anything like that. And yet, what is incredibly exciting even for someone like me, for the entire team, and for my friends in the industry, is that this kind of AI functionality amplifies your capabilities as a
programmer practically overnight. Once you have that vocabulary, that motor memory, to do it on your own, the AI can simply take over from there and eliminate all the tedious work, allowing you to focus on the whiteboard with your colleagues on broader problems you want to solve and let the AI work out the details for you. Another fun exercise could be to go back at the end of the semester and try to solve any of the course assignments. For example, I'll try to do this here in my terminal window. I'm going back to my main directory. I'm going to create an empty file called Mario.c. There's nothing special about this. And I'm going to proceed here in my chat window and say: "Please implement a C program that prints a
left-aligned brick pyramid, using cat symbols for the bricks and using the CS50 library to prompt the user for a non- negative height as an integer." The end. I dare say this is essentially the English description of what exercise set one was for this year's CS50. Implement a program called Mario. C. This is also somehow doing what it's supposed to do. Use only one reference. It's working. He knows, as a clue, that this file is called Mario. C has already seen many of these building blocks in his training data over time.
There is an explanation in English about what I should do, and the CS50 students present in the room will probably recognize here the kind of basic structure that uses a do-while loop to prompt the user to enter a height using the CS50 library that is already included. Print a left-aligned pyramid using some kind of loop and that's it, you're done. And these are relatively simple problems. As you will see, by familiarizing yourselves with the final project, which is a fairly open opportunity to apply your new programming knowledge and skills to a problem of your interest, this will allow you to implement much larger projects, much bigger than was possible until now, certainly in the few weeks we have to do it, thanks to this expansion of
your own skills. So, with that promise in mind, let's talk about how the heck all this actually works. Of course, I just generated a lot of stuff. And that's how we began the story with the generation of these images and these two essays written by children. But what is generative artificial intelligence, or indeed, what is AI itself? And these are some of the fundamental building blocks that will not disappear anytime soon and that in fact led us, as an evolution, to the capabilities you have just seen. Regarding spam, nowadays we consider it almost natural that in our Gmail or Outlook inboxes, most of the spam ends up going into a folder. Well, there isn't a human at Microsoft or Google manually labeling
messages as they arrive to decide whether they are spam or not. They are discovering it using code and, nowadays, using AI that detects what looks like spam. And so, I'll put it in the spam folder, which is probably correct 99% of the time, but there is potentially a failure rate. Other applications may include handwriting recognition. Certainly, Microsoft and Google don't know the handwriting style of everyone here in this room, but they were trained on enough other human beings' handwriting styles that it's likely your handwriting and mine look similar to someone else's. And so, with a very high probability, they could recognize something like Hello World here, as in fact this very digital text. We all use streaming services these days,
like Netflix and similar ones. Well, they're getting really good at knowing that if I watch X, maybe I'll like X too. Why? Well, because of other things I've seen before and maybe liked or disliked, maybe because of other things that people who like movies or series similar to mine have already seen. Therefore, that is also AI. There is no if else, if else, if else construct for every movie or TV series in their database. It's a way to discover in a much more organic and dynamic way what you and I might like. And then, all these current voice assistants, Siri, Alexa, Google Assistant and the like, don't recognize your voice, nor do they necessarily know what questions you're going to ask. There isn't a huge if, else
if block with every possible question in the world waiting. For you or me, of course, this is also generated dynamically, but that's getting ahead of ourselves. Let's go back in time. And some parents in the audience may remember this game, one of the world's first arcade games called Pong. So it was a black and white game, where there were two players, one paddle on the left and one on the right. And using some kind of joystick or trackball, they could move their paddles up and down. And the goal is to make the ball bounce back and forth and ideally catch it every time. Otherwise, you lose a point. It's just an animated gif, so there's nothing really spectacular to see. The score will remain at 15 to 12, repeating indefinitely. Nothing
interesting is going to happen. But it's a good example of a game that lends itself to being solved with code. In fact, for years it has been part of our vocabulary to play not only against the computer, but against the CPU, the central processing unit or, really, against AI. However, AI doesn't need to be nearly as sophisticated as the tools we see today. For example, here is a successor to Pong, known as Breakout, similar in essence, but there is only one paddle and one ball. And the goal is to make the ball bounce off those colored bricks. And you win more and more points, depending on how high you manage to hit the ball. All of us, as human beings, even if we have never played this classic, probably have the instinct of where to
move the paddle if the ball has just come out in that direction. Which way should I move the paddle? That is, probably to the left. And, in fact, that will intercept it on its descent. So, you and I just made a pretty instinctive decision, but it's deeply rooted in us. But we could, in a way, take all the fun out of the game and start quantifying or describing it in a more algorithmic, step-by- step way. In fact, decision trees are a concept from economics, strategic thinking, and also computer science. It's a way to solve this problem in such a way that you'll always play well if you simply follow this algorithm. So, for example, how could we implement a code or decision-making process
for something like Breakout? Well, ask yourself first. Is the ball to the left of the paddle? If so, you already know where we're going with this. Then go ahead and move the paddle to the left. But what if, in reality, the answer was no? Well, you 're probably not just blindly moving the paddle to the right. What should you ask? So, you're right under the ball? If the ball is coming straight towards you, it's not enough to simply go to the right naively and risk missing. So, the other question to ask is: "Is the ball to the right of the paddle?" And is this a yes or no question? If that's the case, well then that's fine, move to the right. But if that's not the case, you should probably stay exactly where you are and not move the paddle.
OK? So this is quite deterministic, if you will. Hmm. And we can translate this into code using pseudocode, say, in a class like CS50. We can say in a loop: well, while the game is happening, if the ball is to the left of the paddle, then move the paddle to the left. Oh, otherwise, if the ball is to the right of the paddle, excuse the typo, move the paddle to the right. Oh, if not, just don't move the palette. And so, these decision trees, as we draw them, have a perfect mapping to the code, or, in fact, to the pseudocode, in this specific case. Which means that this is exactly how the people who implemented the game Breakout or Pong, when creating a virtual player, certainly programmed it. It was as simple as that. But what about
something like the game of tic-tac-toe (three in a row), which some of you may have played for a moment upon entering here on scraps of paper, hmm, that you might have had handy? Hmm. Here we have a tic-tac- toe game board with two O's and two X's. For those who don't know it, this game of cat, also known as tic-tac-toe, consists of an alternation of X and O between two people. And the goal is to form a line with three O's or three X's, either vertically, horizontally, or diagonally. So, this is an ongoing game. Okay, let's think about how you could solve the tic-tac-toe game as a computer, as an AI would.
Well, you might ask yourself: can I get three in a row this turn? Well, if that's the case, play in the square to form a line of three. It's as simple as that. If you can't, however, what should you ask? Okay, can my opponent get three in a row on the next turn? Because if that's the case, you should probably at least block your opponent's next move. At least now you don't lose anymore. But this game, tic-tac-toe, which is relatively simple in itself, becomes a little more difficult to play when it's not obvious where you should throw. Now all of us, as human beings, if we grew up playing this game,
we probably had strategies that we used, like, you really like the center or the top corner or something like that. So we can probably make our next move quickly. But will it be the ideal move? And I dare say that if you lost a game of tic-tac-toe in childhood or more recently, it's simply because you're not good at it, since logically there's no reason for you to lose if you're playing your best. In the worst case scenario, you should force a draw, but in the best case scenario, you should win the game. So, think about that the next time you play tic-tac-toe and lose. Dude, you're doing something wrong, but in your defense it's because the question mark isn't that obvious. Dude,
how do I respond when the answer isn't right in front of me? Should I play to win or to block? Well, one algorithm you could have used all these years is called minimax. And as the name suggests, it's about minimizing one thing and maximizing another. So here too we're going to take some of the fun out of the game and turn it into math, but relatively simple math. So here we have three representative tic-tac-toe boards. O won here, X won here, and the one in the middle is a tie. It doesn't matter how we score these boards, but we need a consistent system. Therefore, I am going to propose that whenever O wins, the score of the game should be one. Whenever X wins, the game score will be plus one. And whenever nobody wins, the score
is zero. Therefore, at this moment, each of these boards has the following values: 1, 0, and 1. Therefore, the objective in this game of tic-tac-toe now is for X to maximize their score, since one is the highest value available. And O's goal in life is to minimize his score. That's how we take the fun out of the game. We transform it into mathematics, where one player only wants to maximize their score, while the other only wants to minimize it. All good? So, let's do a quick consistency check here. Here's a board, there's no color code. What is the value of this board? One, because X actually has a straight line right there in the middle.
Therefore, X is 1, 0, or E1. Otherwise, it would be a tie. So now let's see how we can apply these principles to discover where we should play in the cat. Now, here 's a very easy setup. Only two moves remain. It's not difficult to figure out how to win or tie this game, but we'll use this for simplicity. It's O's turn, for example. So, where can O go? Well, that raises the question, what is the value of the board? Or how do we minimize the value of the board so that O wins? Well, you can move to one of the two places, top left corner or bottom center. Which way should it go? Well, if O goes to the top left corner, we need to consider what the value of that board is. It's the minimum. Okay,
let's see. If O plays here X, obviously X will play here, therefore, will win. Therefore, the value of that board will be 1. Now, since there is only one logical way to go from this configuration to this one, we can well call the value of this board one by transitivity. And therefore, O probably doesn't want to go there, since that's a pretty high score and O wants to minimize it. On the other hand, if O goes to the bottom center corner, then X will go to the top left corner. And now nobody won. Therefore, the value of this board is zero. We can consider this as zero, since that is the only way to reach that conclusion, logically. So now O can decide in a more mathematical and logical way. I want a final result of one or zero. Well
, zero is probably the best option because it's less than one, and therefore that's the minimum possibility. Then O will advance in the lower central part and at least force a tie. And that's where you see the evidence that if you humans are losing the game, it's because you didn't follow this logic. But you could probably do it if there were only two moves left. But the problem is this: let's go back a little to when there were three moves left. There are three blank spaces here . And I've widened my view a bit. The problem is that the decision tree becomes much larger the more moves remain. It becomes a little bigger and bulkier, in the sense that it is essentially doubling its size and width. That's great if you have the luxury of
writing everything down on a piece of paper. But if you're doing it from memory while playing against a fifth grader, if you'll excuse the comparison, you're probably not analyzing all the possible boards and configurations to try to play your best. You're acting on instinct, and your instincts might not be aligned with a tried and tested algorithm, the Minimax, which ideally will make you win the game, but at least will allow you to force a draw if you can't win. But tic-tac-toe isn't that difficult. I mean, how many different ways are there to play tic-tac- toe? We could write a computer program to play tic-tac-toe in a practically ideal way. We could use code like this. If the player is X, for each possible move,
calculate the score of the board at that moment and then choose the move with the highest score. Then you simply test all the possibilities mathematically and then make the decision. Most of us in our minds aren't doing that, but we could. Otherwise, if the player is O, basically do the same, but choose the lowest possible score. So that's the code to implement the cat. But how many ways are there to play tic-tac-toe after all? Well, 255,168. 68. Which means that if we were to draw that tree, it would be really very big and it would take you a long time to think of all those possibilities. So, in your defense, maybe you're not so bad with cats after all. It's simply harder than you imagined as a game. But what about the games that
we, as adults, may be more familiar with? Well, what about chess, which is often used as a measure of, say, how intelligent a computer is, whether it's the Watson of that era playing against it or something else? Well, if we consider only the first four turns of chess, that is, black plays, then white, and then each plays three more times, totaling four pairs of turns, how many different ways are there to play chess? Well, it turns out it's 85 billion just to start the game, and that involves a lot of decisions to consider and then make. And what about the game of Go, in case you're familiar with it? Consider the first four moves: 266 quintillion possibilities. And that's
where we, as human beings, even with our modern PCs, Macs, and phones, have to give up, because I don't have that many bytes of memory on my computer, nor that many spare hours of life to really analyze all those numbers and discover the solution. And that's where AI comes in. When it's no longer as simple as just writing if statements and loops. And it 's no longer as simple as just trying all the possibilities. Instead, you need to write code that doesn't solve the problem directly, but indirectly. You write code to help the computer figure out how to win, perhaps by showing it board configurations that are promising and perhaps by showing it board configurations that it doesn't want to be in, because that will lead to defeat. In other
words, you train him, but not necessarily exhaustively. And that is what we understand today by machine learning. Writing code through which machines learn to solve problems in a general way, being trained with huge amounts of data and then, when faced with new problems, looking for patterns that allow them to apply that previous training data to the problem in question. And reinforcement learning is, in fact, one way of thinking about this. And in fact, we, as human beings, use reinforcement learning, which is a type of machine learning, practically all the time. In fact, a fun demonstration to watch here involves these pancakes. So, actually, let me show you here a short recording of an actual researcher in a lab trying to teach a robot how to flip
pancakes. So, let's see here in this video that there is a robot that has an arm that can go up, down, go left and right. That, of course, is the human being, the researcher. And he's simply going to show the robot, once or more, how to flip a pancake. And cross your fingers. And very well, it seems he did well. And he does it again. It's not exactly the same, but it's pretty close. And now he's going to let the robot try to figure out on its own how to flip that pancake after having trained it only a few times. The first few times, the robot probably won't do very well, because it does n't really understand what the human just did, or what the goal of it all is. But here is the fundamental detail of
reinforcement learning. Behind the scenes, the human is probably rewarding the robot when it does a good job. In other words, the better he flips the pancake, the more he is rewarded. Whether it's by pressing a key and earning a point, for example, or receiving the digital equivalent of a cookie. Or on the other hand, every time the robot makes a mistake and drops the pancake on the floor, it receives a kind of tap on the hand, a punishment to make it repeat that behavior less the next time. And any of you who are parents, which by definition many of you are today, have probably taught your children at some point to do more of one thing and less of
another, either in this way or perhaps only with verbal approval or scolding. And what you see there in the background now is just a quantization of the movements, the X, Y, and Z coordinates, so that he can do more of the X, Y, and Z things that led him to some kind of reward. And now, after nearly 50 attempts, the robot seems to be getting better and better, to the point that, like a good human being, let's see if I can do this without embarrassment, manage to turn things around. That's very good. That's right, well, I've been doing this for a long time. OK. So, we saw how it is possible to reinforce learning through this type of mastery. Let's give an example that will be familiar to you gamers. Have you ever played
a game where there is some kind of map or world that you need to explore up, down, left and right, maybe trying to reach the exit? So, in a simplified way, here is the player on the yellow dot. Here, for example, the map exit is shown in green. And you want to get to that point. And perhaps there are many others in other parts of the world, like lava pits. And you don't want to fall into the lava pit because you lose a life, you lose a point, or there's some penalty or punishment associated with it. Well, with this panoramic view we can obviously see how to get to the green dot, but if you're playing a game like Zelda or something similar, all you can do is move up, down, left, right and more or less hope that everything works out
. So we're going to do exactly that. Suppose the yellow dot chooses a direction at random and goes to the right. Well, now we can in a way take away a life, take away a point, or effectively punish him so that he knows he shouldn't do that. And as long as the player has a little memory, whether it's the human player or the code that implements this, a dark red line is enough to indicate it. Don't do that again, because it didn't lead to a good result. So, maybe next time the yellow dot will go here and there. And there, ah, I don't realize that it's actually the same lava crater, but that's okay.
Use a little more memory and remind me. Don't do that, because I just lost a second life in this story and maybe next time it will be like that. Ah, now I need to remember that. Don't do that. But in practice, I 'm either being punished for doing the wrong thing, or, as we shall soon see, being rewarded for doing more of what went right. And by chance, perhaps I can finally reach the exit that way. And that way I can be rewarded for it. I just got 100 points or whatever, the maximum score. So now, by following those green lines, I can simply follow that path, the same path over and over again. And I always manage to win this game. It's more or less like me today, like 30 years later playing Super Mario Bros, because I can get
through all the secret levels, since I know where everything is. For some reason, that's still stored in my brain. Is that the best way to proceed? Am I really as good at Super Mario Bros. as I think I am? What's wrong with that approach? That's true, exactly. Yes, I've moved far more times than I needed to. And just for fun today, what grade are you in? Seventh grade. Marvelous. So, the observation in the seventh grade is exactly that: we could have followed a shorter path, which is basically that one, although taking some direct steps. And so we will never find that shorter path, we will never achieve the maximum possible score if we naively continue following the well- trodden path. So how can we break out of that pattern? And
this can even be seen in the real world. Another personal example is that, for some reason, I'm the type of person who, when I go to a restaurant for the first time, chooses a dish from the menu and really likes it. I'm never going to order anything other than that dish from the menu again, because I know it's good. But there might be something even better at stake. Except I'm never going to explore that because I'm somewhat stubborn, just like some of you, judging by your smiles, might be too. But what if we took the opportunity to explore just a little bit? And there is this principle of exploring versus exploiting
when it comes to using artificial intelligence to solve problems. So far I have only used the knowledge I already possess. Please do not go through the red walls. Yes, go through the green walls. Explore, explore, explore and I will arrive at a definitive solution. But what if I simply added a little randomness to the process? And perhaps 10% of the time, as represented by this epsilon variable, I, as the computer of the story, generate a random number between zero and 1. And if it is less than that percentage, which will happen 10% of the time, I will make a random move instead
of one that I know will get me closer to the exit. Otherwise, I will truly make the decision with the greatest courage. Now, this doesn't necessarily guarantee me victory on the first try. But if I play repeatedly and introduce a bit of that randomness, I might well find a better solution and thus become a better player, a better winner overall. If I only ordered something different from the menu 10% of the time, I might discover an amazing dish that I would otherwise never have found. And, in fact, using this approach, we can finally find a more optimal path through the maze, which in this case was shorter, presumably maximizing our score and obtaining even better results. So, you can even see this in the game Breakout, especially if you write a
code solution to play that game for you. Let me show you another video recording of an AI playing Breakout. And what this AI is doing is essentially figuring out, perhaps more intelligently than you or I, how to play this game in the best way possible. And what we will see here is that, just like with the pancake-flipping robot, there is a notion of scoring, rewards, and penalties. So, for the time being, the palette is just doing random things. He still doesn't quite know how to play, but after 200 episodes, he realizes that my score goes up if I hit the ball and falls by the same proportion if I miss. And it 's still a little unstable. He still doesn't quite understand what he should do and
why. But if you repeat the process many times and there are enough rewards or punishments, you will see that the result begins to be very good and closer to what a good human being would do. But that's where the algorithm gets a little scary. If you let the game run long enough, or if you and I, humans, play it long enough, maybe we'll discover some trick to the game. I dare say that AI becomes a little frighteningly self-aware in this. After all, if you're smart enough to get past that top line, you can let the game run for you and maximize your score without even touching the ball. Something that I find a little scary is that he simply figured out how to do this without anyone telling him. But this is just a logical continuation of
rewarding good behavior and punishing bad behavior. So, the next time you have the chance to play Breakout, consider this type of strategy instead of doing most of the work yourself. Let the computer do it for you. So, what else is there to consider in this world of AI in the context of machine learning? Well, there is specifically a category of learning that is supervised, and we have been using it for years. And in fact, our first example of spam was certainly supervised, because it was you and I who put the email in the spam folder. And to this day, maybe once a day, I use the keyboard shortcut in Gmail to mark: "Ah, this is spam." You should have noticed that, and that means further training Google's algorithm, based on the
principle that it's not just me, but perhaps thousands of people marking that same type of email as spam. That is supervised learning in the sense that there is a human being involved in the process, a group that is at least doing something. Hmm, then spam detection may be one of those cases, but the problem is that. That would be like someone at Google or Microsoft having to label every email, or someone at Netflix doing the same with every video that exists. It's expensive in terms of labor and there are certainly problems with such a large amount of data. It's simply not realistic for humans to label millions of data points, billions of data points. We need to migrate to an unsupervised model. And that's where the world begins to consider
deep learning, solving problems through code, in a way that not even human beings participate in the process in the same way. And neural networks inspired by the world of biology are, in a way, the inspiration for what is the state of the art, serving as the basis even for today's "rubber duck" and more generally for these things called large language models like ChatGPT and the like. Here is a somewhat abstract representation of a neuron, and it is something in the human body that transmits a signal, let's say, electrically from left to right. And if you have several neurons, they can communicate with each other. So, if I have a thought, I know how to raise my hand because some kind of electrical message was
transmitted from my head to this limb here. So, that's essentially what I remember from ninth-grade biology class. But as computer scientists, we somehow abstract all of this. So, instead of calling these two neurons and drawing them as such, we'll just start drawing the neurons as these little circles. And if there is any kind of connective tissue between them, we simply draw a straight line, an edge between them. So, this is what a computer scientist would call a graph. If you have two neurons of this type here, leading to one here, um, one neuron here, you can think of this as perhaps being two inputs to a problem and now also one output.
We can represent the notion of problem-solving, which is the central theme of CS50 and, more generally, of introductory courses. So, we're going to solve a problem with a neural network without necessarily training it beforehand, simply letting it figure out how to answer that question. Here is a very simple two-dimensional world, an X Y grid and here are two points. The dots in this world are either blue or red, but I still have no idea what makes a dot blue or red. However, if you train me with these two points, I bet I could make predictions, especially if you let me label this world in terms of horizontal X coordinates
and vertical Y coordinates. So, you know what? We can think of this neural network in a very simple way, as representing the X coordinate here and the Y coordinate here. And the answer I want to get is, in quotes, red or blue , or zero or one, or true or false, however you want to think of this representation. So how do I go from a specific X,Y coordinate to a color prediction if I only know the coordinates? Well, from the beginning, perhaps the best thing I can do is simply divide the world into blue dots on the left and red dots on the right. A line of best fit, if you want to call it that, based on very minimal data. Of course, if you give me a third point, it will be quite easy to notice that I was a
bit hasty. This line isn't vertical, so maybe we can rephrase the sentence this way. And now I'm back in action. Now I can predict with greater probability, based on x and y, what the color of the next point will be. If you give me enough points, I can draw a fairly accurate line of best fit. It's not perfect. But here's a clue as to why AI isn't perfect. But 99% of the time, I might be able to predict correctly, and I can do even better if you let me zigzag a little on the line and maybe make it something more than a simple tilt. So what are we really doing by implementing this neural network, even in a simplified way, with only three neurons?
Well, essentially we're trying to arrive at three values, three parameters: A, B, and C. And what do they represent? Well, actually, it's just one solution for this formula. This line we draw can be represented, if you remember high school math, by a formula more or less like this: a x + b x plus a constant c. And we can simply conclude arbitrarily that if this value mathematically gives me a number greater than zero, it is to be expected to be blue. Otherwise, it predicts it will be red. We can, in a way, apply our mathematics just like in the game of tic-tac-toe. What about the real problem that interests us when we define the world in this way? And so, if you give me enough data points,
I can find answers for that A, that B, that C, the so-called parameters in neural networks. In reality, neural networks are not composed of, say, three neurons and some connections. They look more or less like this. And in practice, there are billions of these things here on the screen. In this case, virtually every one of these edges represents some mathematical value that was calculated based on a huge amount of training data. And although I, the computer scientist, may know what these neurons represent here, since these are my inputs, three in this case, and I, the computer scientist, know what this one represents in the end. If you were to, say, remove the lid from this thing and look inside the neural network, even if
there were millions, billions of numbers happening there, I wouldn't be able to say what that neuron represents or why that connection has that weight. It is because of the enormous amount of training data that the math simply works this way. And if you provide me with more data, perhaps I'll change some of these parameters a little more. Thus, the graph may end up looking very different, but my inputs and outputs will be what I use to solve this problem. So, if you want to predict, for example, precipitation from humidity or pressure, you can have two inputs generating that single output, such as advertising expenditures in a given month that could predict sales, simply by having been retrained with those volumes of data. And when
we close the cycle with something like the CS50 rubber duck and great language models like Claude, Gemini, and ChatGPT, what's really happening and all of this is new in recent years. The screenshots here show some of the recent scientific articles that have driven much of this progress in recent years. From OpenAI you have, let's say, a pre-trained generative transformer, which is a very long name, but that's where ChatGPT's GPT comes in. And essentially it is a neural network trained with large volumes of textual information that provides us with the interactive chat function that we have in this class and that we all have more generally in ChatGPT itself. So, here's an example of what really goes on
behind the scenes at these GPTs. Well, here's a paragraph that until recently was a little difficult to end with ellipses. Massachusetts is a state in the New England region, in the northeastern United States. It borders the Atlantic Ocean to the east. The state capital is dot. Well, probably almost everyone who lives in Massachusetts knows that answer. But, if this AI has just been trained on lots and lots of data, there are probably a lot of people who would say "Massachusetts" in one part of a sentence, and then the answer, which I wo n't reveal yet, is in the other part of the sentence. But, in this
example, since the question we are asking is quite far removed from some of the useful keywords, until recently this was a difficult problem to solve, as the distance was very large. In addition, there are these nouns that are being used to replace the proper name. Like, suddenly, we started calling this a state. Down here we call this the state. And it wasn't necessarily obvious to the AI that we were talking about the same thing, as if it were just city, comma, state, where there would be much more proximity. So, to sum up, what we do now, especially to solve problems like these, is first to split a sentence, training data, or similar inputs into something like an array or a list of the words themselves. We created a
representation for each of those words. For example, the word Massachusetts, if you encode it in a certain way, will be represented by an array or vector of floating-point numerical values. So many that the word Massachusetts in a model would use those 1536 floating-point numbers to represent Massachusetts, essentially in an n-dimensional space. So, not just an x-y plane, but somewhere virtually out there. And then, and this has been the key to these GPTs, attention is calculated based on all this data, so that in this image the thicker lines indicate a stronger relationship between those two words. Therefore, Massachusetts and State are interpreted as having a thicker line, indicating greater attention from one word to the other, while
our "a", our "is", and our have thinner lines, because they simply do not represent as strong a signal to the AI as the answer to that question. Meanwhile, when you enter this phrase as: "The state capital is", one word per neuron, the goal here is to get the answer to that question. And yet, this is a much smaller representation of what the real neural network would be. But, in reality, all these LLMs, the great language models, are just statistical models. In other words, what is the word most likely to be generated at the end of this paragraph, based on all the Reddit posts, Google search results, encyclopedias, and Wikipedias it found and was trained on online? Okay, I'm hoping the answer is Boston, but of course, 1% of the time, maybe
less than that, the answer might not be correct. And even the CS50 duck itself is fallible, even though we've written a lot of code to try to minimize those errors. And those errors are what we will call hallucinations, where the AI simply invents something. Perhaps because some crazy human on the internet made something up and it was interpreted as reliable, or simply due to bad luck resulting from a bit of that exploration. 10% of the time, 1% of the time, the AI somehow veered to that side in the larger neural network language model and gave an answer that is actually not correct. So I thought I'd end for today with this final note, a poem by Shel Silverstein, which many of us may have grown up with, about the homework-doing machine, which
years ago somehow predicted the situation we'd be in with these AI machines. He said, "The homework-doing machine." Ah, the homework machine, the most perfect contraption ever seen. Just insert your homework, put in a coin, turn on the switch and in 10 seconds your homework will come out quick and perfect, as it should be. Here 's 9 + 4 and the answer is 3. Th 3. Wow, I guess it didn't turn out as perfect as I imagined it would. So that was the C50. Until next time.
