1 00:00:00,031 --> 00:00:02,575 Let's try out an exercise to improve our prompt 2 00:00:02,595 --> 00:00:05,158 evaluation workflow. So here's the task 3 00:00:05,198 --> 00:00:07,804 I'm going to give you. I want to improve our model 4 00:00:07,944 --> 00:00:10,747 greater a little bit by providing it with more context 5 00:00:10,929 --> 00:00:13,432 on what a good solution should actually look like. 6 00:00:14,073 --> 00:00:16,638 Now, at first glance, that might sound a little bit challenging, but it turns 7 00:00:16,718 --> 00:00:19,282 out you only really have to go through two steps to add 8 00:00:19,301 --> 00:00:21,966 in this additional context. So in step one, 9 00:00:22,065 --> 00:00:24,987 I would encourage you to go back to the prompt where we generate 10 00:00:25,068 --> 00:00:27,890 our data set. And inside that prompt, try asking 11 00:00:27,931 --> 00:00:30,794 it for some solution criteria to be included 12 00:00:30,814 --> 00:00:33,316 in every test case. So ideally, 13 00:00:33,496 --> 00:00:36,018 our test cases that could output should now have 14 00:00:36,078 --> 00:00:38,761 some additional solution criteria key, which 15 00:00:38,841 --> 00:00:41,622 might look like what you see right here. So it might say something 16 00:00:41,683 --> 00:00:44,625 about what a good solution would look like. Maybe you say, well, 17 00:00:44,665 --> 00:00:47,709 a good solution would include this characteristic and this characteristic 18 00:00:47,808 --> 00:00:50,674 and this characteristic. Once we have this additional 19 00:00:50,735 --> 00:00:53,439 solution criteria, we can then insert that into 20 00:00:53,460 --> 00:00:56,543 our grade by model prompt. So 21 00:00:56,603 --> 00:00:59,247 you might find the existing area of that prompt where 22 00:00:59,287 --> 00:01:01,851 we put in our solution to evaluate, and then 23 00:01:01,911 --> 00:01:04,855 right after it, you might add in that newly generated 24 00:01:04,896 --> 00:01:07,459 solution criteria. And that's all it 25 00:01:07,500 --> 00:01:10,224 would really take to give our model greater a little 26 00:01:10,243 --> 00:01:12,927 bit better idea of what a good solution would actually look 27 00:01:12,968 --> 00:01:15,843 like. As usual, I would encourage you to pause 28 00:01:15,862 --> 00:01:18,427 the video right now and give this exercise a shot. 29 00:01:19,269 --> 00:01:21,993 Otherwise, we're going to go through a solution right now. So 30 00:01:22,474 --> 00:01:25,140 the solution really is just going to be these two separate steps. 31 00:01:25,280 --> 00:01:28,004 Should be pretty straightforward. To get started, back 32 00:01:28,064 --> 00:01:30,769 inside my notebook, I'm going to find our generate dataset 33 00:01:30,829 --> 00:01:33,454 function. And inside there, I'll find the really big prompt 34 00:01:33,495 --> 00:01:36,076 we put in. And then when we ask for each 35 00:01:36,096 --> 00:01:38,640 of these different test cases, I'm going to say in addition to a 36 00:01:38,700 --> 00:01:41,284 task in the output format, I also want to 37 00:01:41,384 --> 00:01:44,067 get some solution criteria. 38 00:01:45,170 --> 00:01:47,753 And then I'll put in a string right here just to give our model 39 00:01:47,793 --> 00:01:50,918 an indication of what this key should actually be. So 40 00:01:50,938 --> 00:01:54,802 I'm going to ask for some key criteria for 41 00:01:54,843 --> 00:01:59,329 evaluating the solution. And 42 00:01:59,370 --> 00:02:02,513 that's pretty much it. So I'm going to rerun 43 00:02:02,534 --> 00:02:05,269 the cell. I'm going to go to the cell underneath 44 00:02:05,310 --> 00:02:07,823 it and regenerate the data 45 00:02:07,843 --> 00:02:11,807 set. Okay, 46 00:02:11,828 --> 00:02:15,032 just a couple of seconds, it should be done. There we go. So 47 00:02:15,072 --> 00:02:17,615 now we should have an updated data set.json 48 00:02:17,675 --> 00:02:20,359 file. I'm gonna open that file up. And I should now see 49 00:02:20,399 --> 00:02:23,081 some updated tasks in here, still with the format, but 50 00:02:23,141 --> 00:02:25,944 now I've also got some solution criteria. So 51 00:02:25,965 --> 00:02:28,508 the solution criteria, we can read over, of course 52 00:02:28,549 --> 00:02:31,132 yours is gonna look different than mine, but it's gonna get 53 00:02:31,151 --> 00:02:33,754 going to give some idea on, again, what a good 54 00:02:33,794 --> 00:02:36,361 solution will actually look like. Next up is 55 00:02:36,521 --> 00:02:39,167 step number two. We're going to find our grade by model 56 00:02:39,206 --> 00:02:41,711 function and specifically the prompt inside there. 57 00:02:42,233 --> 00:02:44,877 And we're going to include this newly generated solution 58 00:02:44,918 --> 00:02:47,622 criteria. Again, just to tell the model grader 59 00:02:47,924 --> 00:02:50,468 what a good solution looks like. So for that, I 60 00:02:50,508 --> 00:02:53,955 will go back to the notebook. I will scroll down and 61 00:02:54,014 --> 00:02:56,520 find that grade by model function. Here 62 00:02:56,539 --> 00:02:59,508 it is right here. So I'm going to find the prompt. We 63 00:02:59,549 --> 00:03:02,151 are already putting in the original task, the 64 00:03:02,312 --> 00:03:04,814 output that was generated. And then right after that, I'm going 65 00:03:04,835 --> 00:03:07,657 to put in some note to the model and just say, here's 66 00:03:07,677 --> 00:03:10,381 some criteria that you should use to evaluate the solution. 67 00:03:11,322 --> 00:03:13,903 So criteria you should 68 00:03:14,125 --> 00:03:17,649 use to evaluate the solution. 69 00:03:18,610 --> 00:03:21,132 I'm going to put in some tags. And I'll tell 70 00:03:21,171 --> 00:03:24,175 you why we are adding in these tags very shortly as 71 00:03:24,216 --> 00:03:27,258 we start to discuss prompt engineering. 72 00:03:27,457 --> 00:03:30,141 And then I'm going to interpolate in from the test case, 73 00:03:30,984 --> 00:03:34,449 our solution criteria. And 74 00:03:34,468 --> 00:03:37,312 that key right there, remember our test case is really these 75 00:03:37,493 --> 00:03:40,078 objects, each of these objects one by one. So we know 76 00:03:40,117 --> 00:03:42,741 because we see it right here inside this file, there is a key inside 77 00:03:42,782 --> 00:03:45,306 that dictionary of solution criteria. So 78 00:03:45,325 --> 00:03:47,830 we're taking that sentence right there and putting it 79 00:03:48,390 --> 00:03:50,895 right here. All right, so 80 00:03:50,914 --> 00:03:54,617 now time to run the cell. And 81 00:03:54,638 --> 00:03:57,520 we're going to rerun our pipeline and see how everything 82 00:03:57,561 --> 00:04:01,506 is working. So I'll go down to 83 00:04:02,366 --> 00:04:04,929 the run eval function. And then right after that is where 84 00:04:04,949 --> 00:04:07,632 we actually execute everything. So I'm going to run that. 85 00:04:08,372 --> 00:04:10,936 And then we get our updated score back. Now, I 86 00:04:10,955 --> 00:04:13,598 want to print out the results really quickly, just so we can 87 00:04:13,658 --> 00:04:16,221 see how this is going to affect the actual output. 88 00:04:16,742 --> 00:04:19,324 So we'll do another print of JSON, dumps, 89 00:04:19,706 --> 00:04:25,105 results with an indent of two. So 90 00:04:25,144 --> 00:04:27,651 now we can see the output from our model, so 91 00:04:27,672 --> 00:04:30,218 that's the actual produced output. Here's our test 92 00:04:30,259 --> 00:04:32,785 case, so we can take a look at the task in the solution criteria. 93 00:04:33,487 --> 00:04:36,293 Here's the score, in this case it was 9, and now hopefully 94 00:04:36,314 --> 00:04:38,940 our reasoning section, which is produced by the model grater, 95 00:04:39,281 --> 00:04:41,788 is going to be a little bit more fleshed out than it was before 96 00:04:42,108 --> 00:04:44,394 because we are including that solution criteria.