吳恩達 Agentic AI 第七講 Evaluating agentic AI
- URL: https://www.youtube.com/watch?v=Bo5YHWQWV6Q
- Channel: 扣子說AI
- Fetched: 2026-08-12T18:49:52+00:00
- Language: English (auto-generated)
Transcript
[00:01] I’ve worked with many different teams on
[00:02] building agentic workflows, and I found
[00:05] that one of the biggest predictors for
[00:08] whether someone is able to do it really
[00:10] well versus be less efficient at it is
[00:13] whether or not they’re able to drive a
[00:15] really disciplined evaluation process.
[00:18] So, your ability to drive evals for your
[00:21] agentic workflow makes a huge difference
[00:23] in your ability to build them
[00:26] effectively.
[00:27] In this video, we’ll take a quick
[00:28] overview of how to build evals, and this
[00:31] is a subject that we’ll actually go into
[00:33] much deeper in a later module in this
[00:36] course. But, let’s take a look.
[00:38] After building an agentic workflow like
[00:40] this one for responding to customer
[00:43] order inquiries, it turns out that it’s
[00:45] very difficult to know in advance what
[00:47] are the things that could go wrong. And
[00:49] so, rather than trying to build
[00:51] evaluations in advance, what I recommend
[00:53] is you just look at the outputs and
[00:56] manually look for things that you wish
[00:59] it was doing better. For example, maybe
[01:02] you read a lot of outputs and find that
[01:04] it is unexpectedly mentioning your
[01:06] competitors more than it should. Many
[01:09] businesses don’t want their agents to
[01:11] mention competitors. This just isn’t a
[01:13] good situation. And if you read some of
[01:16] its outputs, maybe you’ll find that it
[01:17] sometimes says, “I’m glad you shopped
[01:19] with us. We’re much better than our
[01:20] competitor CompCo.” Or maybe sometimes
[01:23] they say, “Sure, a sure refund. Unlike
[01:24] RivalCo, we make returns easy.” And you
[01:27] may look at this and go, “Gee, I really
[01:28] don’t want this to mention competitors.”
[01:31] This would be an example of a problem
[01:33] that is really hard to anticipate in
[01:36] advance of building this agentic
[01:37] workflow. So, the best practice is
[01:39] really to build it first and then
[01:40] examine it to figure out where it is not
[01:43] yet satisfactory, and then to find ways
[01:46] to evaluate as well as improve the
[01:48] system to eliminate the ways that it is
[01:50] still not yet satisfactory. Assuming
[01:53] your business considers it an error or a
[01:56] mistake to mention competitors in this
[01:58] way, then as you work on eliminating
[02:01] these competitor mentions, one way to
[02:03] track progress would be to add an
[02:05] evaluation or an eval to track how often
[02:08] this error occurs. So, if you have a
[02:11] named list of competitors like CompCo,
[02:13] RivalCo, the other Co, then you can
[02:15] actually write code to just search in
[02:18] your own output for how often it
[02:20] mentions these competitors by name and
[02:23] count up as a number as a fraction of
[02:25] the responses how frequently it
[02:28] mistakenly mentions competitors. One
[02:30] nice thing about the problem of
[02:33] competitor mentions is is a objective
[02:36] metric, meaning either the competitor
[02:37] was mentioned or not. And for objective
[02:40] criteria, you can write code to check
[02:44] for how often this specific error
[02:46] occurs. But because LLM’s output free
[02:49] text, there’re also going to be criteria
[02:52] by which you want to evaluate this
[02:53] output that may be more subjective. In
[02:56] other words, harder to just write code
[02:58] to output a black and white score. In
[03:00] this case, using a LLM as a judge is a
[03:04] common technique to evaluate the output.
[03:06] So, for example, if you’re building a
[03:08] research agent to do research on
[03:10] different topics, then you can use
[03:13] another LLM and prompt it to maybe say
[03:16] assign the following essay quality score
[03:18] between one and five, where one is the
[03:20] worst and five is the best essay. Here
[03:23] I’m using a Python expression to mean,
[03:26] you know, copy paste the generated essay
[03:28] into this. So, you can prompt the LLM to
[03:30] read the essay and assign it a quality
[03:33] score. Then I’m ask the research agent
[03:35] to write a number of different research
[03:37] reports, for example, on recent
[03:39] developments in black hole science or
[03:42] using robots to harvest fruit. And then
[03:44] in this example, maybe the judge LLM
[03:47] assigns the essay on black holes a score
[03:50] of three, the essay on robot harvesting
[03:52] a score of four, and as you work on
[03:54] improving your research agents,
[03:56] hopefully you’ll see these scores go up
[03:58] over time. It turns out, by the way,
[04:00] that LLMs are actually not that good at
[04:03] these one-to-five scale ratings. You can
[04:05] give it a shot, but personally tend not
[04:07] to use this technique that much myself,
[04:09] but in a later module you’ll learn some
[04:11] better techniques to have an LLM output
[04:14] more accurate scores than asking it to
[04:16] output scores on a one-to-five scale,
[04:17] although some people will do this maybe
[04:19] at initial cut as an LLM as judge type
[04:22] of eval. Just to give a preview of some
[04:25] of the agentic AI evals you’ll learn
[04:27] about later in this course. You’ve
[04:29] already heard me talk about how you can
[04:30] write code to evaluate objective
[04:33] criteria, such as did it match the
[04:34] computer or not, or use an LLM as a
[04:36] judge for more subjective criteria, such
[04:39] as what’s the quality of this essay. But
[04:40] later, you’ll learn about two major
[04:42] types of evals. One is end-to-end, where
[04:44] you measure the output quality of the
[04:46] entire agent, as well as component level
[04:49] evals. So, we might measure the quality
[04:51] of the output of a single step in the
[04:53] agentic workflow. And turns out that
[04:55] these are useful for driving different
[04:57] parts of the development process. And
[04:59] one thing I do a lot as well is just
[05:02] examine the intermediate outputs, or
[05:04] sometimes you call this a traces of the
[05:06] LLM, in order to understand where it is
[05:09] falling short of my expectations. And we
[05:11] call this error analysis, where we just
[05:13] read through the intermediate outputs of
[05:15] every single step to try and spot
[05:17] opportunities for improvements. And it
[05:19] turns out being able to do evals and
[05:21] error analysis is a really key skill.
[05:23] So, we’ll have much more to say about
[05:25] this in the fourth module in this
[05:27] course. We’re nearly to the end of this
[05:29] first module. Before moving on, I just
[05:31] want to share with you what I think are
[05:33] the most important design patterns for
[05:35] building agentic workflows. Let’s go
[05:37] take a look at that in the next video.