tests.jsonl

{"question": "What does the instructor say RAG fundamentally is?", "keywords": ["hack", "performance trick", "hand-wavy"], "reference_answer": "The instructor calls RAG a 'big old hack' — a performance trick that uses encoder and vector similarity to zoom in on probably relevant content. It is described as hand-wavy and experimental rather than a rigorous scientific approach.", "category": "conceptual"}
{"question": "What is the tyranny of RAG?", "keywords": ["tyranny of rag", "doesn't always", "end up the way"],"reference_answer": "The tyranny of RAG is that ultimately what matters is the answers to your questions, and it doesn't always end up the way you think. Even well-reasoned ideas can fail because the right chunks may not get surfaced.", "category": "conceptual"}
{"question": "What is MRR and how is it calculated?", "keywords": ["mean reciprocal rank", "reciprocal", "rank"], "reference_answer": "MRR stands for Mean Reciprocal Rank. It is calculated by taking the inverse of the rank of the first relevant chunk retrieved — 1 if in position 1, 1/2 if position 2, 1/3 if position 3 — and averaging this across all test cases.", "category": "metric"}
{"question": "What is the key difference between RAG and fine-tuning?", "keywords": ["expertise", "generalize", "new skill"], "reference_answer": "RAG adds existing subject matter expertise from a knowledge base but cannot help a model generalize beyond it. Fine-tuning teaches the model a new skill it can generalize to data it has never seen before.", "category": "comparison"}
{"question": "What is query rewriting and why is it useful?", "keywords": ["query rewriting", "knowledge base", "conversation history"],"reference_answer": "Query rewriting is when an LLM rephrases the user's question into a form better suited for querying a knowledge base. It is useful because the original question may not match well in a vector lookup, and it can incorporate conversation history to handle follow-up questions.", "category": "technique"}
{"question": "What problem does hierarchical RAG try to solve?", "keywords": ["spanning", "hierarchical", "achilles heel"],"reference_answer": "Hierarchical RAG tries to solve the Achilles heel of RAG being bad at questions that span many documents. It creates summary documents at various levels of roll-up so a lookup across summaries can surface coarse-grained information before drilling down to fine-grained chunks.", "category": "technique"}
{"question": "What are the two reasons people say RAG is dead?", "keywords": ["context windows", "agentic", "dead"], "reference_answer": "The first reason is that context windows are now so large you could put the entire knowledge base in the context and let the transformer figure out relevance. The second reason is that agentic RAG — where an agent uses tools to retrieve context — has replaced the fixed pipeline approach.", "category": "conceptual"}
{"question": "Why is recall more important than precision in RAG?", "keywords": ["recall", "precision", "irrelevant"], "reference_answer": "Recall is more important because LLMs generally do not care too much if given some irrelevant chunks as long as the relevant content is present. Missing relevant content (low recall) is a bigger problem than having some extra irrelevant chunks in the context.", "category": "metric"}
{"question": "What is the whack-a-mole problem in RAG?", "keywords": ["whack a mole", "pops up", "rag problems"],"reference_answer": "The whack-a-mole problem is that when you fix one RAG retrieval issue, another one pops up somewhere else. Because RAG is experimental and hacky, problems keep appearing as you address them rather than resolving cleanly.", "category": "conceptual"}
{"question": "What three dimensions does LLM-as-a-judge evaluate answers on?", "keywords": ["llm as a judge", "accuracy", "completeness"],"reference_answer": "LLM-as-a-judge evaluates answers on accuracy (how factually correct the answer is), completeness (how thoroughly it covers all aspects), and relevance (how directly it answers the question without extra irrelevant information).", "category": "metric"}
{"question": "What is query expansion in RAG?", "keywords": ["query expansion", "multiple queries", "k times n"],"reference_answer": "Query expansion is when instead of generating one rewritten query, you generate multiple different queries. Each query retrieves documents independently, and the results are merged into a larger candidate pool before reranking.", "category": "technique"}
{"question": "What is semantic chunking?", "keywords": ["semantic chunking", "based on the meaning", "distinct"],"reference_answer": "Semantic chunking is a strategy where documents are split based on the meaning of each block of text rather than by character count alone, trying to divide documents into chunks that reflect distinct concepts so questions are more likely to match a single relevant chunk.", "category": "technique"}
{"question": "What did experimenting with chunk size of 500 characters reveal?", "keywords": ["chunk size of 500", "better than", "1667"],"reference_answer": "Experimenting with a chunk size of 500 characters revealed it performed best of the sizes tested (500, 1000, 1667), producing the highest MRR. Smaller chunks are more likely to tightly match a specific question, though more chunks need to be retrieved to compensate for the smaller context per chunk.", "category": "technique"}
{"question": "What is reranking in a RAG pipeline?", "keywords": ["reranking", "most relevant", "reorder"],"reference_answer": "Reranking is the step where retrieved chunks are reordered by relevance to the question, with the most relevant chunks placed first. It is typically done by an LLM or a cross-encoder scoring each chunk against the query. Less relevant chunks can then be discarded before passing context to the answer LLM.", "category": "technique"}
{"question": "What does the instructor say is the most important topic of the entire course?", "keywords": ["evals", "single most important", "entire course"],"reference_answer": "The instructor says evaluations (evals) is perhaps the single most important topic of the entire course. Evals provide a way to put scientific rigor around the art form that is RAG, giving a quantitative metric to measure and iterate against.", "category": "conceptual"}
{"question": "What is the difference between a system prompt and a user prompt?", "keywords": ["system prompt", "user prompt", "setting the scene"], "reference_answer": "The system prompt specifies the frame, the overall task, the tone, and provides context \u2014 it's your way of setting the scene. The user prompt is the actual message coming from an end user that the LLM should respond to in the context of whatever was set in the system prompt.", "category": "comparison", "original_transcript": "One is called the system prompt and one is called the user prompt. And the system prompt is where you specify the frame, the overall task they're carrying out. You tell them what tone they should take. You give them context. We'll find out later. When we talk about things like Rag that you can supply information in the system prompt, it's your way of setting the scene. The user prompt is the actual message coming from an end user to the LM that it should respond to in the context of whatever you told it in the system prompt."}
{"question": "What format does OpenAI expect for messages sent to its chat completions API?", "keywords": ["Python list", "dictionaries", "role"], "reference_answer": "OpenAI expects a Python list that contains Python dictionaries. Each dictionary has two keys \u2014 one called role with a value like 'user', and another called content which holds the actual message text.", "category": "technique", "original_transcript": "I'm just going to tell you that it is a Python list and the Python list contains a number of dictionaries. Python dictionaries. In this case, we're only going to have one dictionary. There it is. So it's a list with one dictionary in it. And that dictionary is going to have two keys. One key is called role. And the other key is going to be called content. And that role we're going to put in that in for the key role. We're going to have the value being the word user."}
{"question": "Why is Ollama described as packaged software for running open source models?", "keywords": ["packaged", "efficient C++", "GIF file"], "reference_answer": "Ollama is a product that runs fixed open source models that have been packaged and prepared. The code has been optimized into very efficient C++, and the weights have been compressed into a special file called a GIF file that you download to your computer.", "category": "conceptual", "original_transcript": "Well, Olama is itself a product to make it very easy to run fixed open source models in a way that's been packaged and prepared so that in just a couple of steps, like we did yesterday, you can run an open source model yourself on your computer. So it's packaged. It only takes very specific versions of models. The code has been optimized into very efficient C++, and the weights have been compressed into a special file called a GIF file. And you download that onto your computer."}
{"question": "What is the difference between a base model, a chat model, and a reasoning model?", "keywords": ["base model", "chat variant", "reasoning model"], "reference_answer": "A base model just takes a sequence and predicts what comes next, like predictive text. A chat variant or instruct variant is trained to work with system/user prompts and replies. A reasoning model has been further trained to first output thinking steps before giving the answer.", "category": "comparison", "original_transcript": "The starting point is known as a base model, is an LM that's just there to take a sequence of information as the input and to predict what would come next. That's all it does. And these base models, you don't come across them very often. We will do later in the course when we get to work with our own. Um, but they are, they are before it's been taught how to do things like chat with someone."}
{"question": "What is budget forcing in reasoning models?", "keywords": ["budget forcing", "word weight", "thinking trace"], "reference_answer": "Budget forcing is a technique to make a reasoning model think longer by simply inserting the word 'wait' periodically into the thinking trace. Doing so causes the model to reflect on its reasoning, reconsider, and weigh up its arguments more deeply.", "category": "technique", "original_transcript": "And So they discovered these scientists that just simply by adding the word weight periodically into the sequence as you're making it predict the next tokens, causes it to reflect on its reasoning and reason a bit deeper, and challenge itself and weigh up the arguments that it's made and consider whether they're still accurate. And so just just adding the word weight got better outcomes. And that technique is known as budget forcing."}
{"question": "Why do llms hallucinate confidently when they make mistakes?", "keywords": ["plausibility", "trained to sound confident", "most likely next"], "reference_answer": "LLMs are predicting the most likely next word or token in a plausible way, so plausibility is what they've been trained for \u2014 they've been trained to sound confident. When the most likely next word doesn't happen to be the truth, the model still produces it with conviction, which is why hallucinations can be dangerous.", "category": "conceptual", "original_transcript": "They can be very confident when they make mistakes because all they're doing is predicting the most likely next, next word, next token. They're predicting what they believe comes next in a in a plausible way. So plausibility is what they've been trained for. They've been trained to sound confident. That's all they do. The very fact that the plausible most likely next word so often happens to be the truth."}
{"question": "What does it mean that every call to an LLM is stateless, and how is the illusion of memory created?", "keywords": ["stateless", "illusion", "whole conversation"], "reference_answer": "Every call to an LLM is completely stateless \u2014 it has no background from earlier calls. The illusion of memory is created by trickery: when you build your messages list, you don't just give the most recent message, you pass in the whole conversation so far, so the LLM appears to remember.", "category": "conceptual", "original_transcript": "These five points I've got here you need it needs to be second nature to you. It's so obvious. Every call to an LLM completely stateless number two we pass in the whole conversation so far with every input prompt every time. And number three, it gives the illusion that the LLM remembers what you said 30s ago, because you're passing it in every time. It can keep the context of the conversation. The whole thing is a trick."}
{"question": "What is one-shot prompting?", "keywords": ["one shot prompting", "one example", "replicate"], "reference_answer": "One-shot prompting is when you give the model one example of a question and a good answer in the prompt. The model is meant to learn from that example, replicate it, and use it to generate similar outputs.", "category": "technique", "original_transcript": "Just giving multiple answers is also basically like multi-shot prompting. So in this case we're doing single shot prompting. We're giving it one example. And we're saying given this example, learn from it, replicate it, and it's just going to be fine at it. Remember, all it's doing is it's going to take this as the input and it's generating the most likely tokens to come next."}
{"question": "What is streaming in an LLM API call and how do you enable it?", "keywords": ["stream equals true", "typewriter animation", "chunks"], "reference_answer": "Streaming gives the typewriter animation where output appears token by token as the model generates it. You enable it by passing stream=True to chat completions create, which returns a stream object that you can iterate over to get each chunk as it flows back.", "category": "technique", "original_transcript": "you pass in two things. We pass in the model and the messages. And we learned you could pass in a third thing the response format where there is another, there's a fourth thing and it is stream. You can say stream equals true by default. It's false. If you pass in stream equals true, then what you get back is not the response object, but rather you get back an object called stream. And it's something which you can iterate over, like I do right here for chunk in stream."}
{"question": "How does training data relate to model parameters according to the Chinchilla scaling law?", "keywords": ["Chinchilla scaling law", "proportional", "twice as much training data"], "reference_answer": "The Chinchilla scaling law says that the number of parameters in a model is roughly proportional to how much training data it can successfully be trained on. If you double the parameters, you need twice as much training data to take advantage of that capacity.", "category": "conceptual", "original_transcript": "There's something from from a few years ago called the Chinchilla scaling laws, which you may have heard of. It's not talked about so much anymore, but it says that generally speaking, the number of parameters you have in a model is roughly proportional to how much training data it can successfully be trained on, and still absorb that training data. So generally speaking, more parameters means we could do more training. We've got a beefier model."}
{"question": "What is the rule of thumb for converting tokens to words?", "keywords": ["three quarters of a word", "1000 tokens", "750 words"], "reference_answer": "The most common rule of thumb is that a token is roughly three quarters of a word, so 1000 tokens is about 750 words. Another rule of thumb is that a token ends up being about four characters.", "category": "metric", "original_transcript": "The rule of thumb people tend to use is that typically, typically a token ends up being about four characters. If you're if you've got a character count, you're trying to turn it into the number of tokens. Or the most common rule of thumb people use is that, roughly speaking, a token is like three quarters of a word, so a thousand tokens would be about 750 words."}
{"question": "What is the context window of an LLM and what must fit inside it?", "keywords": ["context window", "maximum number of tokens", "input sequence"], "reference_answer": "The context window is the maximum number of tokens that a particular model can fit and consider when generating the next token. What needs to fit isn't just the most recent message but the whole conversation so far, including all the prior messages and the generated tokens.", "category": "conceptual", "original_transcript": "the context window is the maximum number of tokens that any particular model can fit that it can it can look back on. It can consider for when it's generating the next token. It's the maximum length of the input that it can that it can handle. And if you try and pass in more input than that, then it will fail. It will say bigger than my context window. Can't do it. Now here's something to keep in mind that that thing that gets that needs to fit within the context window isn't just your message you've just given to the LM. But as you now understand, it's what the actual input sequence isn't just the most recent message, it's the whole conversation."}
{"question": "Why are output tokens including reasoning more expensive to generate?", "keywords": ["output tokens", "reasoning", "thought process"], "reference_answer": "When you pay for output tokens, that includes any reasoning the model is doing \u2014 output that describes the thought process. With GPT models you don't even see these outputs but you still pay for them because the calculations and processing need to happen.", "category": "metric", "original_transcript": "The other catch is that when you pay for these output tokens that Includes any reasoning that the model is doing for these reasoning models that generate output that describes their thought process. Those are output tokens that you need to pay for, and in the case of gpts models, you actually don't even get to see these outputs. It's all it's all happening behind the scenes, but you still pay for them. And again, people sometimes think that that sounds a bit mean."}
{"question": "What is the difference between training time scaling and inference time scaling?", "keywords": ["training time scaling", "inference time scaling", "orthogonal tracks"], "reference_answer": "Training time scaling means making the model bigger with more parameters and more training data. Inference time scaling means tricks at run time like reasoning or putting more information in the input sequence (like RAG). These are two different orthogonal tracks to getting more out of a model.", "category": "comparison", "original_transcript": "And this idea that there are these two different parallel ways that you can get more out of a model, training time scaling and inference time scaling is something that has really come to the to the forefront in the last year or so, because previously it was all about training time. Everything was about a bigger model. Bigger is better, more parameters, bigger, more training. And in the last year, inference time scaling has really taken off. And you know, two years ago it was Rag, which is inference time. And and then in the last year it's been a lot about different reasoning techniques. And basically these are two different orthogonal tracks to getting more out of your model."}
{"question": "Why was the attention mechanism considered a simplification compared to LSTM?", "keywords": ["attention is all you need", "simplification", "parallelize"], "reference_answer": "The transformer's attention mechanism was a simplification compared to LSTMs, which were harder to parallelize because outputs from one step had to feed the next. The benefit of being able to parallelize and run things concurrently far overtook the fact that it was a simplification, which is why the paper was called 'Attention is all you need'.", "category": "comparison", "original_transcript": "The transformer in some ways was a simplification. It was it was not not doing all this clever stuff. And it turned out that the simplification that you got so much benefit from being able to parallelize and do things together concurrently, a lot of the time that it far overtook the fact that it was a bit of a simplification. But the reason, since it was a simplification, the reason for the title of the paper attention is all you need. Well, the authors were trying to say is, you know what? It turns out we don't need all this clever stuff. If you just have this much simpler construct called attention, it's actually good enough."}
{"question": "What is emergent intelligence in the context of neural networks?", "keywords": ["emergent intelligence", "enough scale", "imitate intelligence"], "reference_answer": "Emergent intelligence is the property that with enough scale \u2014 enough neurons and parameters hooked together \u2014 you start to create not only plausible tokens but ones that imitate intelligence very accurately. It still remains a bit of a mystery why this works so well.", "category": "conceptual", "original_transcript": "So that that phenomenon, the fact that with a big enough neural network, with enough of these kinds of these, these neurons, these little algorithms hooked up together that you get a not only likely tokens, but accurate tokens, intelligent tokens or tokens that that represent an intelligent response. It simulates its. It imitates an intelligent response very accurately. That still a bit of a mystery. And it's what people call the property of emergent intelligence. With enough scale, you start to not only create plausible tokens, but actually ones that imitate intelligence very accurately."}
{"question": "How does Gradio describe what a callback is?", "keywords": ["callback", "passing it in", "any point in the future"], "reference_answer": "A callback is a function name that you pass to Gradio as a variable rather than calling it directly. You're giving the function to Gradio so that Gradio can choose to call it at any point in the future when the user interacts with the interface.", "category": "conceptual", "original_transcript": "we're just passing it in as a variable. And that means that we're giving it to Gradio, so that Gradio can choose to call this function at any point in the future. It knows it as the function we want to be called in the future. And when you do that, it's known as as a callback. This thing shout is a callback that Gradio can call back to any times it wants."}
{"question": "How does Gradio's share=True option actually work behind the scenes?", "keywords": ["HTTP tunneling", "share equals true", "callback right here"], "reference_answer": "When you pass share=True, Gradio sends the front-end code to be hosted by Hugging Face but uses HTTP tunneling to reach back into your machine and call the callback locally when someone interacts. The same technology Ngrok uses lets Gradio call back to your computer.", "category": "technique", "original_transcript": "Gradio will still come back to your computer and call the the share the the shout callback right here on your machine, as we will see in just a second. And if you're wondering, like how on earth does that work? How does Gradio reach back into my machine. It's simply it uses an approach that technical people know as HTTP tunneling. If you've ever used something like Ngrok, it uses that kind of technology."}
{"question": "What three things does Gradio do to provide its magical UI experience?", "keywords": ["svelte", "Starlette", "back end route"], "reference_answer": "Gradio does three things: first, it generates a Svelte front end from your Python UI description; second, it starts a Starlette web server when you call launch that serves the generated front end; and third, it creates back end routes for each of your callbacks so the front end can call them.", "category": "technique", "original_transcript": "It actually uses svelte, the front end framework, which is very nice because it just just generates down to basic JavaScript. So so it uses this to generate a basic front end description of your website. And that's, that's like a generated code based on your high level description in Python, allowing us to describe a UI in Python that gets translated into simple JavaScript UIs."}
{"question": "Why does the instructor recommend using the chat completions API over the responses API?", "keywords": ["responses API", "locked in", "common standard"], "reference_answer": "The instructor recommends the chat completions API because the responses API is locked in to OpenAI's ecosystem and is not common across providers, while the chat completions API is the simple standard common API used across all the different models. Also, the extra functionality in responses API is mostly paid.", "category": "comparison", "original_transcript": "the second thing to be aware of is that responses API is something that is locked in to OpenAI's ecosystem. It's not something that's common and shared ubiquitously across all of the providers. So if you sign up for the responses API, then you're locking yourself in to the OpenAI ecosystem. And you don't want to do that because we're going to be using lots of different models, as people do in industry, and you typically don't want to have yourself be be locked in to one provider versus all of the others. So generally speaking, there are definitely times when it's good to use the responses API. But generally speaking, the chat completions API is the one to pick. The simple standard common API that's used across the different models."}
{"question": "Why use a Python generator with Gradio for streaming responses?", "keywords": ["generator", "yield", "iterate to display"], "reference_answer": "If you provide a callback function that is a generator with the yield keyword, Gradio automatically iterates over it and displays the results as they come. The generator should yield the cumulative response so far, and Gradio refreshes the UI with each yielded value.", "category": "technique", "original_transcript": "Gradio it will see that this isn't a function anymore. It's a generator, a special kind of function. It's a function that you can iterate over. And as a result, Gradio will automatically not just wait until it's finished and show the results, but iterate over it and surely, surely slowly show the response as it comes."}
{"question": "What does temperature control in an LLM call?", "keywords": ["probability distribution", "highest probability", "temperature of zero"], "reference_answer": "Temperature controls how the next token is picked from the probability distribution the model produces. A temperature of zero means always pick the token with the highest probability, while higher temperatures mean occasionally pick tokens with lower probabilities, leading to more variety in output.", "category": "technique", "original_transcript": "what is this temperature? I think it's come up from time to time and I'm not sure I've ever properly explained it. So so let me just take one minute on temperature right now. So temperature is it's probably something you've heard about a fair bit, because all of the LMS take temperature as an input, and it's something that controls how how much variety there is in the output that comes from the model. It's a parameter that controls how different it will be each time you call it. A temperature of zero means that it should be predictable, deterministic, and a higher temperature means there should be more variety, more more flavor, more differences in the answers."}
{"question": "Why does the instructor say that tools don't involve the LLM running code directly?", "keywords": ["LLM informs the code", "Python call", "conversation history"], "reference_answer": "In practice, the code makes a call to an LLM and the LLM responds saying it wants to run a tool. The code then makes that Python call itself, and then calls the LLM a second time with the answer added to the conversation history. The LLM never actually runs code.", "category": "conceptual", "original_transcript": "In practice. The code makes a call to an LLM that says, I want to do this and this and this. And with what comes back, the LLM informs the code in order to do that. I think we should run a tool, this tool, and it's our code that then goes out, makes that Python call, comes back with the answer, and then you call the LLM a second time with the answer and say, hey, this is the conversation history. I asked you a question. You responded and said, please call a tool. I called the tool. Here's the answer to that. Now please answer the question."}
{"question": "Why use JSON for describing tools to an LLM?", "keywords": ["JSON", "trained on a lot", "fixed format"], "reference_answer": "You describe tools to an LLM using JSON because models love receiving JSON \u2014 they have been trained on a lot of it. There's a fixed format of JSON that you use to tell the model it can use these tools, and because it's been trained with this JSON, it knows how to respond when calling a tool.", "category": "technique", "original_transcript": "I mentioned that when we call an LM and say, hey, you're able to use tools, we have to tell it what tools it can use. And the way we tell it is with JSON, because models love to receive JSON. They're trained on lots of it. And there's just a fixed format, a type of JSON that you have to use to provide it, to tell it that it can use these tools and that JSON, it's been trained on a lot of it. So it expects the JSON in this format."}
{"question": "How does the chat callback's history parameter relate to the OpenAI messages format?", "keywords": ["type equals messages", "history", "OpenAI's format"], "reference_answer": "When you set type='messages' in the Gradio chat interface, the history passed to the callback is in a format very similar to OpenAI's messages format. It's a list of dictionaries with role and content fields, making it convenient to pass directly to OpenAI.", "category": "technique", "original_transcript": "When Gradio calls this, uh, this, this callback, it passes in the message that you just sent and history, it passes in the contents of this user interface, the contents of the backwards and forwards. And it does so in the format that's very similar to OpenAI's format. And that's actually what this little flag here type equals messages is saying, hey, Gradio, when you call me back, can you use the OpenAI messages format?"}
{"question": "What was the purpose of using stable diffusion XL Turbo in the colab demo?", "keywords": ["stable diffusion", "turbo", "diffusion model"], "reference_answer": "The Stable Diffusion XL Turbo from stability is not a transformer but a diffusion model, a different kind of model good at generating images. It was used as the first hugging face library demo to generate an image of a class of students learning AI engineering in vibrant pop art style.", "category": "conceptual", "original_transcript": "We are going to be using, uh, something that isn't going to be using a model, a model from the Hugging Face library. But it is not a transformer. It is actually a different kind of model. It is a diffusion model that you may have heard of. It is a kind of model that's good at generating images. And we're making one from a company called stability, uh, which is, uh, the, uh, the, the stability diffusion model, and it's called Excel Turbo."}
{"question": "What is the difference between the hugging face hub and hugging face transformer libraries?", "keywords": ["hub", "Python library", "libraries"], "reference_answer": "The hub is the hugging face website with models, datasets and spaces. The libraries are the open source code packages you use in Python \u2014 including one confusingly called the hugging face hub library, which is the Python library that lets you connect to the hub to download models and data sets.", "category": "comparison", "original_transcript": "The first library that I will mention to you is a library called the Hugging Face hub, which you may remember I told you was the name of the other thing. That Hugging Face does the website and you were like, what? And the reason is because this is a Python library that lets you connect to the hugging face hub to download the models or the data sets that you can see on Hugging Face Hub and have them in your code immediately."}
{"question": "Why does Google Colab give access to GPUs in the cloud?", "keywords": ["matrix calculations", "graphics cards", "linear algebra"], "reference_answer": "Modern AI predicting the next token involves taking billions of parameters and multiplying them through the transformer architecture. This is lots of matrix math \u2014 linear algebra \u2014 and graphics cards were originally designed as custom hardware for matrix calculations in parallel, which turned out to be perfect for AI.", "category": "conceptual", "original_transcript": "you're doing lots of multiplications and adding multiplications and adding is basically matrix math, linear algebra. It's lots and lots of matrix calculations, and they can all happen very efficiently in parallel with one another. They're all sort of independent. So they can all run together in this very, very efficient way. But they all have to run. And it turns out when, when all this is becoming a big thing, in the early 2000, people were saying what we really need is like specialized custom hardware that is that is efficient, that is designed for matrix calculations in in fast, real time."}
{"question": "What is the Tesla T4 GPU available for free on Colab?", "keywords": ["Tesla T4", "15GB", "free"], "reference_answer": "The Tesla T4 is the typical GPU available on Google Colab. It has 15GB of GPU RAM, which is a lot, and it is completely free. You can connect to a Colab with a T4 GPU at any point at no cost.", "category": "metric", "original_transcript": "It's still pretty mighty. It's still a powerful GPU. The typical one is the Nvidia Tesla T4. The T4 is the typical one. It has 15GB of GPU Ram, which is a lot. And here's the thing. It's free. It's it's completely free. You can have a Google Colab with a T4 GPU, and you can have it for free at any point. And we're just gonna do it in just a second. Can you believe it?"}
{"question": "What is the pipelines API in hugging face used for?", "keywords": ["pipelines", "high level API", "inference tasks"], "reference_answer": "Pipelines is a high-level API from Hugging Face that lets you carry out common inference tasks very easily without getting into the low-level details. It's for when you want to do an out-of-the-box, pre-configured inference task using a model that has already been trained.", "category": "conceptual", "original_transcript": "And the higher level one is called pipelines. It's a high level API. It lets you carry out common tasks very easily. The lower level one is is talking about tokenizers and models. It's the real granular API where you get to work with the underlying deep neural network, and it gives you the most power and the most control pipelines. Quick and simple. Get running fast."}
{"question": "What does CUDA refer to when setting up a hugging face pipeline?", "keywords": ["Cuda", "Nvidia", "parallel compute"], "reference_answer": "CUDA is the name of the specific technology invented by Nvidia that runs on graphics cards and allows you to do massively parallel math. It's like a programming language similar to C++ but intended for parallel compute. You specify device='Cuda' if you want a pipeline to use an Nvidia GPU.", "category": "conceptual", "original_transcript": "And device is where you can say if you want it to use a GPU and we do. And the device you specify if you want to use an Nvidia GPU like we have is Cuda. Cuda is the name of the specific technology invented by Nvidia, which runs on graphics cards, which allows you to do this maths in parallel. Massively. Uh, it's like a it's a programming language. It's it's similar to C++, but it's intended for this kind of parallel compute."}
{"question": "What is the small idea behind RAG?", "keywords": ["knowledge base", "relevant details", "shove it in the prompt"], "reference_answer": "The small idea behind RAG is to put relevant details in the prompt by querying a knowledge base \u2014 a database of useful information. Every time the user asks a question, you do a query and shove whatever relevant information you find into the prompt before sending it to the LLM.", "category": "conceptual", "original_transcript": "And I said here we put relevant details in the prompt. Maybe some of it isn't relevant, but but the LM doesn't mind. You say to it, here's some extra information. Maybe the question pertains to this. If so, then then use this when you're answering your question, and then you just shove the relevant information in there in the prompt. That is the basic motivation behind Rag. Nothing more complicated than that."}
{"question": "What is RAG fundamentally according to the instructor?", "keywords": ["big old hack", "performance trick", "hand-wavy"], "reference_answer": "The instructor describes RAG as a big old hack \u2014 a performance trick that uses an encoder and vector similarity to zoom in on probably relevant content. It's a hand-wavy way of throwing out stuff that's probably not important rather than letting the transformer figure out what matters.", "category": "conceptual", "original_transcript": "The first thing is that rag is basically a big old hack Transformers we built, transformers. They have they have layers of the neural network which which are responsible for figuring out how to pay attention to what matters in the input. Uh, and that's what they're good at. And they've been trained. They've been they've been really good at doing that. Our problem is that we've got too much information, and we don't want to waste our powerful transformer spinning his wheels, trying to figure out that only this little tiny bit is important. So we're using Rag. We're using this encoder and vector similarity as a kind of hack, just to say, ah, we can probably zoom in on this piece and that's probably all that matters."}
{"question": "Why do we split documents into chunks for RAG?", "keywords": ["chunks", "fragment", "one part of one document"], "reference_answer": "Documents contain different bits of information, but when a user asks a question, it will probably pertain just to one part of one document. Splitting documents into chunks gives the best chance that one particular fragment matches a question, rather than trying to match an entire document.", "category": "conceptual", "original_transcript": "And the reason is that imagine we've got these documents that they're going to have all sorts of different bits of information on them. We've already seen there's like employee HR records. There's stuff about products. When a user asks a question, it will probably pertain just to one part of one document. And if we just built one one vector associated with a whole document, then it's much less likely that a question is going to match directly with like an entire document like that. So it makes sense for us to try and think about how could we split up each document so that we've got the best chance that one particular fragment is something which can address a few different questions"}
{"question": "What is the difference between an encoder and a vector store?", "keywords": ["encoder", "vector store", "two different things"], "reference_answer": "These are two different things. An encoder or embedding model is what turns text into a vector. A vector store like Chroma is the database where vectors are stored, which is good at quick retrieval of vectors close to a given query vector. The vector store does not create the vectors.", "category": "comparison", "original_transcript": "I do want to distinguish very carefully, and this is going to be a bit of a recurring theme today between these two different steps. Turning text into a vector that uses an encoder, and then storing that vector in a vector database such as chroma, which is one we'll be using. A vector database is a particular type of database that's very quick at doing queries, not SQL queries. In a way you may be familiar, but queries about which piece of data has a vector closest to this one, so they are quick at vector based retrieval. That's their point, but they don't themselves come up with the vectors."}
{"question": "What is the recursive character text splitter?", "keywords": ["recursive character text splitter", "hierarchy", "natural break"], "reference_answer": "The recursive character text splitter tries first to split a document by where there are a couple of empty lines between sections, then by one empty line, then by end of sentences. It has a hierarchy of how to split documents to try to break things at natural breaks in the content.", "category": "conceptual", "original_transcript": "There's a slightly more sophisticated version called the recursive character text splitter. Again, it sounds it sounds like it's something fancy, this one, but it's not. It's it's recursive and it just tries to. It first tries to split things by wherever there's a couple of empty lines between sections, and then it tries to do it by just one empty line and then just by end of sentences and things like that. So it has like a sort of hierarchy of how it tries to split up documents so that it's doing its best job at trying to break it where it thinks there's a natural break in the documents"}
{"question": "What is the difference between recall and precision in RAG evaluation?", "keywords": ["recall", "precision", "proportion"], "reference_answer": "Recall is about the proportion of tests where relevant context appears within the top k retrievals. Precision is about what proportion of the chunks you surface are actually good or relevant. In RAG, recall is typically more important than precision because LLMs handle irrelevant context reasonably well.", "category": "comparison", "original_transcript": "So recall is the proportion of the retrievals. Precision is about saying what proportion of the k of the chunks focus on the chunks themselves, what proportion of them are good, and how often are you surfacing stuff? That's that's not good. That's irrelevant content. That is precision. How precise is your ability to retrieve context? So that's the difference between recall and precision recall against the proportion of tests where relevant context within the top k."}
{"question": "What is query expansion as an advanced RAG technique?", "keywords": ["query expansion", "bunch of them", "k times n"], "reference_answer": "Query expansion is when instead of just one RAG query, you ask the model to write a bunch of different queries and collect similar documents for each one. This gives you k times n chunks \u2014 the amount retrieved per query times the number of queries \u2014 and can surface more relevant context.", "category": "technique", "original_transcript": "Query expansion. This is this is one that a lot of people use. This is just saying. Oh, okay. Look, um, um, we could we could, uh, add to step five. Don't just ask the model to write one query. Ask it to write a bunch of them, a bunch of things that we could call our database lookup for. Let's have three different Rag queries and collect similar documents for each. That's called query expansion. Super common."}
{"question": "What is hierarchical RAG and what problem does it address?", "keywords": ["hierarchical rag", "Achilles heel", "summary of all"], "reference_answer": "Hierarchical RAG addresses RAG's Achilles heel of being bad at answering questions that span many documents. The technique involves summarizing your knowledge base at various levels of rollup \u2014 a summary of all products, a summary of all employees \u2014 and doing a RAG lookup across summaries first to get coarse information.", "category": "technique", "original_transcript": "Number eight, we're on the home stretch hierarchical rag. This is trying to fix a particular Achilles heel with rag that I mentioned before, which is that rag is really bad at answering questions which span lots of documents. It has a really hard time with that. And so there are some, pretty simple, uh, hacks that try and get around this. And one of them is to summarize your knowledge base at various levels of roll up, like a hierarchy, have a kind of a summary of all products, a summary of all employees."}
{"question": "What is reranking in advanced RAG?", "keywords": ["Reranking", "reorder", "most relevant"], "reference_answer": "Reranking is taking the chunks you've retrieved and using an LLM to reorder them in terms of most relevant for the question at the top, all the way down to the least relevant. You can then potentially chop off the unimportant ones before sending the chunks to the final LLM to answer.", "category": "technique", "original_transcript": "Reranking Reranking is saying we might have ended up with lots of different chunks from our data store, particularly if we've done number six query expansion, we might have a lot. We will have people like to say we have k times n, you have k the, the, the, the amount that you retrieve every time times n the number of queries you've got a potentially a bunch of chunks. Well, we could then take all of these chunks and try and reorder them in terms of the ones that's most relevant for the question at the top and all the way down to the least relevant for the question."}
{"question": "What is document pre-processing as an advanced RAG technique?", "keywords": ["document pre-processing", "rewrite", "suitable for the data store"], "reference_answer": "Document pre-processing is when, before putting documents into the vector data store, you first rewrite them through an LLM so they are more suitable for the data store. For example, a table of numbers gets converted into text format that will match a vector lookup much better.", "category": "technique", "original_transcript": "document pre-processing this. This. This is about saying, okay, when we're when we're taking our documents and we're about to put them into our, into our vector data store, maybe first we could rewrite our documents in a way that is going to make them more suitable for the data store. For example, if we had like a table of ticket prices that for traveling from one location to another, it's just like a table of numbers. When you turn that into a vector, it's not going to be very good because it's just going to be like, this is a table of numbers."}
{"question": "Why are evaluations or evals important when working with RAG?", "keywords": ["evals", "North Star", "metric"], "reference_answer": "Working with LLMs is very experimental and iterative, and that can feel unsatisfying. Evals make it satisfying by giving you a quantitative way to measure your setup's performance. The metric becomes your North Star \u2014 you keep iterating until that metric is higher, giving procedure around the alchemy.", "category": "conceptual", "original_transcript": "The way to make it satisfying is to build evals, find a way to measure in a quantitative way the performance of your setup. Then you can experiment, you can iterate, and you have a way to gauge whether you're getting better or worse in a way that you've decided you've signed up for. This is going to be my North Star. This is my metric. I'm going to keep iterating until this metric is higher, and that that gives you a technique. It gives you a procedure around the alchemy."}
{"question": "What is the LLM as a judge approach for evaluating answers?", "keywords": ["LLM as a judge", "reference answer", "score"], "reference_answer": "LLM as a judge is when you ask another LLM to evaluate whether the produced answer is good in the context of the reference answer. You basically say to another LLM: here's an answer, here's a reference answer, please score this answer compared to the reference. LLMs are very good at this task.", "category": "metric", "original_transcript": "It's called LLM as a judge, which is when you ask an LLM to to to think about this and to say whether or not this is a good answer in the context of the reference answer. So you're basically saying to another LLM, hey, here's an answer. Here's a reference answer. Please score this answer compared to the reference answer. And Llms are very good at that. And it's a relatively easy task for an LLM. And it's always good. It's recommended to use a stronger LLM as your judge."}
{"question": "Why are LangChain abstractions like LLM and Retriever both runnables?", "keywords": ["invoke", "retriever", "runnable"], "reference_answer": "In LangChain, if you have an LLM you can call its invoke method, and if you have a retriever you can also call invoke. These shared method names mean they are called runnables in LangChain. Other objects like tools also support invoke, making the framework consistent.", "category": "conceptual", "original_transcript": "And as with many of lang chains abstractions, it responds to a method invoke. If you have an LLM, you can say my LLM dot invoke. If you have a retriever, you can say my retriever dot invoke. And there are many other objects in Lang chain that have this same behavior, uh, tools. Uh, and many more. That's it's a very common part of the framework. It's actually so it's called a runnable in Lang chain if you can call invoke."}
{"question": "What is generalization and why is it the most important word in AI?", "keywords": ["generalization", "unseen data", "patterns"], "reference_answer": "Generalization is the ability of a model to make accurate predictions on data it hasn't seen before. The model learns from training examples and applies the patterns it has learned to new, different inputs. It's what all of AI is striving for \u2014 building models good at generalizing.", "category": "conceptual", "original_transcript": "And this leads me to the single most important word in all of AI. And that is the word generalization, the ability of a model to make an accurate prediction on data that it's not seen before. To make to give an output for a given input, where that input is different to anything it's seen before, if it's able to do that well, you would say the model is able to generalize. It's seen some examples. It has generalized. And now when it's given a new example it does well."}
{"question": "What is transfer learning in the context of fine tuning?", "keywords": ["transfer learning", "pre-trained", "business specific data"], "reference_answer": "Transfer learning is the property that lets you take a model someone else has already pre-trained and train it a little bit more with business specific data. As long as you don't use too much, the model retains the original information it was trained on and also gets better at a particular business task. This is fine tuning.", "category": "conceptual", "original_transcript": "we can take advantage of a property of of these models known as transfer learning. And that property says that actually what you can do, which which turns out to be really effective is you can take a model that someone else has already trained. They did their training and it's done. It's known as pre-trained. It's it's been trained. And you can actually train it a little bit more with some business specific data. And as long as you don't use too much, you can make it so that it retains the original information it was trained on. And it just also gets better at a particular business task."}
{"question": "What is overfitting in machine learning?", "keywords": ["overfitting", "noise in the data", "destroys generalization"], "reference_answer": "Overfitting is when a model has tried to be so accurate in fitting the training data that it has done too well \u2014 learning things about the noise or anomalies that aren't real patterns in the real world. As a result, the model performs badly on new unseen data. It's the thing that destroys generalization.", "category": "conceptual", "original_transcript": "Overfitting is what you call it when you've given it lots of training data, and maybe the training data has some noise in it, or maybe it has some anomalies and your model has has tried to be so accurate in fitting to the training data that it's done kind of too well. It's learned things about the training data that aren't real patterns in the real world. And so that's known as overfitting when it does that and and the the it wouldn't matter if it only affected the training data. The problem is that typically if a model overfits, then when it's given new unseen data, it starts to perform badly because it's, it's it's tried to account for the noise in the data or something that isn't real signal. It's tried to explain that, and that means it's not going to perform well when it's given something new. And so this is known as overfitting. And it's the kind of like the thing that destroys generalization"}
{"question": "Why does the instructor advise starting with a baseline model?", "keywords": ["baseline", "start simple", "state of the art"], "reference_answer": "Starting with a baseline model is great discipline. You need to have something to compare with to be able to judge how well you're performing and whether you are state of the art. The rule of start simple and build from there is so important \u2014 sometimes a simple traditional model beats fancier LLMs.", "category": "conceptual", "original_transcript": "people often do say let's just pull out the LLM. And that that would be an error. It's a great discipline in any in any proper commercial project you would want to start start simple. The rule of start simple and build from there is so important. Uh, even even even if you absolutely know that this is going to be a great model for you, you still need to have something to compare it with, to be able to to judge how well you're performing and whether you are state of the art."}
{"question": "What does dropout do in a neural network?", "keywords": ["dropout", "10%", "zeroed out"], "reference_answer": "Dropout is a technique where some percentage of neurons \u2014 most commonly 10% \u2014 randomly get wiped out or zeroed out during training, and a different 10% each time. This forces the model to generalize and not overfit by ensuring the network as a whole solves problems rather than just localized parts.", "category": "technique", "original_transcript": "what happens is that randomly 10% of these neurons of these little calculations, these these mini linear regression units inside a deep neural network get wiped out, get zeroed out. When you're passing through, when you're doing your training and a different 10% each time and you might be thinking, okay, what? Why that sounds sounds very hacky. Um, a lot about data science is very hacky. So why why exactly do you do that? Well, it's a technique to try to make sure that you generalize well, that your neural network doesn't overfit to the data"}
{"question": "What is a learning rate and why shouldn't it be too small?", "keywords": ["learning rate", "local minimum", "small step"], "reference_answer": "Learning rate is how big a step you take each time after computing the gradients. If learning rate is too small, you risk getting stuck in a local minimum \u2014 finding a small valley near where you began but never reaching the global minimum next door. A larger learning rate would help escape local minima.", "category": "conceptual", "original_transcript": "And that problem is known as being stuck in a local minimum, which is an expression that people use in the real world to when they're debating things sometimes and what that means. You probably already have a good picture of what that means. imagine if the truth is not this nice one valley, but the truth is a valley like that. And next to it is a much bigger valley. And the real lowest loss point is down here. That's what we want to get to. Well, if learning rate is too small, there is this risk that if we start over here somewhere, we might sort of find our way down to that minimum right there."}
{"question": "What does batch size mean and what's the rule of thumb for setting it?", "keywords": ["batch size", "fit in the memory", "powers of two"], "reference_answer": "Batch size is how many data points you put in a batch during training. The basic rule of thumb is to pick the biggest batch size that will fit in the memory of your GPU. People also love using powers of two for batch size like 16, 32, 64, 128.", "category": "technique", "original_transcript": "And then that ties nicely to batch size. Batch size is how many of these data points do you put in a batch? And basically you try and do as much as you can to fit in the memory of your GPU. That's really the rule of thumb that people use. You keep a bigger and bigger batch size until your GPU couldn't fit it. And then and then you reduce down. Now there are the more the pro people will point out that it's more complicated than that. There are some times you obviously you. The ideal case is that you only have a batch of one, that you train one data point at a time, but that would take far, far too long."}
{"question": "What is the inference time constrained decoding trick used for structured outputs?", "keywords": ["constrained decoding", "zero out the probability", "JSON spec"], "reference_answer": "Inference time constrained decoding is a clever trick where OpenAI looks through all possible next tokens and zeros out the probability of any token that would break the JSON spec. As a result the model is constrained to only generate tokens that stay in line with the spec, guaranteeing the output conforms to the schema.", "category": "technique", "original_transcript": "And there's a trick that's called inference time constrained decoding. That's the technical name for it. And even from those words you probably already know what I'm going to say. Basically when when the model generates this probability distribution, OpenAI have written some Python code that says, hang on a second. This model needs to conform. This needs to generate something which conforms to a particular JSON spec. Let me look through all the possible next tokens and see if there are any tokens here that would break the spec a token. That would mean that what comes out wouldn't conform to the spec, and I would simply zero out the probability so that the model, whatever happens, it's not going to generate a token that will break the JSON spec."}
{"question": "What is the agentic trap the instructor warns about?", "keywords": ["agentic trap", "human like responsibilities", "humanizing"], "reference_answer": "The agentic trap is when people are too quick to divide their problem into steps and give human-like responsibilities to those steps with agent names, doing it because that's what human roles would be like rather than because it actually helps solve the business problem. The instructor urges starting with understanding the problem first.", "category": "conceptual", "original_transcript": "And this also has that trap I mentioned before, the agentic trap, when people are too quick to divide up their problem into steps, particularly when they give human like responsibilities to those steps with, with agent names and, and they haven't done it in order to break a problem down. They've done it because that's what. What human roles would be like. And and that might turn out that that's fine. But I just do suggest that you start by understanding the problem. You divide up into smaller LMS in order to solve the problem."}
{"question": "What are the three definitions of agents the instructor lists?", "keywords": ["work for you", "controls the workflow", "tools in a loop"], "reference_answer": "The first is the Sam Altman definition: AI systems that can do work for you independently. The second is that an LLM controls the workflow. The third, the emerging definition, is an LLM running tools in a loop to achieve a goal.", "category": "comparison", "original_transcript": "Sam Altman OpenAI for a while was putting out there. It's AI systems that can work independently. The consensus early in 2025 was that it's about situations where an LLM controls the workflow, and then the kind of emerging definition is that it's where you have an LLM in a loop with tools to achieve a goal."}
{"question": "What is an ensemble agent in the price prediction project?", "keywords": ["ensemble", "multiple different models", "random forest"], "reference_answer": "An ensemble agent is one that brings together multiple different models to figure out how much something is actually worth in the market. The word ensemble comes from the same idea as random forest models, which are themselves ensembles of multiple decision trees.", "category": "conceptual", "original_transcript": "There's going to be an ensemble agent a new word. That's a very exciting word. This is something which is able to figure out how much is something actually worth in the market. And it's called an ensemble because it brings together multiple different models. You may remember we met that word when we were talking about random forest models, which are themselves ensembles. Well, that is going to be our ensemble and it's going to be great."}
{"question": "What is modal and what makes it good for AI inference?", "keywords": ["modal", "serverless", "only pay for the compute"], "reference_answer": "Modal is a serverless platform for AI teams that is incredibly easy to set up \u2014 you define your infrastructure with a little code. With Modal you only pay for the compute that you use, so you spend nothing when the platform is dormant. They also give $30 of free credits a month.", "category": "conceptual", "original_transcript": "modal is a is a really terrific platform for running your models, uh, for for inference on the cloud. They do other things too. But but running the inference is one of their most, most commonly used features. And it's how I use them in my day job. And many people do. There are a bunch of others like modal out there and there's there's for AWS, there's SageMaker, which is their big platform. Um, there's there's uh, run pod. There's there's many things like this where modal really sets itself apart. Is that, first of all, it's incredibly easy to set up and configure, unlike some of the others. When you define your infrastructure, you just do it through a little bit of code, and that's where you set up what kind of machine you want and all sorts of features about the platform. So it's really convenient. Also, with modal, you only pay for the compute that you use."}
{"question": "How does pydantic make structured outputs work behind the scenes?", "keywords": ["pydantic", "JSON schema", "subclass of base model"], "reference_answer": "Pydantic is all about defining a JSON schema. When you create a subclass of base model, you can convert it to JSON and back. OpenAI converts your pydantic class to a JSON schema and adds it to the system prompt telling the model the response must conform to that schema.", "category": "conceptual", "original_transcript": "The reason why you make a subclass of a pedantic base model is that what pedantic is all about is defining a JSON schema. If you if you have something that's a that's a subclass of base model, you can convert it to JSON and you can convert the JSON back to the object again. So the pedantic class that you set up is used to generate a JSON schema, something which says that the JSON must conform to the following spec and it lays it out. It needs to have these fields. And you put like a natural language description by your fields, and that gets included in the JSON schema."}
{"question": "Why did the instructor pin the datasets library to version 3.6.0?", "keywords": ["pinning a version", "force it to use", "things change"], "reference_answer": "The instructor pinned the datasets library to version 3.6.0 because hugging face released an update that broke compatibility, and pinning forces the install to use the specific version that works. This process is called pinning a version, and it's a common activity because over time things change and people break things.", "category": "technique", "original_transcript": "but I want to force it to use version 3.6.0, which is a version from about six weeks ago. Because after that, hugging face just released an update and they changed a bunch of stuff, which means that some some of the hugging face code doesn't doesn't work with that version, blah blah blah. This process is known as pinning a version. It's pinning data sets to 3.6.0. And when you look in this cell, you might see that I've pinned lots more and you might have more in future colabs. And that's because over time, things change and and people break things all the time."}
{"question": "Why is Q LoRA used for fine tuning instead of building a model from scratch?", "keywords": ["Q Laura", "open source model", "fine tuned"], "reference_answer": "When people say 'built my own model from scratch,' they usually mean they took an open source model and fine tuned it using a technique like Q LoRA, which is the most popular technique. It's much more practical than building from scratch because Frontier Labs will release a better base model quickly, making any from-scratch approach expensive to maintain.", "category": "conceptual", "original_transcript": "Normally when people say and every day we built an LM like we have an LM. In my day job, what it means is that you took an open source model and you fine tuned it using a technique like Q Laura, which we'll use, which is, I think, the most popular technique, uh, using techniques like that to fine tune a model to get better at your business objective. That's what it means."}
{"question": "What is loss in machine learning training?", "keywords": ["loss", "how bad is your model", "minimize loss"], "reference_answer": "Loss is a measure of how bad your model is during training, and the goal is to adjust training to minimize loss. In traditional data science a common loss metric is MSE \u2014 mean squared error \u2014 which is the difference between prediction and ground truth squared. For LLMs the most common is cross entropy loss.", "category": "metric", "original_transcript": "And when you're dealing with training, the normal thing that you look at is called loss, which is a measure of how bad is your model. And you're normally trying to adjust training to minimize loss. And if you come from a traditional data science background, then one of the go to loss metrics you start out with is called MSE mean squared error. And it's as simple as you take your prediction from your model, you subtract the ground truth and you square it."}
{"question": "What does the GPT acronym stand for in detail?", "keywords": ["generative", "pre-trained", "transformer"], "reference_answer": "GPT stands for Generative Pre-trained Transformer. The G is generative, meaning the model generates tokens. The P stands for pre-trained, in that it's been trained off tons of data scraped from the internet. The T is for transformer, the architecture used by these models.", "category": "conceptual", "original_transcript": "you're probably familiar with the fact that GPT stands for generative is the G, which means that it's a model which is responsible for generating tokens or predicting what should come next after a sequence of inputs. It's P stands for pre-trained, in that it's been trained off tons and tons of data scraped from the internet and from all sorts of sources. Controversially, that is the P in GPT. And of course the T is transformer. GPT, the generative pre-trained transformer."}
{"question": "What is an open weight model versus an open source model?", "keywords": ["open weight", "training data", "made available"], "reference_answer": "Some say models like Llama aren't really open source \u2014 they are open weight, because Meta has made the weights of the model available, but they haven't made available the training data and methodologies used to build the model. Still, most people call them open source models anyway.", "category": "comparison", "original_transcript": "The people who feel strongly about this stuff sometimes say that they're not really open source. They are open weight because meta has made available all of the weights of the model, but they haven't necessarily made available the training data and methodologies that was used to build the model. But still, we tend to call them open source models."}
{"question": "What is distillation in the context of open source models?", "keywords": ["distillation", "synthetic data", "smaller variants"], "reference_answer": "Distillation is a process where you take a big model like the main deep seek and use it to generate tons of synthetic data. Then you train a smaller model \u2014 like a small Llama or Qwen \u2014 on that synthetic data to try to replicate the larger model's behavior. There are various ways of doing distillation.", "category": "technique", "original_transcript": "They've taken small versions of Llama and Quen, and then they've trained it more with synthetic data that was generated by the big deep seek. So again, they used big deep seek to generate lots of fake data, synthetic data from deep seeks training. And then they've used that to educate these smaller Lama and Shen models. And that process is known as distillation. There's actually various different ways of doing distillation, but that's one of them."}
{"question": "What is the difference between Grok with a Q and Grok with a K?", "keywords": ["grok with a Q", "Elon Musk", "proprietary hardware"], "reference_answer": "Grok with a K is Elon Musk's xAI model. Grok with a Q is a vendor that runs open source models on the cloud using custom built proprietary hardware that is super fast. Grok with a Q has been around since 2017 or 2018, so it was actually Elon Musk who copied their name.", "category": "comparison", "original_transcript": "There's one super confusingly that's called grok spelt with a Q that is different to Elon Musk's grok spelt with a k. So we'll use both grok in due course. But but watch out for that one too. That's a different thing. That is it's basically an ability to run open source models super fast on the cloud using proprietary hardware."}
{"question": "What is the mixture of experts architecture?", "keywords": ["mixture of experts", "smaller models", "specialist"], "reference_answer": "A mixture of experts model contains lots of smaller models within it, and it directs traffic to different specialist models depending on the kind of question you're asking. Models like Mistral are called mixture of experts because they route queries to specialists.", "category": "conceptual", "original_transcript": "One of them is called Mistral because it's a type of model called a mixture of experts, which means that it has lots of smaller models within it, and it directs traffic to different specialist models depending on the kind of question you're asking. So that's Mick's trial."}
{"question": "What is multi-shot prompting?", "keywords": ["multi-shot prompting", "examples", "system prompt"], "reference_answer": "Multi-shot prompting is when you insert multiple concrete examples of questions and good answers into the prompt \u2014 usually the system prompt. You use that to bias the model so that as it predicts next tokens it's more likely to predict something consistent with your examples.", "category": "technique", "original_transcript": "And Multi-shot prompting is when you insert into the system prompt. Again, it could be the user problem, but usually the system prompt. Concrete examples of if you get this question, this is how you should answer, and you use that to bias the model so that as it's predicting what should come next, it's more and more likely to predict something consistent with the examples that you gave in the system prompt. That's how it works."}
{"question": "What is the role of 'tool' in conversation history when calling tools?", "keywords": ["role tool", "tool call ID", "second time"], "reference_answer": "When calling tools, in addition to system, user, and assistant roles, there's a role called tool. After the LLM returns a message asking for a tool call, you add a new message with role tool whose content is whatever came back from the function call, and a tool call ID matching the request. This is then sent in the second call to the LLM.", "category": "technique", "original_transcript": "we have user, we have system, user and assistant as the possible role values. There's another one. There is role tool. And that's what I've got here. Role tool. And the content is whatever came back from the call. So whatever came back from our function we shoved that as another message in the set of messages with role of tool."}
{"question": "Why did the instructor avoid streaming when implementing tool calling?", "keywords": ["streaming", "really janky", "piecing together the JSON"], "reference_answer": "Streaming when supporting tool calling is perfectly possible but really janky. You have to do all sorts of nonsense to check whether the finish reason is tool calls and to piece together the JSON to call the tool. The instructor avoided this because it would be too hacky and messy, and instead used a simpler non-streaming approach.", "category": "conceptual", "original_transcript": "I just printed them. I used a simpler thing and there was a reason for that. It turns out that streaming back when you're also trying to support tool calling is perfectly possible, but it's really janky. You have to do all sorts of nonsense to see whether the finished reason is tool calls, and to be like piecing together the JSON to call the tool. You can you can Google it and you'll see that I think on OpenAI's forum there's some examples of how to do it, but it's it's messy, there's lots of code."}
{"question": "What is a hybrid model in the context of LLMs?", "keywords": ["hybrid model", "decide how much thinking", "reasoning"], "reference_answer": "A hybrid model is a variation on the reasoning thinking model that's able to decide how much thinking it does. In some modes it's much more similar to just a chat model and hardly does any reasoning, while in others it will reason a lot, deciding based on the question how much to reason. GPT five and Gemini 2.5 Pro are hybrid models.", "category": "conceptual", "original_transcript": "And in some modes it's much more similar to just being a chat model. It hardly does any reasoning at all. And in others it will go ahead and reason quite a lot, and it decides how much to reason based on the question you ask how how much of a puzzle it is, or if it's just in a chat mode, kind of just just a simple hi there, then it won't bother reasoning the answer to that. And so this kind of model that's able to decide how much thinking to do is what's known as a hybrid model."}
{"question": "What does the OpenAI Python client library actually do under the hood?", "keywords": ["client library", "wraps", "HTTP endpoint"], "reference_answer": "The OpenAI Python client library is a Python client library that wraps a call to an HTTP endpoint. It's lightweight code that manufactures an HTTP request to an endpoint and turns the JSON response into Python objects. It's just vanilla code that wraps a web request.", "category": "conceptual", "original_transcript": "It's a Python client library that wraps a call to an HTTP endpoint. It's very simple. It's completely open source. You can open it, look at it, look at all the code. It's perfectly simple. Some people, the first time they come across this, think that when you're dealing with OpenAI, the object and the code, that somehow we're sort of running GPT and we have some some fancy code from OpenAI. Not at all. It's vanilla code that just wraps making a web request."}
{"question": "What is the chinchilla scaling law concept of needing more parameters?", "keywords": ["diminishing returns", "double the number", "scale your training data"], "reference_answer": "When training a model, if you reach diminishing returns on training data with the current parameter count, you'll need to double the number of parameters in your neural network to take on twice as much training data. The scaling law relates parameters to the amount of training data you can absorb.", "category": "conceptual", "original_transcript": "imagine that you're trying to train a model with, with with 8 billion parameters, and you start to find that you're let's say you're halfway through training your training data and you're seeing that you're getting diminishing returns. Your model's not doing any better. It seems to be sort of stuck at the halfway point in your training data, and you've got like, like double as much to go. Then what you have to think is, okay, that's my sign that I need to double the number of parameters."}
{"question": "Why doesn't the instructor recommend using LLMs to convert PDFs to markdown?", "keywords": ["horrible abuse", "Python code", "converting between different file formats"], "reference_answer": "Using an LLM to turn a PDF into markdown is a horrible abuse of LLMs \u2014 it's not what they're good for. Python code is what's good for that. There are libraries that you can pip install or uv add that are fantastic at parsing PDFs. Converting between different file formats is a solved problem and doesn't need artificial intelligence.", "category": "conceptual", "original_transcript": "And you know what? I would not use an LLM to turn a PDF into markdown. That that is a horrible abuse of using Llms. It's not what they're good for. You know what is good for that Python code? Remember, we were software developers once. There's plenty of libraries that you can pip install or you've add in our case that are fantastic at parsing PDFs, and you can convert them to, to markdown or to just to text or whatever makes sense for your business case. Converting between different file formats is a solved problem."}
{"question": "What is semantic chunking in advanced RAG?", "keywords": ["semantic chunking", "meaning", "meaningful way"], "reference_answer": "Semantic chunking is when you chunk text not by randomly breaking it into sections, but based on the meaning of each block of text. You divide up the page in a meaningful way so that the document is broken into chunks that reflect, for example, the different stages of someone's career or different facets of a product.", "category": "technique", "original_transcript": "A very similar is something I didn't actually mention with number one, which is when you're chunking, there's a type of chunking called semantic chunking, which is when you chunk not just based on like randomly breaking things up into different sections, but based on the meaning of each each block of text. Trying to divide up your your page in a sort of meaningful way, such that your document is broken up into different chunks that reflect maybe the different stages of someone's career or the different facets of a product or something like that."}
{"question": "What is the difference between an arena and a leaderboard for comparing models?", "keywords": ["leaderboards", "arenas", "head to head"], "reference_answer": "Leaderboards are tables that rank different models against different criteria like coding, language understanding, or reasoning. Arenas are places where models are put head to head, and the instructor describes them as fun places to look when comparing different models.", "category": "comparison", "original_transcript": "you can look at them on leaderboards, which is which are the sort of tables which rank the different models against different criteria. And there are also arenas where models are put head to head, which are some fun places to look. And we will look at all of this over the next couple of days."}
{"question": "What does it mean when an LLM has a knowledge cutoff?", "keywords": ["knowledge cutoff", "training ends", "limited knowledge"], "reference_answer": "Knowledge cutoff is when the model's training goes up until and beyond that point it has limited knowledge. The model was trained at some point in time and the date when the last day was trained is known as the training cut off \u2014 beyond that, the model has no built-in information.", "category": "conceptual", "original_transcript": "They have this famously. This knowledge cutoff is when their training goes up until and they have limited knowledge beyond that point. So like if Gemini is is helping fix your code, it will often do this janky thing when it changes your model. If you're using GPT, it will take out GPT three or GPT 4.1, and it will say you've used a model that doesn't exist and it will put in GPT 3.5 turbo like two years ago, like like something which is so outdated it's like an embarrassment."}
{"question": "What is a token from a vocabulary perspective?", "keywords": ["chunk of text", "fragment of a word", "vocab"], "reference_answer": "A token is a chunk of text \u2014 it might be a word, or a fragment of a word, or even two words if they're really common. You build up a vocab of these fragments of words or chunks of characters. This middle ground compromise allows the model to recognize various fragments of words as having meaning.", "category": "conceptual", "original_transcript": "the breakthrough, the idea was to have some sort of a middle ground where you have a chunk of text. It might be a word, or it might be a fragment of a word, or it might even be two words. If you have two words that are really common, uh, and you build up a vocab, that is these fragments of words, these chunks of characters."}
{"question": "What is prompt caching and when can it save money?", "keywords": ["prompt caching", "cached tokens", "input context"], "reference_answer": "Prompt caching is a pro feature that automatically lets you pay less if you prompt with the same or similar input context multiple times within a certain period. With OpenAI, the beginning of the prompt must match identically up to where you no longer need to cache. The cached tokens portion is significantly cheaper.", "category": "technique", "original_transcript": "So this is a pro feature called prompt caching. And it's something which can automatically allow you to pay less if you prompt with the same input context or similar input context. Um, several times in a row within a certain period of time. So there is a pro feature for you, and I've given a little bit more explanation about the different rules. There's some fine print for OpenAI and Anthropic and Gemini. The the important thing to know is that typically and certainly for OpenAI, the beginning of the prompt must match identically all the way up until the point where you no longer need to cache."}
{"question": "What did the student's puzzle about two volumes of Pushkin reveal about brainteasers?", "keywords": ["Pushkin", "four millimeters", "two covers"], "reference_answer": "The Pushkin puzzle reveals that the first page of the first volume is closest to the second volume, and the last page of the second volume is closest to the first volume. So the worm only has to gnaw through the two covers \u2014 four millimeters total \u2014 which is counterintuitive and tripped up the instructor.", "category": "conceptual", "original_transcript": "Volume one would be here on your bookshelf. Volume two would be next to it. Imagine those two books put on your bookshelf what the students said to me when I said however much I said, like like 4.4. He said, uh, go and get two books yourself and go and put it on a bookshelf and look for yourself. And I said, don't be ridiculous. And he said, no, do it, do it. And so I did it. And then I was like, ah, it's because if you didn't get it, the first page of the first volume is not where your mind sort of thinks it is. Over. Over here on the left, it's in the middle. The first page of the first volume is closest to the second volume, and the last page of the second volume is closest to the first volume. So the worm only has to gnaw through the two, uh, covers. That's it, which is four millimeters"}
{"question": "What is context engineering and how does it differ from prompt engineering?", "keywords": ["context engineering", "broadly", "input sequences"], "reference_answer": "Context engineering is the evolution of prompt engineering \u2014 it's about thinking broadly about all the information you can give to an LLM to best set it up for success. That includes business specific information stitched into the prompt and equipping the LLM with tools, all about better input sequences for better outputs.", "category": "conceptual", "original_transcript": "And this new word context engineering. Context engineering. The new prompt engineering is about thinking broadly about all of the information that you can give to an LLM to best set it up for success. And that can be business specific commercial information that you're giving it in the prompt, and you're maybe stitching it together in a really compelling way."}
{"question": "What approach does the instructor recommend for a three-way conversation between LLMs?", "keywords": ["three way conversation", "single user prompt", "conversation so far"], "reference_answer": "For a three-way conversation, you have only one user prompt instead of a history of user/assistant turns. The prompt sets the scene \u2014 'You are Alex in conversation with Blake and Charlie. The conversation so far is as follows...' \u2014 then describes the conversation in any format that makes sense, and asks for the response as Alex.", "category": "technique", "original_transcript": "for the user prompt. You only have one user prompt. The prompt has to respond to you're not going to have the history, the user prompt assistant, user assistant, just the single user prompt every time. And you might be thinking, how does that work? It needs to know the history of the conversation. It's stateless I know this. Yes, but this is what the user prompt is. You are Alex in conversation with Blake and Charlie. The conversation so far is as follows."}
{"question": "What is the purpose of LightLM as an abstraction layer?", "keywords": ["light LM", "lightweight", "any model"], "reference_answer": "LightLM is a lightweight abstraction layer that gives you a simple interface to any model. You just import completion and call it with a model in the format provider/model and messages. It supports many providers including managed services like bedrock and vertex, making it convenient to switch between models.", "category": "conceptual", "original_transcript": "an abstraction layer called light LM. And a light LM does what it says on the tin. It is very light. It is a lightweight abstraction layer that just gives you a simple interface to any model. And personally, I really love light lm I use it a lot myself. It gives you such an easy way to switch between different models. I mean, you can also just use the OpenAI client library, but light LM is just particularly simple."}
{"question": "What is open router and how is it different from an abstraction layer?", "keywords": ["open router", "routes", "one API key"], "reference_answer": "Open router is a router platform that gives access to many models by routing your request to different providers. You have one account with open router AI and one API key, and when you specify the model you can choose any provider's model. It's different from an abstraction layer because it's a remote service you connect to.", "category": "comparison", "original_transcript": "There is another alternative to going to all of these websites. There is a platform called Open Router and Open Router. Is this really cool way that you can you can have one account with open router AI. And when you specify the model, you can choose any of these models and from any of these providers. And that gives you like one account, one amount to pay up front. And you can access all of them."}
{"question": "What is a knowledge base in the context of RAG?", "keywords": ["knowledge base", "database", "background information"], "reference_answer": "A knowledge base is just a fancy term for a database \u2014 a database full of useful information that you'd like your model to know about. When the user asks a question, the system queries the knowledge base for relevant information and shoves it into the prompt as background context for the LLM.", "category": "conceptual", "original_transcript": "And when you have like a database full of background information like that, it's called a knowledge base, just a fancy term for a database. And basically when we when we put this in place, we just always take whatever information we can and we shove it in the prompt. And I said here we put relevant details in the prompt."}
{"question": "Why does the instructor say RAG can suffer from a whack a mole effect?", "keywords": ["whack a mole", "fix one problem", "another one pops up"], "reference_answer": "RAG can become like whack a mole because you fix one problem \u2014 say chunking strategy makes one question fail \u2014 and another one pops up somewhere else. You're constantly getting after these little weird RAG problems because RAG is basically a hack and not a particularly scientific approach.", "category": "conceptual", "original_transcript": "And sometimes it can be a bit of a of a whack a mole that you fix one problem and just another one pops up somewhere else, and you fix that and another one pops up and you're constantly getting after these little weird rag problems. And this is, this is all because it's basically a hack. And so these are the side effects of the fact that it's not a particularly scientific approach."}
{"question": "What is the golden data set used for in RAG evaluation?", "keywords": ["golden data set", "reference answers", "set of questions"], "reference_answer": "A golden data set is a curated collection of questions and their reference answers that you'll use to measure your RAG performance. For each question, you also identify a few keywords that should appear in the retrieved context, and you have a perfect answer to compare your system's output against.", "category": "conceptual", "original_transcript": "Step one is to build your golden data set collate curate a set of questions and answers that are going to be your your reference answers against which you will measure your performance. So if it's in Shoreham, then you build this set of questions about the company that are great prototypical questions. And and with them you basically there are many different ways to do it. It's not there's one way but the way that I'm going to use for this week, and it's a common kind of way, is you come up with a set of questions, and then for each one, you can identify a few keywords that should be in the relevant context."}
{"question": "Why are batch sizes typically powers of two?", "keywords": ["power of two", "urban legend", "memory point of view"], "reference_answer": "People love using powers of two like 16, 32, 64, 128 for batch size \u2014 there's a sense it makes sense from a memory point of view and that GPUs perform faster on powers of two. The instructor personally thinks that's a bit of an urban legend, as larger batch sizes generally run faster regardless of power-of-two alignment.", "category": "conceptual", "original_transcript": "the other rule of thumb, and this one is crazy, is that people really love using powers of two for batch size. Like you'll see 16, 32, 64, 128 people always have powers of two. And, you know, again, there's this this sense that it makes sense from a memory point of view to keep it to being a power of two, and that GPUs perform faster when it is a power of two. Personally, I think that's a bit of an urban legend. I don't know if it's true."}
{"question": "What is the difference between fine tuning a frontier model and fine tuning an open source model?", "keywords": ["fine tuning a frontier model", "API calls", "guts of a model"], "reference_answer": "With fine tuning a frontier model, you're really just making API calls to tell OpenAI to fine tune the model \u2014 you don't get into the guts of training. With fine tuning an open source model you roll up your sleeves and get to the guts of the model, working with LoRA matrices and target modules.", "category": "comparison", "original_transcript": "with with fine tuning a frontier model, you're really just making API calls to tell OpenAI, hey, I'd like you to fine tune this. You don't really get to get into the guts of doing the training yourself, and it's generally understood that OpenAI uses this technique called Laura themselves when you call them to do this fine tuning. But they don't reveal that particularly. And they don't they don't announce how they do fine tuning. So we're not actually going to be looking at the guts of Laura today. We're just going to be trusting that that's what's happening."}
{"question": "What are target modules and rank R in LoRA?", "keywords": ["target modules", "Lora matrices", "rank"], "reference_answer": "Target modules are the layers of the neural network that you target for your LoRA matrices \u2014 often the attention layers. R is the number of dimensions of your LoRA matrices, short for rank, and is the R in LoRA. Alpha is how much you scale up the result of multiplying LoRA A and LoRA B before adding it to the base model.", "category": "conceptual", "original_transcript": "the target modules you'll remember this hyperparameter is saying which layers of the neural network, which ones are we going to target for our Lora matrices. And the target modules I said you often start by targeting the attention layers. And then sometimes you add on other layers as well. Those are called the target modules. R is the number of dimensions of your your Lora matrices. R is short for rank I guess, which is another word for dimensions. And it's the r in Lora and alpha is once you've done this multiplied Laura A and Laura B"}
{"question": "What is quantization in the context of training models?", "keywords": ["quantization", "four bits", "frees up so much memory"], "reference_answer": "Quantization is reducing the precision of the model's floating point numbers down to eight bits or even four bits. While that makes results poorer, it frees up so much memory that you'll be able to run more training and try more intensive things. It's a tradeoff between explanatory power and capacity.", "category": "technique", "original_transcript": "quantization is about okay. So the model is a proper full floating point number. We can reduce the precision down to eight bits or even four bits and see that, that, you know, that's definitely going to make your results poorer. But it frees up so much memory that you'll be able to run more training. You'll be able to to try more intensive things. And so when you're thinking overall, you know, you've got a limited budget instead of time to try and get to a result, it might make sense to sacrifice some of the explanatory power of your base model in order to have so much more room to do much more."}
{"question": "What is the five step strategy for solving a business problem with AI?", "keywords": ["understand the problem", "five steps", "productionize"], "reference_answer": "The five steps are: first, understand the problem you're working with; second, prepare and select the right model for the task; third, customize that model; fourth, figure out how to apply it to the problem at hand; and fifth, productionize.", "category": "conceptual", "original_transcript": "And the five steps are, first of all, to understand the problem that you're working with. Then to prepare, select the right model for the task that we already covered in week four. Customize that model. Figure out how to apply it to the problem at hand and then productionize. These are the five steps. And it's no surprise at all that my first step was going to be all about understanding."}
{"question": "What does the instructor say about reasoning models versus chat models for creative content?", "keywords": ["chat models", "creative", "fluid content"], "reference_answer": "Chat models are often better at more creative content generation. Reasoning models can overthink and produce content that is cold and analytical, whereas chat models tend to produce more fluid content. The instructor notes this sounds hand-wavy with no good metrics, more of an anecdotal observation.", "category": "comparison", "original_transcript": "this is I put a question mark here because people aren't sure about this, but it does seem that chat models are often better at more creative kind of just just content generation. If you want something to write an email for you, uh, sometimes you find that the reasoning models kind of overthink. Perhaps what they come up with is quite cold and and analytical, whereas chat models tend to produce more, um, fluid content. But this sounds hand-wavy. There's not any there's not really good metrics on this. It's more of an anecdotal way that that people think about it."}
{"question": "What does the instructor mean by 'autonomy' in agentic AI?", "keywords": ["autonomy", "master of its own destiny", "what it wants to do next"], "reference_answer": "Autonomy is the idea that you're allowing an LLM to be master of its own destiny, choosing what it wants to do next. In practice it just means the LLM takes in an input sequence and generates an output sequence that can include instructions about what activities it wants to do next.", "category": "conceptual", "original_transcript": "Another word people love to throw around with their genetic AI is the word autonomy. Autonomous. this idea that you're you're allowing an LLM to sort of be master of its own destiny, choosing what it wants to do next. Of course it sounds. It sounds like voodoo, but it's all again, it all just comes down to it taking in an input sequence, generating an output sequence. But when it generates that output sequence, that can include some instructions about what activities it wants to do next."}
{"question": "What is the rule for ordering content in a prompt to maximize prompt caching?", "keywords": ["beginning of the prompt", "static content", "current date"], "reference_answer": "The beginning of the prompt must match identically all the way up until the point where you no longer need to cache. So if you have changing information like today's date, put it at the end of your prompt rather than the beginning. Put your static content at the start so that the cache hits work.", "category": "technique", "original_transcript": "the important thing to know is that typically and certainly for OpenAI, the beginning of the prompt must match identically all the way up until the point where you no longer need to cache. So if you have something like today's date that you insert in your Certain your prompt. If you insert that the date and time at the very beginning, you will never get prompt caching because it will be different from the very start of your prompt. So you want to put that kind of information that might change at the end of your prompt, not at the beginning, and put your static content at the start."}
{"question": "What happens to the GPU memory on Colab when working with diffusion models?", "keywords": ["GPU Ram", "8.5GB", "diffusion model"], "reference_answer": "When using a diffusion model like Stable Diffusion XL Turbo on a T4 GPU in Colab, the GPU Ram zoomed up to 8.5GB. The way these diffusion models work is to dissolve a picture into white noise through inference steps, and then iterate in reverse from a prompt back to a real image.", "category": "technique", "original_transcript": "you can see sure enough that the GPU Ram zoomed up there and is at 8.5GB. One one thing that was important that I didn't mention is that this parameter here number of inference steps. So, uh, this is a bit hand-wavy, but you may know, the way these diffusion models work is that they started with a picture which was had a description of it like a, like a prompt. And then they gradually dissolve that away into white noise through through various steps. And the way a diffusion model is trained is then doing that in reverse, starting with the fuzz with a prompt, and then trying to iterate to get towards the full picture."}
{"question": "How does FAISS differ from other vector databases?", "keywords": ["Fyss", "Facebook AI similarity search", "in-memory"], "reference_answer": "FAISS stands for Facebook AI Similarity Search and is from Meta. It is not really a database \u2014 it's just an in-memory version that takes vectors and does in-memory vector search. So it's one of the most popular open source vector tools but it's a more lightweight in-memory option compared to a real database like Chroma.", "category": "comparison", "original_transcript": "Um, Fyss is very popular. It's Facebook's. It stands for Facebook AI similarity search from meta before the name change, I guess. And Fyss sounds better than mice. Uh uh, so uh, so Fyss is also it's very popular, but it is not really a database. It's just an in-memory, uh, version that is able to just take vectors and do in-memory vector search. So these are some of the most popular open source ones."}
{"question": "What is the all-MiniLM-L6-v2 model?", "keywords": ["all mini lm", "sentence transformers", "popular"], "reference_answer": "The all-MiniLM-L6-v2 is a popular open source encoder from Hugging Face's sentence transformers package. It's the encoder model that the instructor and many others use frequently and that you'll come across many times when working with embeddings.", "category": "conceptual", "original_transcript": "And then Hugging Face has a ton of open source encoders in a package called hugging Face sentence Transformers. And the one that is, I think, most popular, certainly the one that we all use and the one that I'm sure you'll come across many, many times is called the All mini lm, l6 v2. Very popular indeed."}
{"question": "What is the responses API versus the chat completions API?", "keywords": ["responses API", "managed tools", "paid functionality"], "reference_answer": "OpenAI's responses API has more functionality including managed tools like running code or browsers remotely, but most of that extra functionality is paid. It's also locked into OpenAI's ecosystem unlike the chat completions API which is ubiquitous across providers.", "category": "comparison", "original_transcript": "Most importantly, it has one called the responses API. And if you go to OpenAI's website, they they will tell you in their docs that they encourage you to use the responses API instead of the chat completions API, because it has more functionality. It has functionality to to use things called managed tools and various other things. And so they tell you that that is the better, more powerful API to use. And this is sneaky of OpenAI. And so I want to tell you I want to caution you against taking its word for it. Yes, it's true. The responses API is more powerful, it has more functionality. But there's probably two things you should know about that. Number one, that extra functionality is mostly paid functionality."}
{"question": "How does the instructor describe inference?", "keywords": ["inference", "running a model", "new inputs"], "reference_answer": "Inference is a fancy name for running a model. Inference is what you do when you have a model trained and you have some new inputs and you want to get the output. After training is done, the activity of using the model on new inputs is called inference.", "category": "conceptual", "original_transcript": "And if you're new to the word inference, inference is a fancy name for saying running a model. Inference is what you do when you got a model and you've got some new inputs and you want to get the output, that is called inference. And we'll be doing a lot of that in the next eight weeks."}
{"question": "What is the agent loop concept?", "keywords": ["agent loop", "tools in a loop", "achieve a goal"], "reference_answer": "The agent loop is the modern definition of agentic AI: an LLM running tools in a loop to achieve a goal. You have an LLM equipped with tools, and your Python code calls it in a loop again and again until it has achieved its objective.", "category": "conceptual", "original_transcript": "an agentic system is one where there's a there's an agent, an LM that runs tools in a loop to achieve a goal. So you've got an LM or maybe multiple llms, and there's some sort of an objective, and it's equipped with a set of tools, the kinds of things we did in week two. And it in order to, to carry out that objective. It can call a tool. And then if it hasn't yet achieved that objective, your Python code calls it again in a loop. And it keeps being called again and again and again."}
{"question": "What are the two types of tests when evaluating a RAG system?", "keywords": ["effective is your retrieval", "actual answers", "model metric"], "reference_answer": "There are two types of tests: first you measure how effective your retrieval is by surfacing the right context with metrics like MRR and NDCG. Second you test the actual answers themselves against a reference answer, often using LLM as a judge. The retrieval metric is closely tied to model performance.", "category": "comparison", "original_transcript": "there are two types of tests that you would do using your test data. First of all, you would measure how effective is your retrieval. How good are you at retrieving relevant content? And there's a bunch of metrics you can measure. There's actually lots and lots and lots of metrics, and people have their favorite ones."}
{"question": "What is the order of play for the Price is Right project?", "keywords": ["curating data", "evaluating frontier models", "fine tuning"], "reference_answer": "The order of play is: first curate data and evaluate frontier models, then pre-process the data, then build baseline models and traditional ML, then neural networks and frontier LLMs, then fine tuning a frontier LLM, then fine tuning an open source LLM, then deploying agentic AI.", "category": "conceptual", "original_transcript": "the order of play again on week six. This current week. It's about curating data, evaluating frontier models, where you already curated the data in day one. We pre-processed it. It was great yesterday. Now baseline models and traditional ML, then neural networks and frontier llms and then fine tuning a frontier LLM which goes into the fine tuning open source LLM and into deploying Agentic AI."}
{"question": "What is the purpose of synthetic data generated by LLMs for the test knowledge base?", "keywords": ["synthetic data", "fake data", "generated by an LLM"], "reference_answer": "The test knowledge base for InsureLLM was all fake synthetic data generated by an LLM rather than typed by hand. The instructor used the same techniques students used in week three exercises to generate this synthetic data, and noted that LLM-generated employee records were initially too effusive and gushing.", "category": "conceptual", "original_transcript": "I would love to tell you that this is all synthetic. It's all fake data. I'd love to tell you that I sat there and typed all this out for you, but as well, you know, of course I did no such thing. I used an LLM to generate this synthetic data using the same kind of techniques that you hopefully used in the exercises in week three, so that all of this is fake data generated by an LLM, which is excellent, and you should look into one of it."}
{"question": "Why is a small chunk size sometimes problematic in RAG?", "keywords": ["chunks need to be bigger", "first name", "heading"], "reference_answer": "If chunks get too small, the chunk might only contain the first name and not the full name from the heading of the document. Even if it gets selected, having that chunk in the context isn't enough to answer the question. Sometimes chunks need to be bigger and sometimes smaller, depending on the question.", "category": "conceptual", "original_transcript": "if we do that, there's a risk that we get so small that we don't even have Maxine's name. We only have her first name, say, which is a very problem that we have. Uh, and we don't we don't have the part like the heading of the document which has her name on it. And as a result, the chunk is too small. And even though it gets selected, having that chunk in the context isn't enough to answer the question. And so that's another kind of problem. And so sometimes chunks need to be bigger. Sometimes they need to be smaller."}
{"question": "What is query rewriting as an advanced RAG technique?", "keywords": ["query rewriting", "rewrite this", "querying a knowledge base"], "reference_answer": "Query rewriting is the flip side of document pre-processing. The user asks a question, but that question might not be exactly right for doing a RAG-based lookup. You call an LLM saying 'the user has asked this question, please rewrite this in a way that's going to be most suitable for querying a knowledge base'.", "category": "technique", "original_transcript": "There's a flip side that's called query rewriting, which is much the same thing. When the user asks a question, the user asks a question. Maybe that question isn't exactly right for doing a rag based lookup, or maybe to a point to a problem that we face before a couple of days ago. The question might be a follow on question from something earlier, and to do exactly the right lookup, you might want to kind of combine that with what we've said before in some clever way. And that can be really hard to do with code. But guess what? Llms have an easy time with it. So use an LLM, call an LLM and say the user has asked this question. Please rewrite this in a way that's going to be most suitable for querying a knowledge base"}
{"question": "What does the GPT response look like when asked about its training date?", "keywords": ["June the 7th 2024", "training cut off", "training set"], "reference_answer": "When asked 'what is today's date,' GPT-4.1 mini might respond with something like 'today's date is June the 7th 2024,' which is approximately when its knowledge cut off was. The model has no trained information beyond its training cut off date.", "category": "conceptual", "original_transcript": "we're now going to call the function with the question what is today's date. So it will then make this call to GPT and maybe wondering what this is going to say. And it's going to say today's date is June the 7th 2024. And you're like, what? What's going on? Well, the point is that that's like a year and a half away from me and even more for you. I imagine the point is that these models as, as we said last week when we were asking them to talk about their weaknesses, they are trained at some point in time, and there's a point at which the training data ends, where the date which the last day they were trained and that date is known as the training cut off."}
{"question": "What is a coder agent?", "keywords": ["coder agent", "execute code", "Docker container"], "reference_answer": "A coder agent is an LLM that has the ability to execute code in order to carry out its task, perhaps in a secure way like a Docker container so it's ring fenced and sandboxed. A coder agent doesn't necessarily mean an agent that writes code \u2014 it means an LLM call that's able to execute code in carrying out its task.", "category": "conceptual", "original_transcript": "you could give it a tool that can execute Python code, perhaps in a secure way, like in a in a Docker container or something. That means that it's ring fenced, sandboxed, so that that kind of when you do that, it's sometimes called as having a coder agent. It's it's an LLM that has the ability to execute code in order to carry out its task. A coder agent doesn't always doesn't necessarily mean an agent that writes code. It can really mean an agent, an LLM call that's able to execute code in carrying out its task."}
{"question": "What is the alternative to lots of LLM calls in a database tool implementation?", "keywords": ["SQL Lite", "database query", "Sqlite3"], "reference_answer": "Instead of looking things up in a Python dictionary, you can use SQL Lite as part of the UV project to do an actual database query. You import sqlite3, create a connection, and write SQL queries like SELECT price FROM prices WHERE city = ?, using parameter substitution to avoid SQL injection attacks.", "category": "technique", "original_transcript": "the easiest way to do this without needing you to do lots of configuration and set things up is to use SQL Lite, which is so quick and easy and just part of the UV project, and you can just use it. So we can just import Sqlite3 and we can write some database queries. If you'd rather build something a bit more industrial strength, you should feel free to replace these SQL queries with something like a Postgres lookup or something like that."}
{"question": "What is an end point in the context of API calls?", "keywords": ["endpoint", "http url", "hitting some web address"], "reference_answer": "An endpoint is an HTTP URL which you can call to make an API request by hitting some web address. That web address is known as an endpoint. There's nothing special about it \u2014 it's just a way to have an API exposed at a particular URL.", "category": "conceptual", "original_transcript": "It's one of those words people throw around all the time. But you might not. And for people that don't know about it, you should quickly find out. I explain it all in the Technical Foundations guide. There's nothing special at endpoint. It's an http url which you can call to, to to to make an API request by hitting some web address. And that web address would be known as an endpoint. It's a way that you have an API."}
{"question": "What does the chinchilla scaling concept imply about model performance from diminishing returns?", "keywords": ["8 billion parameters", "halfway through training", "twice as many parameters"], "reference_answer": "If you're training an 8 billion parameter model and find diminishing returns halfway through your training data, with more data to go, that's a sign you need to double the number of parameters. You will need twice as many parameters to absorb twice as much training data effectively.", "category": "conceptual", "original_transcript": "imagine that you're trying to train a model with, with with 8 billion parameters, and you start to find that you're let's say you're halfway through training your training data and you're seeing that you're getting diminishing returns. Your model's not doing any better. It seems to be sort of stuck at the halfway point in your training data, and you've got like, like double as much to go. Then what you have to think is, okay, that's my sign that I need to double the number of parameters. I will need twice as many parameters in my neural network to be able to take on twice as much training data"}
{"question": "What does the alpha hyperparameter do in LoRA?", "keywords": ["alpha", "scale that up", "before you add it"], "reference_answer": "Alpha in LoRA controls how much you scale up the matrix product of LoRA A and LoRA B before you add it to the base model. The convention is to set alpha as twice the value of R \u2014 that's what people typically do.", "category": "conceptual", "original_transcript": "alpha is once you've done this multiplied Laura A and Laura B, and you're multiplying two matrices, which you may remember from linear algebra, you multiply two matrices like that, you get one bigger one, uh, and as a result, uh, alpha is how much you scale that up by before you add it to the base model. Uh, and it's, it's typically, it's always a thing that you just double R that's what you do."}
{"question": "Why might a higher dropout value take longer to train?", "keywords": ["higher values of dropout", "longer to train", "generalize"], "reference_answer": "Higher values of dropout tend to mean you can generalize better, but they take much longer to train because the whole neural network has to learn to respond to different inputs rather than letting localized parts of the network focus on specific input types. The randomness forces more general learning.", "category": "conceptual", "original_transcript": "And higher values of dropout tend to mean that you can generalize better. But of course, as you could probably guess, higher values of dropout means it takes much longer to train because you're doing this, this thing of removing bits. So it's got to learn in some way. The whole neural network has got to be able to learn how to respond to different inputs, rather than allowing it just to cheat and just kind of have different parts of the neural network be focused on different types of input."}
{"question": "What is an epoch in machine learning training?", "keywords": ["epoch", "complete pass", "batches"], "reference_answer": "An epoch is a complete pass through all of your data. You often do multiple epochs because at each step you're only taking a small step in the right direction. Within each epoch, data is randomly selected into batches, so different batches are used each time across epochs.", "category": "conceptual", "original_transcript": "So epochs we covered last week and epoch of course, is a complete pass through all of your data. And you often do multiple epochs because you might think, but why are we putting in the same input data again? Remember in this in this cycle you're only you're finding the gradients and you're only doing a small step in the right direction. So it can make sense to make multiple steps as you get better and better. There's another reason why epochs make sense. Remember that we don't we don't pass data through one data point at a time. We batch up our data into batches and those batches every time we start an epoch, the batches are randomly selected."}