It’s important to instrument your AI applications! I hope this can more or less be taken as given just as you’d expect a non-AI-infused app to capture logs. When you’re evaluating your LLM-powered system, you need to have capture the inputs and outputs both at an end-to-end level in terms of the way the user experiences things as well as with more fine-grained granularity for all the internal workings. My goal with this blog is to first demonstrate how Phoenix and litellm can work together, and then to make sure that we are able to group all spans together under a single trace. I’ll write the blog as I work so at this point I’m not sure exactly how this will turn out. Basic logging with litellm + phoenix As a reminder, here’s how we make an LLM call with litellm: import litellm completion_response = litellm.completion( model="openrouter/google/gemma-3n-e4b-it:free", messages=[ { "content": "What's the capital of China? Just give me the name.", "role": "user", } ], ) print(completion_response.choices[0].message.content) # prints 'Beijing' The Phoenix docs explain how to set up basic logging for litellm: install the following pip packages: arize-phoenix-otel openinference-instrumentation-litellm (litellm, obviously) set up the necessary environment variables with API key etc to ensure that traces get sent to the right account and endpoint Let’s assume we’re using the hosted Phoenix Cloud version for now. Then we can rerun our example, with some slight tweaks: import litellm from phoenix.otel import register # configure the Phoenix tracer tracer_provider = register( project_name="hinbox", # Default is 'default' auto_instrument=True, # Auto-instrument your app based on installed OI dependencies ) completion_response = litellm.completion( model="openrouter/google/gemma-3n-e4b-it:free", messages=[ { "content": "What's the capital of China? Just give me the name.", "role": "user", } ], ) print(completion_response.choices[0].message.content) So we first register the Phoenix tracer, specify the project (already set up in Phoenix Cloud) and then run our litellm completion as previously. In the terminal we see he following logs: 🔭 OpenTelemetry Tracing Details 🔭 | Phoenix Project: hinbox | Span Processor: SimpleSpanProcessor | Collector Endpoint: https://app.phoenix.arize.com/v1/traces | Transport: HTTP + protobuf | Transport Headers: {'api_key': '****', 'authorization': '****'} | | Using a default SpanProcessor. `add_span_processor` will overwrite this default. | | ⚠️ WARNING: It is strongly advised to use a BatchSpanProcessor in production environments. | | `register` has set this TracerProvider as the global OpenTelemetry default. | To disable this behavior, call `register` with `set_global_tracer_provider=False`. Beijing So immediately there are a lot of things to consider. It seems that we’ll want to use the BatchSpanProcessor that it suggests, and also it seems like I might not want to set this as the global tracing provider, too. In Phoenix Cloud, I see this: Basic Phoenix tracing interface As you can see, we’ve captured the input and output messages for the completion, it’s tracked the latency of the call (1.16s, which seems pretty slow actually!). There is also some sort of an annotation interface though I’ll explore that down the line maybe. I immediately notice that I’m missing things like the system attributes for where the call was made, also metadata like the temperature and other settings. I’d also like to see things like token counts (which you can get in Phoenix but they’re sort of buried) as well as the estimated cost of the call(s) and so on. We can see about adding some of that down the line. BatchSpanProcessor for production usage Let’s next move on to adding BatchSpanProcessor as the message suggested, which is as simple as adding batch=True to the tracer provider registration code. What this does is make sure that spans are processed in batches before they’re exported to Arize. This takes away some of the network costs that you incur when sending the spans one by one. I’ve also made sure to turn off the registration of this tracing provider as the global one: import litellm from phoenix.otel import register # configure the Phoenix tracer tracer_provider = register( project_name="hinbox", # Default is 'default' auto_instrument=True, # Auto-instrument your app based on installed OI dependencies set_global_tracer_provider=False, batch=True, ) completion_response = litellm.completion( model="openrouter/google/gemma-3n-e4b-it:free", messages=[ { "content": "What's the capital of China? Just give me the name.", "role": "user", } ], ) print(completion_response.choices[0].message.content) And I get this in the terminal: 🔭 OpenTelemetry Tracing Details 🔭 | Phoenix Project: hinbox | Span Processor: BatchSpanProcessor | Collector Endpoint: https://app.phoenix.arize.com/v1/traces | Transport: HTTP + protobuf | Transport Headers: {'api_key': '****', 'authorization': '****'} | | Using a default SpanProcessor. `add_span_processor` will overwrite this default. Beijing It’s actually somehow a bit annoying to still see a message about the fact that I’m using a default SpanProcessor. It’s unclear to me why I need to care that this is a default one. The message is taking up real estate in the logs and it seems important (otherwise why would they have included it?) but it’s also unclear to me what the alternative is and why I’d want to overwrite the default. I think for now I’ll leave it. Using the litellm callbacks as an alternative If we stray away from the official supported way to handle tracing with Phoenix, there’s also the community-supported in-built litellm option: import litellm litellm.callbacks = ["arize_phoenix"] completion_response = litellm.completion( model="openrouter/google/gemma-3n-e4b-it:free", messages=[ { "content": "What's the capital of China? Just give me the name.", "role": "user", } ], metadata={"PROJECT_NAME": "hinbox"}, ) print(completion_response.choices[0].message.content) This achieves a similar result, though I was unable to get the trace to land anywhere other than the default project. Arize’s docs mention a PHOENIX_PROJECT_NAME environment variable but it seems this isn’t respected or used by the litellm implementation. Indeed when I look at the implementation, I don’t see this being used anywhere, so it seems that the community-driven implementation isn’t really the way forward. I just wanted to mention it, however, since some of the ‘callback’ integrations for tracing in litellm are really nicely implemented (like the one for Langfuse, e.g.) so I wanted to try it out at least. One trace, multiple spans For anything beyond a simple LLM call, which means most real-world LLM applications, we’ll want to be capturing multiple spans as part of a single trace. LLM Tracing Tools’ Naming Conventions (June 2025) Side-note: I dug into how some of the major LLM tracing providers name their primitives. I was reassured that we seem to have coalesced around ‘trace -> span’ and that the OpenTelemetry way seems to have been adopted by most. Tracing nomenclature (June 2025) Grouping spans under a single trace I updated the code such that we now have a function that makes two separate LLM calls. I’d want them to both be registered as spans under the same trace: import litellm from phoenix.otel import register tracer_provider = register( project_name="hinbox", # Default is 'default' auto_instrument=True, # Auto-instrument your app based on installed OI dependencies set_global_tracer_provider=False, batch=True, ) def query_llm(prompt: str): completion_response = litellm.completion( model="openrouter/google/gemma-3n-e4b-it:free", messages=[ { "content": prompt, "role": "user", } ], ) return completion_response.choices[0].message.content def my_llm_application(): query1 = query_llm("What's the capital of China? Just give me the name.") query2 = query_llm("What's the capital of Japan? Just give me the name.") return (query1, query2) if __name__ == "__main__": print(my_llm_application()) But these just get registered as two separate traces/calls. The key bit of the documentation is the ‘Using Phoenix Decorator’ section, it seems. If I add a decorator on top of my function and get the specific tracer, it seems I am able to start to group things together: import litellm from phoenix.otel import register tracer_provider = register( project_name="hinbox", # Default is 'default' auto_instrument=True, # Auto-instrument your app based on installed OI dependencies set_global_tracer_provider=False, batch=True, ) tracer = tracer_provider.get_tracer(__name__) def query_llm(prompt: str): completion_response = litellm.completion( model="openrouter/google/gemma-3n-e4b-it:free", messages=[ { "content": prompt, "role": "user", } ], ) return completion_response.choices[0].message.content @tracer.llm def my_llm_application(): query1 = query_llm("What's the capital of China? Just give me the name.") query2 = query_llm("What's the capital of Japan? Just give me the name.") return (query1, query2) if __name__ == "__main__": print(my_llm_application()) This works and I see this in the Phoenix Cloud dashboard: Grouped traces under a single span See how it’s taken the function name as the name of the span. And it’s grouped those two LLM calls that happen within the function as we wanted. We can also update the decorator to denote different kinds of spans that we want to capture: The kinds of spans you can choose from I’m immediately a bit confused by the interface again, because when you click on the ‘Traces’ tab in Phoenix Cloud you actually still just see ‘spans’: Spans in the Traces tab In the documentation it isn’t clear to me how to create a trace that includes an llm span and an embedding span, for example. What’s even more frustrating is that the tracer decorator object doesn’t implement all the span types, just agent, chain and llm it seems. I tried something like this but it just ended up producing 3 separate traces in Phoenix Cloud. I looked at the documentation for using base OTEL instead of the Phoenix decorators, but there was also nothing in there on how to denote the trace instead of just the span. I was wondering if their ‘Sessions’ primitive was the way forward here, but they’re pretty clear in stating that a Session is a “sequence of traces”. So I’m at a bit of a dead end with Phoenix for now. I might return to Braintrust or Langfuse since these seem to have better support for what I’m trying to do (i.e. group spans together underneath a trace). I’m really reluctant to try to instrument hinbox with Phoenix when I’m unable even to get this basic grouping working properly with some dummy code. Update: solution from the Arize team I posted this blog on the Arize slack and they got back to me with a solution: import litellm from phoenix.otel import register tracer_provider = register( project_name="hinbox", # Default is 'default' auto_instrument=True, # Auto-instrument your app based on installed OI dependencies set_global_tracer_provider=False, ) tracer = tracer_provider.get_tracer(__name__) @tracer.llm def query_llm(prompt: str): completion_response = litellm.completion( model="openrouter/google/gemma-3n-e4b-it:free", messages=[ { "content": prompt, "role": "user", } ], ) return completion_response.choices[0].message.content @tracer.agent def query_agent(prompt: str): return "I am an agent." @tracer.chain def my_llm_application(): query1 = query_llm("What's the capital of China? Just give me the name.") query2 = query_llm("What's the capital of Japan? Just give me the name.") agent1 = query_agent("Who are you?") return (query1, query2, agent1) if __name__ == "__main__": print(my_llm_application()) And you can see how this looks in the Phoenix Cloud dashboard: Grouped spans Judging from the code it seems like the way the span is constructed simply depends on how you assemble the hierarchy of spans. For instance, if I wanted to consider the top-level entity for this ‘trace’ (i.e. a grouping of spans) then I could use this code: import litellm from phoenix.otel import register tracer_provider = register( project_name="hinbox", # Default is 'default' auto_instrument=True, # Auto-instrument your app based on installed OI dependencies set_global_tracer_provider=False, # batch=True, ) tracer = tracer_provider.get_tracer(__name__) @tracer.llm def query_llm(prompt: str): completion_response = litellm.completion( model="openrouter/google/gemma-3n-e4b-it:free", messages=[ { "content": prompt, "role": "user", } ], ) return completion_response.choices[0].message.content @tracer.agent def query_agent(prompt: str): return "I am an agent." @tracer.tool(name="query_embedding", description="Query embedding") def query_embedding(prompt: str): return [0.1, 0.2, 0.3] @tracer.agent def my_llm_application(): query1 = query_llm("What's the capital of China? Just give me the name.") query2 = query_llm("What's the capital of Japan? Just give me the name.") agent1 = query_agent("Who are you?") embedding1 = query_embedding("What's the capital of China? Just give me the name.") return (query1, query2, agent1, embedding1) if __name__ == "__main__": print(my_llm_application()) And now instead of this trace being of kind ‘chain’, it’s now of kind ‘agent’, which some internal spans also being of kind ‘agent’. In a conversation in the Arize Slack I got the following clarification: “Traces as the concept under”signals” is basically a unique identifier of spans (think “span” of time). See https://opentelemetry.io/docs/concepts/signals/traces/ In most cases if you filter spans by “roots” (e.g. spans that don’t have parents) and or look at the collective set of “traces” they will roughly look the same. Most of the time this is the view you want when looking at telemetry. Spans are too noisy to be looking at in isolation. While the two tabs feel largely overlapping, it’s a bit intentional as there’s actually no real object called a trace - it’s just a series of spans. You will see these abstractions in most observability platform.” The line that: “there’s actually no real object called a trace - it’s just a series of spans” Was extremely clarifying, actually. It explains the fuzziness between the spans and traces tab in the Phoenix dashboard. I also got some clarification around the missing @tracer.embbeding and @tracer.reranker decorators: “We emit spans for embedding text to vectors (like”adda”), guardrailing via thinks like guardrals or content moderation, and reranking things via things like cohere. However it’s sorta rare for people to manually write these. We will have decorators for them but right now they are typically emitted from autoinstrumentors like langgraph where there are common patterns for these things. We will have decorators for them very soon - but things like reranking are much more complex than things like tool calling so we are codifying these primitives now.” So there you have it! Some clarity. I’ll have to play around to see whether I go with the Langfuse route or the Phoenix route and which feels most ergonomic in the hinbox codebase. Appreciate the quick feedback from the Phoenix team, though!

Today was the first session of Hamel + Shreya’s course, “AI Evals for Engineers and PMs”. The first session was all about mental models for thinking about the topic as a whole, mixed in with some teasers of practical examples and advice. I’ll try to keep up with blogging about what I learn as we go. Most of the actual content will go up online at some point in the future, I’m assuming, so not much point writing up super detailed notes. (There is also a book coming, which I assume will be great, and about which you can learn more here.) So in general I’ll try to be doing the following as I blog along: highlight things I found interesting or inspiring based on the formal ‘lectures’ anything that comes up while doing the practical ‘homework’ (there are some optional exercises assigned to ground everything) contextualise or situate things that come up in my own experience having worked on a few LLM-driven projects Today, fresh out of the first class, I wanted to write about the mental model of the ‘three gulfs’ that they propose, the improvement loop that they suggest is how to measurably improve your applications, and also prompting through the lens of evals. Finally I’ll round off with a bit about what I’ll be exploring this week. The Three Gulfs: Specification, Generalization and Comprehension So there’s this image that they shared in the book chapter preview discussion that came up again during the lesson today: The three gulfs of LLM application development (They’ve shared it already in the YouTube discussion + I see it on Twitter being shared so I think I’m not sharing something I ought not to!) The course is very practically focused, especially so for application developers, so this diagram is in that context. The diagram offers up a way of thinking about LLM application development that pinpoints the places where you might do your work, and it’s also a way of thinking through things systematically, too. I was especially interested in the differentiation between the gulf of specification and the gulf of generalisation, since these can often feel similar, but actually the way to get out of them is actually slightly different. I’ll go into a bit more detail below, but basically with the gulf of specification you might want to be working on your prompts + how specific you are, whereas with the generalisation gulf you might need things like splitting up your tasks or making sure your system is outputting things in a structured way, etc etc. Note also that the world of tooling also doesn’t help you in a specific or targeted way to focus on one aspect of this diagram. Too often the tools try to cover the whole picture and probably also muddy the water by eliding the differences between the different tasks and challenges of each island or the gulf in between. All this is pretty abstract, so let’s go through them one by one. The Gulf of Comprehension in Practice This was seen as sort of the starting point for thinking through LLM Application improvement. At this point your big problem is that you’re trying to understand the data that comes your way from your users. You’re trying to understand the inputs to your application (what your users are typing, assuming that text is the medium of communication / input) and you’re trying to understand what the application or LLM is outputting. The challenge comes because you can’t read every single log or morsel of data. You have to filter things down somehow! If this were something more like traditional ML you’d have statistics to help boil down your data, but mostly we’re talking about unstructured text data so it’s much more unwieldy. This challenge means that people often get stuck at this point. This is where POC applications live, breathe and eventually die. You have enough sense that things are ‘kinda’ working, but you don’t really know what the failure modes are, so you don’t know how to improve it. You’ve tried out one or two things in a halfway systematic way, but really you have no idea what’s working well and what’s not. On Tools vs Process Hamel made the good point that it’s probably not so useful to think about tools too much when thinking at this stage. Generally speaking what’s going on is most often actually a process problem and trying to go straight to ‘what tool do I need’ is probably avoiding the real issue. The Gulf of Specification in Practice This is the place where you are trying to translate intent into precise instructions that the LLM will follow. You’re trying to be explicit and specific in the hope that the LLM will do what you want it to do, and not do the things that you don’t want it to do. The obvious manifestation of this is people writing bad prompts. It might seem that it’s also present when you try to have an LLM solve one problem when it’s either unsuited for that task or the task needs to be broken up and so on, but that’s the sister gulf of generalisation. Here, we’re focused on how to improve the specificity of your prompts. When you split things up and highlight the fact that prompts are something that you’ll need to work on and to improve, it becomes clear that it’s something you wouldn’t want to outsource or to skimp time on. Really the prompt writing is the thing that you (at certain moments, and where it’s identified as the thing needing focus / improvement) want to be working on in partnership with domain experts. For small applications, you might be the same person as the domain expert! For bigger projects, you might be working with domain experts. Just be aware that often the domain expert might not necessarily be detached enough to be able to figure out what needs the focus, or where the weaknesses of a prompt are. That’s what the iterative process / error analysis and everything else that’ll be taught in the course is for (see below and see future posts). Another point Hamel made was about why prompts are actually so important: “you have to express your project taste somewhere”. Given that your application might be fully / mostly driven by LLMs, the prompt is actually a really crucial place to express this taste and as such might be thought of as your ‘moat’. I know just from having experienced a variety of LLM-driven applications, it’s quite easy to tell the ones where the product team gave their prompts and their specification some real love. It’s the difference between POC junk that will die a slow and lonely death and something that delights and solves real user problems. Gulf of Generalisation Shreya didn’t really get into the details around the generalisation gulf in practical terms in this lesson, but I think this one can be a sort of place of comfort for the technically-minded to make refuge in. It’s one where there’s a ton of tools and technologies and techniques to play with, and vendors also live in this space and try to claim that their particular product or special sauce is the thing to help you and so on. The Improvement Loop for LLM Applications We also got a high-level overview of the loop that allows you to iteratively improve an LLM application: The analyse, measure and improve loop; adapted from an image used in the course There’s a lot to unpack in all these different stages, and we didn’t really get into the details in the session today but you can see how this offers a really powerful way of thinking through what it means to iteratively improve an LLM application. Learning how to implement this in a practical way will be the main thing I want to get good at by the end of this course. The process is made up of a bunch of techniques, but in my experience companies or use cases that struggle with improving what they built also lack the scaffold of this loop to orient themselves. Prompting through the lens of evals As we explored above, prompting is sort of the table stakes of improving your LLM application. In order to get good at prompting, it can help to appreciate what they are good at and what they struggle with. So, as Shreya put it, “leverage their strengths and anticipate their weaknesses” (when prompting). At this point Shreya got into some points around what kinds of things went into a good prompt but I think I’ll write a separate blog on that and I don’t want to just regurgitate what we listened to. Today was more of a high-level introduction, and in any case it was much more about the outer-loop process instead of the inner loop (where tooling + specific techniques play more of a role.) A slide from a talk I gave about the inner loop vs the outer loop of GenAI development So it’s great that the course gets into the weeds (esp in the course materials, which include the draft of the book Hamel & Shreya are writing) but I think the really useful thing they’re doing is situating the tactical improvements and techniques within the strategic patterns and workflows that teams and individuals should be doing to work on these LLM applications. At a high level, what are we talking about: how to tease out failure scenarios for these applications and their behaviours conversely, how to understand exactly which domains it does well for Things I want to think about more There was a ton of really rich discussion around prompting in the Discord. I’m interested in exploring more: cross-provider prompting decisions (i.e. how prompting an OpenAI model differs from what you do with a Llama model or whatever) prompts that work with reasoning models vs non-reasoning models the tradeoffs of whether you put your instructions in system prompts vs user instructions In general there’s been a bunch of noise recently about so-called ‘leaked’ system prompts from a bunch of LLM API providers and I’ve mainly been struck by just how detailed they are. I consider myself pretty good at improving and iterating on prompts, but I’ll admit I’m not writing these multi-thousand word tomes. I’d like to explore which scenarios it makes sense to do so, and how to calculate at what point it makes sense from a cost or latency perspective to do so. As I’m sure you can detect, I’m really enthusiastic about the lesson to come and will work in the meanwhile on some of the readings that have been set as well as the homework task of writing a system prompt for a LLM-powered recipe recommendation application!

Google released an updated iteration of their Deep Research tool that uses the new 2.5 Pro model. This was taken from a post originally made on Twitter, so please excuse the terseness. First impressions: a bit too eager to jump into a deep research task even when I just ask a clarifying question quite verbose, just like the OpenAI version. Not sure why both play this up a lot. It looks impressive but in practice I think we need more entry points into this. The ‘Executive Summary’ and other concluding headers are nice touches but I feel maybe there should be some more adherence to user requests for short reports. (I get that as UI it’s maybe weird to think for 10 mins and then spit out a very concise version, but it might actually be more useful.) I continue to be annoyed about how these Gemini DR reports handle footnotes (i.e. as endnotes whacked on at the end of the report). Almost a deal-breaker IMO. It’s almost like GDR tries to show how scholarly and serious it is by giving you these walls of prose (vs OpenAI DR which throws in a lot more bullet points). Not sure one is better than the other but would appreciate a bit more flexibility! The portability of these reports has always been not great. Yes you can export them to Google Docs but markdown (+ other options) would have been much better. In practice, this means that whenever I use GDR the report stays stuck there and I’m far less likely to share it with anyone, whereas the OpenAI DR reports I drop parts/all into a Github Gist etc. These reports have been getting better and better, all things considered. I’ve been following along and using GDR from the early days (even pre-OpenAI DR) and this latest version is the best version of it so far (as you’d hope!) (It’s also a little bit annoying that GDR has removed any way to use the older versions of GDR with Gemini Pro 2.0 and 1.5 etc. Makes it harder to actually compare these things.) Please let’s get an ipad version of the Gemini iPad app soon, too? Feels a bit regressive to have to use GDR on the web interface always. For serious research (as opposed to simply generating a nice report on some area where you don’t know much about already), all these tools remain hamstrung by the quality of the sources. In areas where I am (or very recently used to be) a leading scholar / researcher, the difference between what I’d expect (in terms of taste / discernment for picking out these sources) is especially egregious. Make the models better, yes, but have better filters + retrieval. So yeah, these tools are getting good! Kudos to the teams who are implementing this stuff. Hard to make it perform reproducibly well on so many open-ended uses. But more work to be done! IMO the really great implementations of this ‘deep research’ pattern will all be in-house where you can have control over: source selection (i.e. high-quality inputs only, not just some random things on the internet) how long it spends thinking about a particular area / loop of the research (or decides to backtrack and dig deeper etc) output types / templates / length different modalities of Q&A (sometimes you want reports, other times you want a quick question answered, other times you want visual guides etc etc.) different models for different kinds of tasks possibly you have little sub-research agents / processes which will go off and work on some hypothesis, possibly involving actual datasets / analysis of tabular data etc, something clearly missing from the current versions we have A few other things: GDR’s ‘clarification step’ (which I’ve heard them discuss on podcasts etc) is not as good or useful as the OpenAI DR clarification questions. In practice, because it’s buried under a concealment button that you have to click etc, and where the entire UI seems to be screaming at you to ‘Start Research’, you basically never update or amend the research plan. And when you do, it’s really not clear what’s changed because you don’t get some feedback or diff that your comments were understood; you just get an entire new research plan (again buried under the concealment button) Going forward we’re probably going to want / need ways of navigating the layers to this research. A global overview report will have subsections that (should you wish) can be expanded into their own more detailed or granular reports. This is how research works, after all. Not just endless new reports all trailed one after another pointed in the same direction. The other thing that I think we’re really going to need to work on is research taste. Like the LLMs that power them, GDR and OpenAI DR offer a level of research taste developed to the mean. (I know people are thinking about this since it came up on Dwarkesh’s podcast with the AI 2027 guys, but they were focused on scientific research.) I think there’s not a single answer for this which is, again, why I see the end result as people bringing these things in-house where they get to develop and refine what makes their particular flavour of research unique. (In the human-generated research world this is very much the case, where certain institutions (or even particular authors) are known for how deep they go, or what kinds of sources they prefer, or how they choose to feature or highlight the primary sources they access, and so on.) There are many possible variations of how this manifest, and I hope that we’re headed into a world where all the AI ‘deep researchers’ will be unique and quirky in all the best senses of that word.

Finally back on track and reading the next chapter of Chip Huyen’s book, ‘AI Engineering’. Here are my notes on the chapter. Overview and Core Philosophy “Data will be mostly just toil, tears and sweat.” This is how we start the chapter :) This candid assessment frames dataset engineering as a discipline that requires both technical sophistication and pragmatic persistence. While the chapter’s placement might have been suitable earlier in the book, its position allows it to build effectively on previously established concepts. Data Curation: The Foundation Data curation addresses various use cases including fine-tuning, pre-training, and training from scratch, with specific considerations for chain of thought reasoning and tool use. The process addresses three fundamental aspects: Data Quality: The equivalent of ingredient quality in cooking Data Coverage: Analogous to having the right mix of ingredients Data Quantity: Determining the optimal volume of ingredients Quality Criteria Data quality encompasses multiple dimensions: Relevance to task requirements Consistency in format and structure Sufficient uniqueness Regulatory compliance (especially critical in regulated industries) Coverage Considerations Coverage involves strategic decisions about data proportions: Large language models often utilize significant code data (up to 50%) in training, which appears to enhance logical reasoning capabilities beyond just coding Language distribution can be surprisingly efficient (even 1% representation of a language can enable meaningful capabilities) Training proportions may vary across different stages of the training process Quantity and Optimization A key phenomenon discussed is ossification, where extensive pre-training can effectively freeze model weights, potentially hampering fine-tuning adaptability. This effect is particularly pronounced in smaller models. Key quantity considerations include: Task complexity correlation with data requirements Base model performance implications Model size considerations (OpenAI notes that with ~100 examples, more advanced models show superior fine-tuning performance) Potential for using lower quality or less relevant data for initial fine-tuning to reduce high-quality data requirements Recognition of performance plateaus where additional data yields diminishing returns Data Acquisition Process The chapter provides a detailed example workflow for creating an instruction-response dataset: Initial dataset identification (~10,000 examples) Low-quality instruction removal (reducing to ~9,000) Low-quality response filtering (removing 3,000) Manual response writing for remaining high-quality instructions Topic gap identification and template creation (100 templates) AI synthesis of 2,000 new instructions Manual annotation of synthetic instructions Final result: 11,000 high-quality examples Data Augmentation and Synthesis Synthesis Objectives Increasing data quantity Expanding coverage Enhancing quality Addressing privacy concerns Enabling model distillation Notable Research: An Anthropic paper (2022) found that language model-generated datasets can match or exceed human-written ones in quality for certain tasks. Note that some teams actually prefer AI-generated preference data due to human fatigue and inconsistency factors. Synthesis Applications The chapter distinguishes between pre-training and post-training synthesis: Synthetic data appears more frequently in post-training Pre-training limitation: AI can reshape existing knowledge but struggles to synthesize new knowledge LLaMA 3 Synthesis Pipeline A comprehensive workflow example: AI generation of problem descriptions Solution generation in multiple programming languages Unit test generation Error correction Cross-language translation with test verification Conversation and documentation generation with back-translation verification This pipeline generated 2.7 million synthetic coding examples for LLaMA 3.1’s supervised fine-tuning. Model Collapse Considerations The chapter addresses the risk of model collapse in synthetic data usage: Potential loss of training signal through repeated synthetic data use Current research suggests proper implementation can avoid collapse Importance of quality control in synthetic data generation Model Distillation Notable example: BuzzFeed’s fine-tuning of Flan T5 using LoRa and OpenAI’s text-davinci-003 generated examples, achieving 80% inference cost reduction. Data Processing Best Practices Expert Tip: “Manual inspection of data has probably the highest value to prestige ratio of any activity in machine learning.” - Greg Brockman, OpenAI co-founder Processing Guidelines The chapter emphasizes efficiency optimization: Order optimization (e.g., deduplication before cleaning if computationally advantageous) Trial run validation before full dataset processing Data preservation (avoid in-place modifications) Original data retention for: Alternative processing needs Team requirements Error recovery Technical Processing Approaches Deduplication strategies include: Pairwise comparison Hashing methods Dimensionality reduction techniques Multiple libraries are referenced (page 400) for implementation. Data Cleaning and Formatting HTML tag removal for signal enhancement Careful prompt template formatting, crucial for: Fine-tuning operations Instruction tuning Model performance optimization Data Inspection The chapter emphasizes the importance of manual data inspection: Utilize various data exploration tools Dedicate time to direct data examination (recommended: 15 minutes of direct observation) Consider this step non-optional in the process