More from Alex Strick van Linschoten
I previously tried (and failed) to setup LLM tracing for hinbox using Arize Phoenix and litellm. Since this is sort of a priority for being able to follow along with the Hamel / Shreya evals course with my practical application, I’ll take another stab using a tool with which I’m familiar: Braintrust. Let’s start simple and then if it works the way we want we can set things up for hinbox as well. Simple Braintrust tracing with litellm callbacks Callbacks are listed in the litellm docs as the way to do tracing with Braintrust. So we can do something like this: import litellm litellm.callbacks = ["braintrust"] completion_response = litellm.completion( model="openrouter/google/gemma-3n-e4b-it:free", messages=[ { "content": "What's the capital of China? Just give me the name.", "role": "user", } ], metadata={ # "project_id": "1235-a70e-4571-abcd-234235", "project_name": "hinbox", }, ) print(completion_response.choices[0].message.content) You can pass in a project_id or a project_name and the traces will be routed there. Here’s what it looks like in the Braintrust dashboard: Our first trace logged in Braintrust Note how you can’t see which model was used for the LLM call, nor any cost estimates. The docs mention that you can pass metadata into Braintrust using the metadata property: “braintrust_* - any metadata field starting with braintrust_ will be passed as metadata to the logging request” (link) This seems a bit rudimentary, however. If we take a look at the full tracing documentation on the Braintrust docs we can see that they seem to recommend wrapping the OpenAI client object instead: import os from braintrust import init_logger, traced, wrap_openai from openai import OpenAI logger = init_logger(project="hinbox") client = wrap_openai(OpenAI(api_key=os.environ["OPENAI_API_KEY"])) # @traced automatically logs the input (args) and output (return value) # of this function to a span. To ensure the span is named `answer_question`, # you should name the function `answer_question`. @traced def answer_question(body: str) -> str: prompt = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": body}, ] result = client.chat.completions.create( model="gpt-3.5-turbo", messages=prompt, ) return result.choices[0].message.content def main(): input_text = "What's the capital of China? Just give me the name." result = answer_question(input_text) print(result) if __name__ == "__main__": main() This indeed does label the span as answer_question but it doesn’t do much else. Even the model name isn’t captured here. Instrumenting a series of calls to handle ‘deeply nested code’ (as their docs puts it) even didn’t log the things it was supposed to: import os import random from braintrust import current_span, init_logger, start_span, traced, wrap_openai from openai import OpenAI logger = init_logger(project="hinbox") client = wrap_openai(OpenAI(api_key=os.environ["OPENAI_API_KEY"])) @traced def run_llm(input): model = "gpt-4o" if random.random() > 0.5 else "gpt-4o-mini" result = client.chat.completions.create( model=model, messages=[{"role": "user", "content": input}] ) current_span().log(metadata={"randomModel": model}) return result.choices[0].message.content @traced def some_logic(input): return run_llm("You are a magical wizard. Answer the following question: " + input) def simple_handler(input_text: str): with start_span() as span: output = some_logic(input_text) span.log(input=input_text, output=output, metadata=dict(user_id="test_user")) print(output) if __name__ == "__main__": question = "What's the capital of China? Just give me the name." simple_handler(question) This is adapted from the example they pasted in their docs as their one isn’t even a functional code example on its own. It is seeming increasingly clear that Braintrust isn’t going to be the right choice, at least as long as I want to keep using litellm. I know that Langfuse has a very nice integration with litellm, so I think I’ll pivot over to that now. Basic tracing with Langfuse and litellm Simple tracing is easy: import litellm litellm.callbacks = ["langfuse"] def query_llm(prompt: str): completion_response = litellm.completion( model="openrouter/google/gemma-3n-e4b-it:free", messages=[ { "content": "What's the capital of China? Just give me the name.", "role": "user", } ], ) return completion_response.choices[0].message.content def my_llm_application(): query1 = query_llm("What's the capital of China? Just give me the name.") query2 = query_llm("What's the capital of Japan? Just give me the name.") return (query1, query2) print(my_llm_application()) We specify langfuse for the callback and each llm call is logged as a separate trace + span. Here you can see what this looks like in the dashboard: Basic trace and span in Langfuse dashboard The litellm docs include information on how to specify custom metadata and grouping instructions for Langfuse. Notably, we can specify (as of June 2025, at least!) things like a session_id, tags, a trace_name and/or trace_id as well as custom trace metadata and so on. So we can get most of what we want to specify in the following way: import litellm litellm.callbacks = ["langfuse"] def query_llm(prompt: str, trace_id: str): completion_response = litellm.completion( model="openrouter/google/gemma-3n-e4b-it:free", messages=[ { "content": "What's the capital of China? Just give me the name.", "role": "user", } ], metadata={ "trace_id": trace_id, "trace_name": "my_llm_application", "project": "hinbox", }, ) return completion_response.choices[0].message.content def my_llm_application(): query1 = query_llm( "What's the capital of China? Just give me the name.", "my_llm_application_run_789", ) query2 = query_llm( "What's the capital of Japan? Just give me the name.", "my_llm_application_run_789", ) return (query1, query2) if __name__ == "__main__": print(my_llm_application()) This looks like this in the Langfuse dashboard: Spans grouped into traces This is honestly most of what I’m looking for in terms of my tracing. If I were to use a non-OpenRouter model, moreover, I’d also get full costs in the Langfuse dashboard, e.g.: LLM costs in Langfuse dashboard As such, I can monitor costs from within OpenRouter and have the option to keep track of costs in Langfuse by passing custom metadata should I wish. I’ll make a separate blog where I actually go into how I set up + instrumented hinbox for this kind of tracing while continuing to use litellm.
Today was the first session of Hamel + Shreya’s course, “AI Evals for Engineers and PMs”. The first session was all about mental models for thinking about the topic as a whole, mixed in with some teasers of practical examples and advice. I’ll try to keep up with blogging about what I learn as we go. Most of the actual content will go up online at some point in the future, I’m assuming, so not much point writing up super detailed notes. (There is also a book coming, which I assume will be great, and about which you can learn more here.) So in general I’ll try to be doing the following as I blog along: highlight things I found interesting or inspiring based on the formal ‘lectures’ anything that comes up while doing the practical ‘homework’ (there are some optional exercises assigned to ground everything) contextualise or situate things that come up in my own experience having worked on a few LLM-driven projects Today, fresh out of the first class, I wanted to write about the mental model of the ‘three gulfs’ that they propose, the improvement loop that they suggest is how to measurably improve your applications, and also prompting through the lens of evals. Finally I’ll round off with a bit about what I’ll be exploring this week. The Three Gulfs: Specification, Generalization and Comprehension So there’s this image that they shared in the book chapter preview discussion that came up again during the lesson today: The three gulfs of LLM application development (They’ve shared it already in the YouTube discussion + I see it on Twitter being shared so I think I’m not sharing something I ought not to!) The course is very practically focused, especially so for application developers, so this diagram is in that context. The diagram offers up a way of thinking about LLM application development that pinpoints the places where you might do your work, and it’s also a way of thinking through things systematically, too. I was especially interested in the differentiation between the gulf of specification and the gulf of generalisation, since these can often feel similar, but actually the way to get out of them is actually slightly different. I’ll go into a bit more detail below, but basically with the gulf of specification you might want to be working on your prompts + how specific you are, whereas with the generalisation gulf you might need things like splitting up your tasks or making sure your system is outputting things in a structured way, etc etc. Note also that the world of tooling also doesn’t help you in a specific or targeted way to focus on one aspect of this diagram. Too often the tools try to cover the whole picture and probably also muddy the water by eliding the differences between the different tasks and challenges of each island or the gulf in between. All this is pretty abstract, so let’s go through them one by one. The Gulf of Comprehension in Practice This was seen as sort of the starting point for thinking through LLM Application improvement. At this point your big problem is that you’re trying to understand the data that comes your way from your users. You’re trying to understand the inputs to your application (what your users are typing, assuming that text is the medium of communication / input) and you’re trying to understand what the application or LLM is outputting. The challenge comes because you can’t read every single log or morsel of data. You have to filter things down somehow! If this were something more like traditional ML you’d have statistics to help boil down your data, but mostly we’re talking about unstructured text data so it’s much more unwieldy. This challenge means that people often get stuck at this point. This is where POC applications live, breathe and eventually die. You have enough sense that things are ‘kinda’ working, but you don’t really know what the failure modes are, so you don’t know how to improve it. You’ve tried out one or two things in a halfway systematic way, but really you have no idea what’s working well and what’s not. On Tools vs Process Hamel made the good point that it’s probably not so useful to think about tools too much when thinking at this stage. Generally speaking what’s going on is most often actually a process problem and trying to go straight to ‘what tool do I need’ is probably avoiding the real issue. The Gulf of Specification in Practice This is the place where you are trying to translate intent into precise instructions that the LLM will follow. You’re trying to be explicit and specific in the hope that the LLM will do what you want it to do, and not do the things that you don’t want it to do. The obvious manifestation of this is people writing bad prompts. It might seem that it’s also present when you try to have an LLM solve one problem when it’s either unsuited for that task or the task needs to be broken up and so on, but that’s the sister gulf of generalisation. Here, we’re focused on how to improve the specificity of your prompts. When you split things up and highlight the fact that prompts are something that you’ll need to work on and to improve, it becomes clear that it’s something you wouldn’t want to outsource or to skimp time on. Really the prompt writing is the thing that you (at certain moments, and where it’s identified as the thing needing focus / improvement) want to be working on in partnership with domain experts. For small applications, you might be the same person as the domain expert! For bigger projects, you might be working with domain experts. Just be aware that often the domain expert might not necessarily be detached enough to be able to figure out what needs the focus, or where the weaknesses of a prompt are. That’s what the iterative process / error analysis and everything else that’ll be taught in the course is for (see below and see future posts). Another point Hamel made was about why prompts are actually so important: “you have to express your project taste somewhere”. Given that your application might be fully / mostly driven by LLMs, the prompt is actually a really crucial place to express this taste and as such might be thought of as your ‘moat’. I know just from having experienced a variety of LLM-driven applications, it’s quite easy to tell the ones where the product team gave their prompts and their specification some real love. It’s the difference between POC junk that will die a slow and lonely death and something that delights and solves real user problems. Gulf of Generalisation Shreya didn’t really get into the details around the generalisation gulf in practical terms in this lesson, but I think this one can be a sort of place of comfort for the technically-minded to make refuge in. It’s one where there’s a ton of tools and technologies and techniques to play with, and vendors also live in this space and try to claim that their particular product or special sauce is the thing to help you and so on. The Improvement Loop for LLM Applications We also got a high-level overview of the loop that allows you to iteratively improve an LLM application: The analyse, measure and improve loop; adapted from an image used in the course There’s a lot to unpack in all these different stages, and we didn’t really get into the details in the session today but you can see how this offers a really powerful way of thinking through what it means to iteratively improve an LLM application. Learning how to implement this in a practical way will be the main thing I want to get good at by the end of this course. The process is made up of a bunch of techniques, but in my experience companies or use cases that struggle with improving what they built also lack the scaffold of this loop to orient themselves. Prompting through the lens of evals As we explored above, prompting is sort of the table stakes of improving your LLM application. In order to get good at prompting, it can help to appreciate what they are good at and what they struggle with. So, as Shreya put it, “leverage their strengths and anticipate their weaknesses” (when prompting). At this point Shreya got into some points around what kinds of things went into a good prompt but I think I’ll write a separate blog on that and I don’t want to just regurgitate what we listened to. Today was more of a high-level introduction, and in any case it was much more about the outer-loop process instead of the inner loop (where tooling + specific techniques play more of a role.) A slide from a talk I gave about the inner loop vs the outer loop of GenAI development So it’s great that the course gets into the weeds (esp in the course materials, which include the draft of the book Hamel & Shreya are writing) but I think the really useful thing they’re doing is situating the tactical improvements and techniques within the strategic patterns and workflows that teams and individuals should be doing to work on these LLM applications. At a high level, what are we talking about: how to tease out failure scenarios for these applications and their behaviours conversely, how to understand exactly which domains it does well for Things I want to think about more There was a ton of really rich discussion around prompting in the Discord. I’m interested in exploring more: cross-provider prompting decisions (i.e. how prompting an OpenAI model differs from what you do with a Llama model or whatever) prompts that work with reasoning models vs non-reasoning models the tradeoffs of whether you put your instructions in system prompts vs user instructions In general there’s been a bunch of noise recently about so-called ‘leaked’ system prompts from a bunch of LLM API providers and I’ve mainly been struck by just how detailed they are. I consider myself pretty good at improving and iterating on prompts, but I’ll admit I’m not writing these multi-thousand word tomes. I’d like to explore which scenarios it makes sense to do so, and how to calculate at what point it makes sense from a cost or latency perspective to do so. As I’m sure you can detect, I’m really enthusiastic about the lesson to come and will work in the meanwhile on some of the readings that have been set as well as the homework task of writing a system prompt for a LLM-powered recipe recommendation application!
Google released an updated iteration of their Deep Research tool that uses the new 2.5 Pro model. This was taken from a post originally made on Twitter, so please excuse the terseness. First impressions: a bit too eager to jump into a deep research task even when I just ask a clarifying question quite verbose, just like the OpenAI version. Not sure why both play this up a lot. It looks impressive but in practice I think we need more entry points into this. The ‘Executive Summary’ and other concluding headers are nice touches but I feel maybe there should be some more adherence to user requests for short reports. (I get that as UI it’s maybe weird to think for 10 mins and then spit out a very concise version, but it might actually be more useful.) I continue to be annoyed about how these Gemini DR reports handle footnotes (i.e. as endnotes whacked on at the end of the report). Almost a deal-breaker IMO. It’s almost like GDR tries to show how scholarly and serious it is by giving you these walls of prose (vs OpenAI DR which throws in a lot more bullet points). Not sure one is better than the other but would appreciate a bit more flexibility! The portability of these reports has always been not great. Yes you can export them to Google Docs but markdown (+ other options) would have been much better. In practice, this means that whenever I use GDR the report stays stuck there and I’m far less likely to share it with anyone, whereas the OpenAI DR reports I drop parts/all into a Github Gist etc. These reports have been getting better and better, all things considered. I’ve been following along and using GDR from the early days (even pre-OpenAI DR) and this latest version is the best version of it so far (as you’d hope!) (It’s also a little bit annoying that GDR has removed any way to use the older versions of GDR with Gemini Pro 2.0 and 1.5 etc. Makes it harder to actually compare these things.) Please let’s get an ipad version of the Gemini iPad app soon, too? Feels a bit regressive to have to use GDR on the web interface always. For serious research (as opposed to simply generating a nice report on some area where you don’t know much about already), all these tools remain hamstrung by the quality of the sources. In areas where I am (or very recently used to be) a leading scholar / researcher, the difference between what I’d expect (in terms of taste / discernment for picking out these sources) is especially egregious. Make the models better, yes, but have better filters + retrieval. So yeah, these tools are getting good! Kudos to the teams who are implementing this stuff. Hard to make it perform reproducibly well on so many open-ended uses. But more work to be done! IMO the really great implementations of this ‘deep research’ pattern will all be in-house where you can have control over: source selection (i.e. high-quality inputs only, not just some random things on the internet) how long it spends thinking about a particular area / loop of the research (or decides to backtrack and dig deeper etc) output types / templates / length different modalities of Q&A (sometimes you want reports, other times you want a quick question answered, other times you want visual guides etc etc.) different models for different kinds of tasks possibly you have little sub-research agents / processes which will go off and work on some hypothesis, possibly involving actual datasets / analysis of tabular data etc, something clearly missing from the current versions we have A few other things: GDR’s ‘clarification step’ (which I’ve heard them discuss on podcasts etc) is not as good or useful as the OpenAI DR clarification questions. In practice, because it’s buried under a concealment button that you have to click etc, and where the entire UI seems to be screaming at you to ‘Start Research’, you basically never update or amend the research plan. And when you do, it’s really not clear what’s changed because you don’t get some feedback or diff that your comments were understood; you just get an entire new research plan (again buried under the concealment button) Going forward we’re probably going to want / need ways of navigating the layers to this research. A global overview report will have subsections that (should you wish) can be expanded into their own more detailed or granular reports. This is how research works, after all. Not just endless new reports all trailed one after another pointed in the same direction. The other thing that I think we’re really going to need to work on is research taste. Like the LLMs that power them, GDR and OpenAI DR offer a level of research taste developed to the mean. (I know people are thinking about this since it came up on Dwarkesh’s podcast with the AI 2027 guys, but they were focused on scientific research.) I think there’s not a single answer for this which is, again, why I see the end result as people bringing these things in-house where they get to develop and refine what makes their particular flavour of research unique. (In the human-generated research world this is very much the case, where certain institutions (or even particular authors) are known for how deep they go, or what kinds of sources they prefer, or how they choose to feature or highlight the primary sources they access, and so on.) There are many possible variations of how this manifest, and I hope that we’re headed into a world where all the AI ‘deep researchers’ will be unique and quirky in all the best senses of that word.
Finally back on track and reading the next chapter of Chip Huyen’s book, ‘AI Engineering’. Here are my notes on the chapter. Overview and Core Philosophy “Data will be mostly just toil, tears and sweat.” This is how we start the chapter :) This candid assessment frames dataset engineering as a discipline that requires both technical sophistication and pragmatic persistence. While the chapter’s placement might have been suitable earlier in the book, its position allows it to build effectively on previously established concepts. Data Curation: The Foundation Data curation addresses various use cases including fine-tuning, pre-training, and training from scratch, with specific considerations for chain of thought reasoning and tool use. The process addresses three fundamental aspects: Data Quality: The equivalent of ingredient quality in cooking Data Coverage: Analogous to having the right mix of ingredients Data Quantity: Determining the optimal volume of ingredients Quality Criteria Data quality encompasses multiple dimensions: Relevance to task requirements Consistency in format and structure Sufficient uniqueness Regulatory compliance (especially critical in regulated industries) Coverage Considerations Coverage involves strategic decisions about data proportions: Large language models often utilize significant code data (up to 50%) in training, which appears to enhance logical reasoning capabilities beyond just coding Language distribution can be surprisingly efficient (even 1% representation of a language can enable meaningful capabilities) Training proportions may vary across different stages of the training process Quantity and Optimization A key phenomenon discussed is ossification, where extensive pre-training can effectively freeze model weights, potentially hampering fine-tuning adaptability. This effect is particularly pronounced in smaller models. Key quantity considerations include: Task complexity correlation with data requirements Base model performance implications Model size considerations (OpenAI notes that with ~100 examples, more advanced models show superior fine-tuning performance) Potential for using lower quality or less relevant data for initial fine-tuning to reduce high-quality data requirements Recognition of performance plateaus where additional data yields diminishing returns Data Acquisition Process The chapter provides a detailed example workflow for creating an instruction-response dataset: Initial dataset identification (~10,000 examples) Low-quality instruction removal (reducing to ~9,000) Low-quality response filtering (removing 3,000) Manual response writing for remaining high-quality instructions Topic gap identification and template creation (100 templates) AI synthesis of 2,000 new instructions Manual annotation of synthetic instructions Final result: 11,000 high-quality examples Data Augmentation and Synthesis Synthesis Objectives Increasing data quantity Expanding coverage Enhancing quality Addressing privacy concerns Enabling model distillation Notable Research: An Anthropic paper (2022) found that language model-generated datasets can match or exceed human-written ones in quality for certain tasks. Note that some teams actually prefer AI-generated preference data due to human fatigue and inconsistency factors. Synthesis Applications The chapter distinguishes between pre-training and post-training synthesis: Synthetic data appears more frequently in post-training Pre-training limitation: AI can reshape existing knowledge but struggles to synthesize new knowledge LLaMA 3 Synthesis Pipeline A comprehensive workflow example: AI generation of problem descriptions Solution generation in multiple programming languages Unit test generation Error correction Cross-language translation with test verification Conversation and documentation generation with back-translation verification This pipeline generated 2.7 million synthetic coding examples for LLaMA 3.1’s supervised fine-tuning. Model Collapse Considerations The chapter addresses the risk of model collapse in synthetic data usage: Potential loss of training signal through repeated synthetic data use Current research suggests proper implementation can avoid collapse Importance of quality control in synthetic data generation Model Distillation Notable example: BuzzFeed’s fine-tuning of Flan T5 using LoRa and OpenAI’s text-davinci-003 generated examples, achieving 80% inference cost reduction. Data Processing Best Practices Expert Tip: “Manual inspection of data has probably the highest value to prestige ratio of any activity in machine learning.” - Greg Brockman, OpenAI co-founder Processing Guidelines The chapter emphasizes efficiency optimization: Order optimization (e.g., deduplication before cleaning if computationally advantageous) Trial run validation before full dataset processing Data preservation (avoid in-place modifications) Original data retention for: Alternative processing needs Team requirements Error recovery Technical Processing Approaches Deduplication strategies include: Pairwise comparison Hashing methods Dimensionality reduction techniques Multiple libraries are referenced (page 400) for implementation. Data Cleaning and Formatting HTML tag removal for signal enhancement Careful prompt template formatting, crucial for: Fine-tuning operations Instruction tuning Model performance optimization Data Inspection The chapter emphasizes the importance of manual data inspection: Utilize various data exploration tools Dedicate time to direct data examination (recommended: 15 minutes of direct observation) Consider this step non-optional in the process
More in AI
Internalize this graph, we post it on this blog a lot. If trends continue, China will have 10x the electricity of the US in 10 years. Whenever you think about geopolitical strategy, always keep it in mind, because whatever your plans are must consider this. Here’s a strategy that would make sense for America long term. Sell as many AI chips into China as possible, use the money to purchase solar panel and battery factories, and build out electricity capacity as quickly as possible. Fundamentally, compute is a function of electricity. That’s the sort of long term thinking that gives America a chance. With robotic labor, immigrants like Jensen and Elon, a business friendly culture, and an incredible chunk of land, it’s not so clear who is ahead in 50 years. Regardless, I do know this strategy would lead to massive prosperity for the citizens of both countries. However, that’s not close to what we see happening. We see a lot of short term zero sum thinking in action. It’s very scary what the US “AI race” mentality leads to. When exactly does the race end? Whatever capabilities you have today will eventually be matched, unless… …you believe that you can use ASI in a decapitating first strike. I’m worried that this is what these people see as the only way out. It probably doesn’t work, can you outthink steel and drone production? I don’t see how, see my critiques here. But I’m not totally sure, I’m not a superintelligence. And if it does work, that may be the end. My likelihood of imminent WWIII went up a lot with the nationalization of NVIDIA.
I really want a lot of venture capital investors to work on their self-esteem. After a few decades of supporting and mentoring New York City’s tech community, I hope I’m credible when I say I care a lot about how it thrives. So one of the most disappointing—and frankly surprising—things I’ve seen lately is how many investors in the greatest city in the world have been lowering themselves, sucking up for approval from a cohort of Silicon Valley tycoons who hate our city, hate our values, and want us to fail. I’m furious because there are so many good people and good ideas here that deserve to thrive, and aren’t getting the chance. Why would anyone sit by while outsiders put our community down, only to let those same outsiders profit off us when we succeed anyway? In fairness, I’m not talking about all of our city’s investors. Some are content to keep their heads down and hope the worst of the West Coast doesn’t notice them. But imagine if our other civic institutions worked this way: the Knicks timidly stepping onto the court and standing back while opponents charge the rim, or the Liberty letting the other team fire off shots unchallenged. This is not the New York City I know. What real New Yorker, when faced with a taxi cab headed toward them in a crosswalk, would say, “Oh, sorry, I was going to be walking here, but you go right ahead and run me over, good sir!” It’s pathetic, and it’s bad for business. Our city is watching, and our kids are too. HQ Zero I’ve been lucky enough to watch brilliant friends build innovative businesses here, having each other’s backs and catalyzing a community that lifted up the city we all love. And then, over and over, we watched our lawmakers line up to give billions in handouts to West Coast tycoons who’d already won the game. Even when the community stopped those giveaways, the companies still hired just as many people here as they’d promised—because people want to work in New York City. Our tech community's own data shows that, while the Bay Area lost tech jobs over the last 5 years, we've grown by 23%. We never needed to bribe these guys in the first place. (Unsurprisingly, many of the same people who once demanded access to New York taxpayers’ pockets started trashing our city and its most vulnerable residents the moment the freebies dried up.) The lesson is simple, for the public sector and private investors alike: we don’t need to buy the love of people who hate New York, or act as if we’re inferior and desperate to import an economy from elsewhere. We don’t do that shit here. We don’t need to be thirsty. We can invest in the people who already love this city enough to build the future here. We are, you may learn, second to none. Fortunately, we’ve replaced the timid, backwards-looking leadership of the past with people who believe in our ability to lead, build, and invent a better future. It’s no surprise that New Yorkers are more optimistic than they’ve been in decades, or that we have the lowest crime in recorded history—nor that the people who hate our city are lying about all of it. The very same tycoons who felt entitled to taxpayer handouts now feel entitled to treat our investors as feeder funds for their own goals. There is no future in funneling deals to firms trying to undermine our entire civic ecosystem. Building value, having values I’m still committed to the radical social ethos that you should be able to build tech companies worth hundreds of millions of dollars, that founders and employees can get fabulously wealthy while workers are paid fairly, that investors earn a solid return, and that everybody pays an appropriate tax rate to the city that made it all possible. I don’t care if it makes me a radical to think those opportunities shouldn’t be hoarded by the same three or four predatory jerks who can’t stop trashing New York. It’s no secret that the rise of authoritarianism and bald-faced bigotry among Silicon Valley investors has led them to demonize our city. What’s almost too much to bear is watching people I’ve respected in New York look the other way while those leaders embrace rank misogyny, blatant antisemitism, unapologetic Islamophobia, and profound racism—so virulent that the most powerful venture capital firm in the world hired a partner on the basis that he strangled an unarmed Black man to death. I’m grateful that many of you tell me privately that you object and still hold onto your moral compass, which makes it all the more heartbreaking to watch so many go silent publicly. Where is your self-respect? Because I’ll be honest: they are laughing at you for debasing yourselves in front of them, and they enjoy shitting on the city you call home. You should do work you’re proud of, that you feel good telling your family about, and that you’re excited to share with your community. You should also do the math. A tiny handful of firms control the vast majority of late-stage funds, and you are not going to differentiate yourself by being a mindless feeder fund for the same 10 cancer capital firms as everyone else. If your only edge is competing on who can kiss Andreessen’s ass the most, you’re already fucked, kids. Look at how they already have to tap-dance—calling things “super intelligence” instead of “artificial intelligence” just to prove their fealty. How long until you’re asked to publicly rename your work, or tell a big lie about your city, just to stay in the room? I’ve heard too many people in this town claim they don’t have any power, that their only job is returning capital to LPs. All you have to do is the simplest thing in the world: refuse to play along. By contrast, if you bet on this wonderful community, connect to the next generation of innovators, and fight for the future instead of against it, there is limitless opportunity here. If you have to put it in a way your LPs can understand, explain that there is huge alpha in going the way that a16z is not. Call it “Anti-authoritarian Dynamism”. Explain that you’re going to strangle their deal flow, since founders of color and next-gen innovators don’t want to talk to the firms who hate New York City and hire killers. Tell them you’re choking off the supply chain of capital to the office where Daniel Penny works. Side with the founders, not the funders Just in case you don’t already know: Zohran Mamdani is your best friend. He wants the city to win, and our businesses to thrive. And those West Coast VCs who act deranged when you mention his name? They want your city destroyed, and when it's not, they lie and say that it already has been destroyed. Pick the side of your neighbors and your community, and speak up for the founders and entrepreneurs building the opportunities you want to be part of. Just as importantly, speak up for yourself. It’s not viable in the long term for every outcome—and the small number of possible exits for your portfolio—to be controlled by the same handful of gatekeepers. I used to love helping friends turn cool new products into companies that provided good jobs; now that’s virtually impossible because the scale has gotten so absurd, and so few players decide who gets to participate. When I talk to people who are still early in their tech careers, the only names they know in the investor world are the usual suspects — not yours. They’ve never heard of your firms, of many of New York's most thoughtful and interesting and supportive investors, and they’re downright skeptical when I tell them to work with NYC-based investors instead of making a trip to Sand Hill Road. That all changes the moment New York’s tech leaders put their power behind an alternative narrative. Simply saying “we are here, we exist, and we are not a satellite of the same tired agenda you’ve been hearing from Silicon Valley for years” would be hugely valuable for the entire New York City tech ecosystem. It's going to be especially resonant after the entire country just picked a political path that looks a lot like New York City's values — and is based on rejecting Big Tech's values. It would also, I expect, do wonders for your soul after years of enduring the quiet indignities of watching some horrid people chip away at your self-esteem. Now, I know most of you aren’t inclined to go to war. That’s fine — you can still be nice. Jalen Brunson is really nice! Karl-Anthony Towns is really nice! Sabrina Ionescu is nice! But they really like to win, and they don’t take shit from anyone. You have to decide if you want to win, or if you want to get bullied on the playground by the people who actually have their heads in the game. As a repeat founder who spends my time with makers — both the next generation of creators, and the people who’ve built great companies before them — I know where my loyalty lies. Those of us who build things know who we are. Those of us who love New York City are having one of the great moments of the modern era. Do you know who you are? Are you going to join us?
[Everyone is doing guest posts these days, so here’s one about the Weil conjectures from (in alphabetical order) Bin Dong, Nailin Guan, Jiedong Jiang, Ruochuan Liu, Christian Merten, Xiangyu Pan, Tingxiao Shi, Bin Wu, Edison Xie and Wei Yang] We … Continue reading →
20 years ago to this day, I started my new technical blog. I'd had a personal one for some time, but most of the stuff that happened there had already migrated to social media. I wanted somewhere to post the things that would only be of interest to my techier friends, and maybe of some use to anyone else that came across it. Things have changed a bit, but they've stayed the same in some ways. Earlier this year I finished my 48-part (!) LLM from scratch series, but it had predecessors -- the third post I made here was part one of a 13-part series where I worked through setting up automated backups for my homelab NAS, and then later on I spent a few months building a reverse proxy in C, not to mention the eight-month, ten-post series on fine-tuning LLMs. It does seem like there's some value to putting all of this online. I rarely check my logs and referrers, but people do still seem to be arriving here having searched for stuff that I've written about, and every now and then there's a kind "thanks for this!" comment left on an old post where I explained something I'd just learned. I wrote a bit more about the blog's history last year, and came to the conclusion that the best posts -- both for me to write, and as far as I could tell for other people to read -- were the ones where I wrote up a project that I'd been working on. I blogged about that in more detail, and it seemed to strike a chord on Hacker News; I think it still holds. Recently, Sean Goedecke suggested that the best blog posts are those where you write about things that you're in the process of learning, and I agree. You learn more than you would if you didn't write it up, and because things are fresh in your mind, you may well write more clearly about it than an actual expert in the subject might have done. Anyway, I'm going to carry on blogging. I'm working on my biggest LLM-related project so far (dozens of training runs, so my study aircon is getting quite the workout even with the weather turning colder), and look forward to posting about it later on this month. I hope it's at least as fun to read as it is to write!
Announcing STABL: Societal Transformation and AI Bottlenecks Lab