How a LinkedIn Post Led Me to a Munich Basement with Millions of Euros Worth of Robotics Equipment My LinkedIn feed has become a stream of robotics content over the past few months. As someone diving deep into AI robotics after years in ML/AI/RL, I’ve been deliberately connecting with people pushing the boundaries of the field. So when Nicolas Keller’s post about Munich being “the world’s best place to build robots” appeared in my feed, it immediately got my attention. A bimanual manipulation hackathon in Munich, organized in just three weeks. How cool is that? Here’s what they promised: Hands-on with dual-cobot humanoid upper-body setups, 2 Franka Emika Pandas, 2 depth cameras, 1 rgb, RTX5090 in each station Teleoperation with Meta Quest VR headsets Contributing to the MINGA research paper (with co-author potential) Collaborating in Munich’s growing robotics ecosystem I had three weeks to apply, get accepted, and arrange travel during what turned out to be Oktoberfest (the Lederhosen on the robot in the picture should have been a hint! It just made the stay extra expensive). I have recently missed the LeRobot’s worldwide hackathon and wanted to jump on the opportunity. The prospect of 30+ dual-arm robot setups, high-end GPUs, industry mentors and meeting other people passionate about robotic manipulation made it a no-brainer decision for me. The hackathon experience Most hackathons are short and they only involve software. Hardware Hackathons are much more rare, especially where hardware is provided and high end. The organizers promised a lot - and I have to say that they over-delivered, even though they operated on a very short timeline! 3 weeks! The hackathon wasn’t perfect; we hit some technical issues with the provided codebase and the robot controllers.The lab space (KI Fabrik in the Deutsches Museum) was full of amazing robots, powerful workstations, and a 3D workshop, but it also had its downsides - it was hot, humid and you could get trapped there due to the limited number of keys! The schedule was focused on building: after one day of setup and tutorials, it was essentially “09:00–open ended — Building time” for six straight days. The main communication happened on Discord. The setup was industrial-grade: 30+ Franka Emika Panda dual-arm configurations for about 40 participants. Each setup came with Meta Quest headsets running custom teleoperation software. There was a full workshop with 3D printing capabilities for custom grippers. The compute power was serious—workstations with the latest hardware that most of us don’t have access to. The initial goal of the organizers was to attract local students (mostly from TUM), but the hackathon was just too attractive. The organizers ran the selection process based on a Typeform where you had to justify your presence (CV, motivation, experience) and the final mix of people contained: PhD researchers, startup founders, industry engineers, and ambitious students. There was a significant number of people who traveled from other countries. Everyone wanted to be there. Industry engagement and realistic use cases The real differentiator was the industry backing. BMW and Siemens provided realistic challenges to be solved, explained the details, provided physical materials and sponsored prizes. Additionally, there were helpful lectures and mentoring from Nvidia, Hugging Face LeRobot and KIT (a big German university). When Sunday’s final presentations came, BMW and Siemens employes showed up to judge the results personally. This wasn’t academic theory - teams were working on problems that companies actually need solved, with the decision-makers accessible throughout and present for the final outcomes. The main organizers were TUM and Poke & Wiggle. I was very impressed with both. TUM showed great support for students and entrepreneurship. Poke&Wiggle people pulled everything together from the technical side. They were staying late and even hosted some participants coming from abroad in their homes! They were testing their own software stack while building what they claimed would become the largest public bimanual manipulation dataset. My strategy & Learning Probably my favorite thing about this hackathon was that it was extremely collaborative. Yes, people tried to win, but I was able to learn both from my team as well as from others. Team formation & collaboration I came to the event without knowing anyone and needed to form a team. It was actually a pretty common experience, many people didn’t know who to pair with. I have been in setups like this before and that experience helped. To form a good team you want the best people, but also it’s really hard to assess people very quickly and people who seem great at the start might not be able to give the best performance. E.g. they might not be fully available, you might not get along very well, etc. So my strategy was to talk to most people, see how they think and how experienced they are. I was selecting for getting along, enthusiasm and general intelligence. It was less important to me that someone was inexperienced as long as they were energetic and open minded. We weren’t the most effective or best organized, but we really enjoyed our time together, made good progress and learned a lot. Our team also shifted a bit during the 7 days (one person got sick and we adopted another one). We used a WhatsApp group to share resources, set up a GitHub repo for shared scripts and shared some notes on a Google doc. Our strategy was as follows: get familiar with the teleoperation trying out all available tasks and assess the task feasibility for the human operators train and deploy the models ASAP to test the pipeline (yes, we detected bugs and further limitations) pursue the most promising tasks, refine the dataset collection and experiment with the models for good performance Learnings We didn’t manage to win any categories. I think that my group was more focused on learning and experimentation, instead of purely competing to win. Some takeaways: If the task can’t be done by the human, the robot won’t be able to do it Training loss during model training is not a good predictor of the real life performance Robot safety mechanisms were critical to avoid breaking the robot Evaluation in real life is risky and some simulation setup would be helpful End-effector control with inverse kinematics was often causing the robots to get stuck due to joint limits; it required special care during teleoperation for the demos so that the actual policy wouldn’t block the robot Two arms are much harder than one Dataset quality matters a lot (recovery examples, non-Markovian states are confusing, noise/operators in the setup) What we tested 3 different task setups with hundred+ demonstrations each variations in model training (steps, parameters, models, action space) and dataset selection (recovery episodes ratio, bad episodes) different control modes (delta and absolute) we could only try actions in the EE space, the joint space controllers weren’t working correctly SMOLVLA and ACT models from Lerobot libraries performed experiments for generalization and resilience (e.g. messing with cameras was making robots much less effective) we also visualized attention maps for the ACT Models Here is an example of an attention map: (Kudos to physical AI Interpretability Repo - the author was there during the event and helped us a bit with the setup). Misc I got a pretty good feel for teleoperation in VR (it’s hard though!) I managed to get the arms to crash with each other and got my robots stuck countless times I spent some time setting up a simulation environment with Panda in MuJoCo and playing with Isaac Sim (approach abandoned in the end) I read several papers recommended by other participants I wasn’t able to try Pi0/Pi0.5 as the ready snapshots are for the same type of robot (Panda), but in a very different action space and we didn’t have time/resources for fine-tuning from scratch (80GB+ GPU memory required) What I wished I could do: try out Groot / Pi0.5 simulation, sim-to-real, and RL fine-tuning for the trained VLA (a simple VLA would be perfect!) Posts describing the experiences of some of my teammates: @Artur and @Andrea. We Need More of This After seven days of intense collaboration in a basement, we didn’t revolutionize the future of robotics, but we all learned a lot and everyone came back home more experienced and inspired. It was a great event! And I want to see more events like this in Europe, because the future of robotics doesn’t have to happen in SFO (or China) Europe has the ingredients for world-class robotics innovation. We have strong engineering talent and strong reasons to invest (aging population)! I live in Poland now, a place that produces some of the world’s best software engineers, but the innovation is lacking. I talked to two universities in Warsaw and they don’t really innovate or even follow the current state of the art yet for embodied intelligence. And it’s a shame, because with libraries such as LeRobot, open hardware, and open-source simulation engines, the space is now much more accessible. I was very impressed with TUM and many of the students. I would like to support the ecosystem in Poland and Europe. Munich proved it’s possible. Let me know if you would like to help! Next steps I left the event pretty drained, but also very excited! JI am still following up on the various threads I started during the hackathon. I also started to look at another exciting challenge that is focused on household tasks in the simulation. Currently I’m especially interested in the approaches combining VLAs with RL, like the ones outlined in the SimpleVLA paper and would love to participate in more hardware hackathons, ideally combining simulation and real world learning.

Part 1 of a series on practical robotics experimentation As part of my journey into robotics, I found myself facing a classic problem: I wanted to experiment with different learning approaches for robotic manipulation, but I needed a flexible playground where I could quickly test ideas without being locked into any single framework or workflow. The result is gym-so100-c, a simulation environment built around the Standard Open Arm SO101 that bridges multiple machine learning libraries—Stable-Baselines3, the imitation library, and Hugging Face’s LeRobot—all in one cohesive environment. The “Sim-First” Decision Even though I had access to physical robots, I chose a sim-first approach for a simple reason: I’m more of a software person who enjoys the comfort of my home office and the flexibility to keep working while traveling, rather than spending long days in a lab. This decision shaped everything about the project. I needed a simulation that was: Physically realistic enough to eventually transfer to hardware Fast enough for thousands of training episodes Flexible enough to work with different learning paradigms Simple enough that I could understand and modify every component Why Build Another Gym Environment? You might wonder: why not just use an existing simulation? I wanted something both flexible and reflecting my hardware platform well in simulation. The robotics world offers many compelling options. The Simulation Landscape I Considered: Isaac Sim was tempting—NVIDIA’s powerhouse with photorealistic rendering and advanced physics. But it requires a proper GPU setup, and I wanted to start experimenting immediately rather than waiting for hardware upgrades. ManiSkill is an exciting newer option with great task diversity and modern ML integration. I’m actually quite excited to try this next—it seems to hit the sweet spot of realism and ease of use. Gazebo/ROS represents the traditional robotics stack: mature, well-supported, with endless plugins. But the learning curve felt steep for someone coming from a pure ML background, and I wanted to focus on learning algorithms rather than robotics middleware. PyBullet similar to MuJoCo. Also popular in RL enviornments. Why I Chose MuJoCo + gym-aloha:The answer came down to immediate productivity. I could adapt gym-aloha and start experimenting within days, not weeks. It’s proven in the papers I was trying to replicate, has excellent contact modeling for manipulation, and enjoys a large ecosystem of compatible tools. The Gym Interface Standard:OpenAI Gym (now Gymnasium) defines a standard interface that every reinforcement learning environment implements: obs, info = env.reset() # Start a new episodeobs, reward, terminated, truncated, info = env.step(action) # Take an action This simple interface is incredibly powerful because it means the same environment can work with: RL libraries like Stable-Baselines3 (SAC, PPO, HER…) Imitation learning libraries like imitation Custom training loops or evaluation pipelines Any future framework that follows the standard The Evolution Plan:This environment is just the beginning. As I move toward more realistic scenarios, I’ll likely migrate to Isaac Sim or ManiSkill. But for rapid prototyping and algorithm comparison, this MuJoCo setup has been perfect. Key insight: There are many good options. If your requirements are not very specific, look for something popular and that has something similar to what you need that you can quickly adapt. This flows naturally from the question and sets up the technical details that follow. Standing on the Shoulders of ALOHA Rather than building from scratch, I adapted the gym-aloha project, which implements the dual-arm ALOHA platform used in several influential imitation learning papers. The ALOHA Foundation:ALOHA (A Low-cost Open Hardware Arm) proved that effective manipulation learning was possible with relatively simple hardware. The gym-aloha implementation provided: MuJoCo physics foundation Well-designed observation and action spaces My Adaptation:I modified gym-aloha for a single SO101 arm (5-DOF + gripper) to match my hardware target: # Simple registration exampleregister( id="gym_so100/SO100CubeToBin-v0", entry_point="gym_so100.env:SO100Env", max_episode_steps=700, nondeterministic=True, kwargs={"obs_type": "so100_pixels_agent_pos", "task": "so100_cube_to_bin"},) Once registered, creating the environment is straightforward: import gym_so100 # triggers env registrationimport gymnasium as gymenv = gym.make("gym_so100/SO100CubeToBin-v0")obs, info = env.reset() The Task: Cube-to-Bin I focused on one fundamental manipulation task: bin-a-cube. A red cube starts at a random position on the table, and the goal is to place it inside a fixed gray bin. This task is deceptively simple but covers the core challenges of manipulation: Perception: Locating the cube and understanding spatial relationships Planning: Approaching the cube from a graspable angle Control: Executing smooth, coordinated motion Manipulation: Grasping, lifting, and precise placement The MuJoCo scene includes: Robot: SO101 single arm with position-controlled actuators Workspace: Table, free-moving cube, and goal bin Sensors: Joint positions, gripper state, and camera views Sites: Tracking points for reward computation and success detection Control Paradigms: Joint vs. End-Effector Space Following gym-aloha’s design, I implemented joint-space control where actions directly specify target joint positions: Aspect Joint-space control Action Target joint positions → data.ctrl Control loop Actuators drive joints toward commanded positions Learning Policy learns in robot’s natural DOF Transfer Direct mapping to real hardware I chose joint-space over end-effector control for cleaner transfer to my target hardware. While end-effector control (where you specify gripper poses and let MuJoCo’s constraints solve for joint angles) can be more intuitive, it adds complexity that I wanted to avoid initially. What’s Inside the Environment As of August 2025, gym-so100-c includes: Simulation Environment & Tasks: MuJoCo-based SO101 simulation environment Cube-to-bin task with configurable reward shaping Training Integration Scripts: Reinforcement learning with Stable-Baselines3 (SAC) Imitation learning with the imitation library Training integration with LeRobot (ACT, Diffusion, VLAs) Control & Data Collection: Teleoperation via keyboard or gamepad Episode recording for demonstration datasets Dataset conversion to LeRobot format Early Design Lessons Physics Engine Choice: MuJoCo was the obvious choice for its speed, stability, and excellent contact modeling. It works well on CPU, it was perfect on MacBook. I am excited about IsaacSIM, but I don’t have a powerful GPU yet. I am open to trying other engines in the future. Observation Space Design: I experimented with different observation combinations: pixels_agent_pos: Camera images + joint positions agent_pos: Joint positions only (for faster training) pixels: Camera images only (for vision-based policies) The mixed approach (pixels_agent_pos) worked best, giving policies both rich visual information and precise proprioceptive feedback.This is also what ACT paper and most SOTA approaches are using. Reward Engineering: In Reinforce Learning an agent collects rewards interacting with the environment and tries to maximize them. The trick is to make such rewards that the agent does what you want from it and also that it can learn pretty well. Shaping the reward structure (designing the incentives) is called Reward Engineering and it’s suprisingly tricky to get right! For example if you reward getting close to the block too much, the agent might chose to hover over it and never grab it. If you don’t give any rewards until the task is complete and if the task is too hard, there is no feedback to learn from. I implemented both sparse rewards (success/failure only) and dense rewards with approach shaping. Dense rewards proved much more reliable for learning, though they required more careful tuning. I also implememented HER (hindsight experience replay) that automates the reward shaping by trying to teach the robot behaviors from previous trajectories, but the results were mixed in my setup. The Integration Challenge The real value of this environment isn’t just the simulation — it’s the integration layer that lets me seamlessly move between different learning approaches. In the next post, I’ll dive into what I learned from training experiments with SAC, behavior cloning, and modern imitation learning methods. But the foundation was crucial: having a single environment that could work with multiple learning paradigms, generate consistent datasets, and provide reliable evaluation metrics. Sometimes the unglamorous infrastructure work is what makes everything else possible. Next up: Training experiments and what I learned about the practical differences between reinforcement learning and imitation learning approaches. What’s your experience with simulation environments for robotics? Have you found certain design decisions that made experimentation much easier or harder? I’d love to hear about it.

A deep dive into building an automated illustration pipeline for storytelling applications Introduction AI image generation has revolutionized creative workflows, but there’s a significant difference between generating a single stunning image and producing hundreds of consistent illustrations for a complete project. When building storylearner.app, we faced the challenge of generating book illustrations that maintained visual consistency while telling compelling stories through imagery. We’ve used powerful Gemini multimodal models, mostly because they are fast and because the experimental ones are available for free. This article explores the technical and creative challenges of large-scale AI illustration generation, showcasing techniques for achieving visual consistency and building robust pipelines that can handle the complexity of full book illustration projects. Below are images created for a chapter of a book adapted for the storylearner platform: The article has a companion colab notebook, so you can play with the examples yourself. The Consistency Challenge Understanding Visual Consistency in AI AI image generation models operate somewhat like a company of talented artists, each with amnesia. Every generation is essentially a fresh start unless you provide explicit context. This creates unique challenges when you need to maintain character consistency, style coherence, and narrative flow across hundreds of images. Consider this simple experiment: generating three images of “the same pig with wings and a top hat flying over a futuristic city” in different weather conditions. Even with identical prompts and seeds, subtle inconsistencies emerge—eye colors change, proportions shift, and the overall character can feel different. The Impact of Seeds and Prompts Seeds act like selecting a specific artist from your AI company. Using the same seed with identical prompts yields consistent results, but even small prompt variations can dramatically alter the output. We discovered that: Same model + same prompt + same seed = identical results Same model + same prompt + different seed = completely different character Same model + slightly altered prompt + same seed = often produces a different character entirely This sensitivity means that scaling up requires careful orchestration of all these variables. Building Consistency Through Reference Images The Reference Image Approach The most reliable method we found for maintaining character consistency involves using reference images. Here’s how it works: Generate an initial character/scene using carefully crafted prompts Upload this image as a reference for subsequent generations Include explicit instructions like “Use the supplied image as a reference for how the pig should look like” This approach significantly improves consistency, though it can also cause new issues in some very specific cases. With the following reference image: You can get this different, but consistent one! Here is an example of triggering a specific case: This specific case can happen when a prompt is pretty similar to the one that generated the initial image and the seed is identical.So, when using the reference image and similar prompt, I would actually recommend to change the seed or not set it. See the reference colab for details. Practical Implementation # Upload reference imagefiles = [client.files.upload(file="reference_character.png")]# Create parts with reference and promptparts = [ types.Part.from_uri( file_uri=files[0].uri, mime_type=files[0].mime_type, ), types.Part.from_text( text=prompt + "\nUse the supplied image as a reference for character appearance" ),]# Generate with referenceresponse = client.models.generate_content( model=IMAGE_MODEL, contents=parts, config=types.GenerateContentConfig( response_modalities=['Text', 'Image'], # No seed set. )) The Storylearner.app Illustration Pipeline High-Level Architecture Our production pipeline consists of three main stages: Idea Generation: Story text + guidelines → 3 illustration concepts per scene Idea Selection: Multiple concepts → best ideas chosen for the complete set Image Generation: Selected ideas + style guidelines + reference images → final illustrations Stage 1: Brainstorming Illustration Ideas Rather than feeding story text directly to image generation (which often produces poor results), we separate conceptualization from execution: class IdeaGenerator: def generate_illustration_ideas(self, text: str, context: str): prompt = """ You are a visual scene designer. Based on the story below, describe 3 different highly detailed and imaginative illustration ideas. Do not include any people or humanoid figures. Focus on setting, atmosphere, lighting, symbolic objects, and environmental storytelling. Story: {text} Context: {context} """ # Returns 3 detailed scene descriptions per text excerpt This approach generates rich, detailed scene descriptions that serve as blueprints for image generation. Stage 2: Intelligent Selection To avoid repetitive illustrations (like “a ship, a ship, a ship” in a sea voyage story), we use an AI selector to choose the best combination of ideas: class IllustrationSelector: def select_illustrations(self, chapter): prompt = """ Choose the best idea for each illustration considering that: - The set should be diverse - Illustrations shouldn't contain people - Prefer illustrations matching the provided titles """ # Returns optimal selection indices Stage 3: Consistent Generation The final generation stage uses: Style guidelines (detailed visual specifications) Reference images for style consistency Persistent chat sessions for maintaining context Retry mechanisms for handling API limitations Visual Guidelines and Style Consistency Crafting Effective Style Guidelines We developed comprehensive style guidelines that go beyond simple style names: Style: watercolorTechnique: Combine soft watercolor washes with fine ink line work for contrast and detail.Brushwork: Embrace visible brush strokes, blooming, and natural texture.Ink Lines: Use varied line weights for depth; apply cross-hatching or stippling for texture.Color Palette: Limit to a few harmonious hues with gentle gradations.Forms: Use simplified, geometric shapes; focus on essence over detail.White Space: Treat negative space as part of the composition.Texture: Highlight watercolor paper's natural texture and color variation.Atmosphere: Create light, airy scenes with openness and subtle contrast.Aesthetic: Preserve a hand-drawn look—embrace imperfections and human touch. Chat-Based Generation for Context Continuity Using persistent chat sessions helps maintain consistency within illustration sets: def generate_set_of_illustrations(ideas_with_file_paths, pass_image=True): chat = client.chats.create( model=IMAGE_MODEL, config=types.GenerateContentConfig(response_modalities=["Text", "Image"]), ) # Initialize with style guidelines and reference image initial_prompt = f""" You are a creative artist helping on an illustration project. Create {len(ideas_with_file_paths)} beautiful illustrations. VISUAL GUIDELINES: {VISUAL_GUIDELINES} """ # Generate each illustration within the same chat context for idea in ideas_with_file_paths: response = chat.send_message(format_illustration_prompt(idea)) # Process and save generated image Avoiding Common Pitfalls Critical Design Decisions Through extensive experimentation, we identified several key strategies: Avoid Human Close-ups: Character face consistency is extremely challenging. Focus on environmental storytelling instead. No Violence or Gore: Keep illustrations family-friendly and avoid content that might trigger safety filters. Diversify Scene Types: The selection stage prevents repetitive imagery across the complete set. Decouple Ideation from Generation: Separating concept creation from image generation improves both quality and debuggability. Real-World Example: Illustrating The Three Musketeers Let’s walk through illustrating a chapter from The Three Musketeers: Context and Settings First, we establish the story context: Setting: France, primarily Meung and Paris, early 17th centuryHistorical Context: Political tensions between French monarchy and Cardinal RichelieuMain Characters: D'Artagnan, Athos, Porthos, Aramis, Cardinal Richelieu, Milady de Winter Generated Ideas For the chapter opening, our system generated these concepts: “The Jolly Miller Inn Chaos”: Exterior scene with a yellow pony, scattered debris, and dramatic lighting hinting at recent altercation “Broken Sword”: Close-up of shattered steel on cobblestones, symbolizing lost honor and broken dreams “Inn Kitchen Aftermath”: Dimly lit interior with earthenware, bandages, and flickering candlelight Selection and Generation The selector chose the most diverse and narratively appropriate ideas, which were then generated using our reference image and style guidelines, producing illustrations that maintain visual consistency while telling the story effectively. Key Insights and Best Practices Consistency vs. Perfection: Perfect consistency isn’t always necessary—visual coherence in style and mood often matters more than exact character matching. The Artist Analogy: Think of AI models as artists with amnesia. You need to provide context, references, and clear instructions for each interaction. Pipeline Modularization: Breaking the process into idea generation, selection, and execution improves quality and maintainability. Style Guidelines Matter: Detailed, specific style descriptions work better than simple style names. Reference Images Are Crucial: Upload and reference style examples for best consistency results. Long Sessions are Fragile: It often works until a point, and at some point it fails poorly, e.g. inserting objects from a previous illustration into the following ones. Conclusion There is still a huge gap between a carefully handcrafted demo on an AI company blog and practical usage of the technology at scale. Things don’t work out well straight out of the box, but you can make these amazing tools work for you with help of systematic thinking about consistency, quality control, and robust engineering practices. While challenges remain, I hope you’ll enjoy the techniques we’ve developed at storylearner.app to power your own projects!

In this post I’ll walk through how I upgraded a set of tangled Python notebooks—responsible for thousands of LLM calls—into a robust content-adaptation studio powered by: an async FastAPI pipeline, a disk-first Next.js frontend, and a small suite of custom CLI tools. Re-engineering the stack was essential for my own sanity: the notebooks were fragile, slow to iterate on, and far too labor-intensive to babysit. I was also facing content quality challenges that were pretty much impossible to address in the old code base, that re-engineering unlocked. My hope is that the story also nudges you to build (or level-up) your own tooling instead of settling for one-off notebooks. We’ll cover: Engineering constraints – huge text volumes, strict meaning preservation, multiple target languages. The original notebook setup – what worked and where it hurt. Pain points – why small hacks no longer cut it. The new architecture – key design choices, novel elements (with screenshots), and how they solve the earlier pain. Outcomes & takeaways – higher quality, less toil, faster experiments, and patterns you can reuse in your own LLM workflows. Shape of the problem Shape of the problem: large quantities of text that need to be processed in a very specific way: retain meaning of the original (no text disappearing or altered significantly) consistent between parts translated & simplified resulting content needs to consistently have good quality automatic quality assessment and quality repair (revisions) A similar challenge would be relevant in translating legal documents, healthcare documents, etc. Story Learner Book Adaptation needs In my project StoryLearner I offer adapted books for language learning at a specific level. For example, “Las Aventuras de Sherlock Holmes, in A2, Spanish”. We use LLMs for both language/level adaptation as well as for illustrations.It’s a lot of LLM calls (easily thousands for a single book), because books are long and there are many elements for a single adaptation. A book has chapters, chapters have parts for easier reading. Each book/chapter/page has an custom illustration. Additionally, there are titles and descriptions to be adapted and transcribed. Books are long and we want the adapted text to retain the meaning, while making the language simple (aligned with the target level), natural sounding and correct. Trouble with a flaky, slow pipeline and hard to assess output The pipeline was was quite a feat! It was a lot of LLM calls, built mostly in colab/jupyter notebooks. RAW BOOK | v[book_stripping] | v[chapter_extraction] | v[chapter_simplification] (English) | v[chapter_partification] (English) | \ | \ | ---> [illustration_generation] | v[adapt] ──▶ [Lang 1] │ [Lang 2] │ [Lang 3] │ [Lang 4] │ [Lang 5] │ [Lang 6] | Illustration generation was on its own a pretty interesting pipeline (more about it in a separate post!). Here is one of the adaptation notebooks: As you can see, it has its own table of contents on the side. It’s easily thousands of lines of code and prompts. And the hundreds of outputs (text and images) could make it very, very, long. To the point that it would have rendering issues. On top of it there was: Separate notebook for book_narration Another notebook for upload to storylearner (via API). The “pipeline” worked. It serialized partial outputs in a way that partial redos/continuations were possible, it offered decent visualization. It was adjustable (just add/tweak a notebook cell!). However it was fragile and assessing quality/redoing content was painful and slow. And it was very frustrating to me, especially since the quality was important to my partners (language schools). Main pain points LLM reliability issues (no resources, surprise safety controls kicking in, running out of quota or LLMs not following instructions) broke downstream steps. The adaptation process was slow, it wasn’t taking advantage of paralellization well The pipeline was already so brittle enough that meaningful experiments were nearly impossible Colab/notebooks encouraged slapping things together instead of proper engineering with encapsulation and tests Python notebooks having rendering bugs because they were so long and had so many outputs (large text/many images) Low confidence in language level, name/format consistency, and preserved meaning without having a strict review/repair process and having some examples of problems with quality. Reviewing 60+ chapters across six languages was slow and manual - was infeasible for me. A copy of an adaptation notebook per book (for visualization/auditability) was duplicating the code and making it harder to maintain New Content Adaptation Tools I haven’t built everything at once. I started with a frontend using the existing disk format of book adaptations, then as it was easier to see what was going on, I progressively built more and more backend migrating specific functionalities. Fast API gives a nice UI out of the bat to call APIs, which was nice for trying things out. But the workflows were simply to long to drive them by hand, so that is how CLI tools came to be. CLIs/Frontend were in big part written by copilot coding agent. I also extensively discussed the component prompts with LLMs 😀. Backend – bookadaptation (FastAPI) all functionality behind endpoints, all async / internally parallel where safe (28 separate endpoints) All IO and LLM calls done async Pipeline stages are classes; 11 adapter subclasses (Gemini Flash, Gemini Pro, GPT-4o, etc.). skip and use_cached flags run a no-op or reuse artefacts while the file tree stays unchanged. STAGE_DEPENDENCIES mapping declares primary & secondary inputs and outputs for every stage. Hierarchical on-disk structure; files get a _{revision_number} suffix for multi-round outputs. Heavy use of controlled generation (using schemas) SQL instrumentation: All prompts, schemas, settings, outputs, latency, and errors logged to SQLite. Frontend – ContentTools (Next.js Server Components) Specialised views: Book language adaptations Illustrate all artefacts of the adaptation pipeline (inputs, outpus) Easy debugging of what happened during the QA Easy comparisons between experimental implementations “Final for publish” view. Illustrations: All book illustrations as grids (chapters, chapter parts) Illustration deep dives - ideas and the best ideas Reads JSON & WebP directly from disk—no extra HTTP hop. CLI tools (all async friendly python) adaptation CLI Adapts and revises every chapter until it’s good enough Drives the pipeline stages implemented in the service (many endpoints right) Gracefully retries LLM driven decisions: Uses output from the overall review to finish the chapter or go for more rounds or partially skip stages narration CLI illustration CLI publishing CLI Everything runs locally, but could as well run on a server. Yes, I ended up building a lightweight custom pipeline orchestration… 💀 Flow of a book adaptation with automatic QA (multiple revisions) RAW BOOK | v[book_stripping] | v[chapter_extraction] | v[chapter_simplification] (English) | v[chapter_partification] (English) | \ | \ | ---> [illustration_generation] | v[adapt] ──▶ [Lang 1] │ [Lang 2] │ [Lang 3] │ [Lang 4] │ [Lang 5] │ [Lang 6] | v[review_chapter] ←──────────────┐ | │ v │[revise_chapter] │ | │ v │[review_consistency] │ | │ v │[revise_consistency] │ | │ v │[review_meaning_cohesion] │ | │ v │[revise_meaning_cohesion] │ | │ v │[review_titles] │ | │ v │[revise_titles] │ | │ v │[review_chapter_title] │ | │ v │[revise_chapter_title] │ | │ v │[review_overall] ──────────────┘ | v [promote_content] | | +--> [book_narration] (from adapted text) | +--> [upload / publish] QA rounds are controlled by the output of the review_overall stage. Examples of frontend enabling fast QA Example of problematic images that can be ’easily spotted’ by a trained eye. Chapter level issues overview for an adaptation workflow: Part level debugging/review of what was suggested/applied in the QA pipeline: Review and Revision—kept deliberately apart One key design choice was to decouple “reviewer” from “reviser”. The reviewer node reads the necessary context and produces a structured list of issues,while the reviser node sees only the affected slice plus those suggestions. Why keep them separate? Audit clarity The reviewer’s JSON lives as its own artefact, so you can diff, grep, or hand-editthe feedback without touching the text itself. It’s also easier to audit automatic revisions and spot ‘additional helpfulness’. Smaller prompts, cheaper calls A reviser that operates on just the target part + suggestions uses far fewer tokensthan one that re-ingests the whole chapter. Less collateral damage Narrow context means the reviser can’t “helpfully” rewrite good paragraphs in other sections or even just completely forget them. True parallelism Parts are context-isolated, so multiple revisions can run concurrently—no giant chapter-wide lock. Targeted rollbacks If a revision introduces a new issue, it’s easy to rollback. What the flow looks like review_meaning_cohesion scans consistency_revised_stories and writes meaning_cohesion_review_1.json: { "issues": [ {{ "part_number": 1, "suggestion": "Change sentence '...sentence...' to '...corrected sentence...' to ensure consistency with previous parts.", "reason": "Inconsistent character name across parts.", "severity": "high" }}, ... ], } During the revise_meaning_cohesion stage, revisions to specific parts are applied concurrently, e.g. reviser for part 8 only sees the text of part 8 and the suggestions for part 8. Other novel elements Stable DAG via no-op nodes – skipping or caching never changes filenames or dependencies, so UI and controller logic stay simple. Declarative STAGE_DEPENDENCIES – each endpoint validates its own inputs and fails fast if artefacts are missing. Pluggable adapter subclasses – swapping models or prompt strategies is a config change, not a refactor. Direct-disk reading Server Side React Components – Suprisingly trivial frontend code. SQLite error forensics – a single query surfaces “prohibited-content” or other LLM failures. Hot-reload mid-run – tweak prompts or error handling while a 60-chapter fan-out is running; retries pick up the change without restart. WebP illustration storage – generated art compresses very well; files are roughly 10× smaller than raw outputs. LLM-driven controller decisions – the CLI uses the review_overall output to decide whether to launch another revision round and which stages to skip. Personal wins The biggest win is defending my personal sanity. I no longer have to babysit a set of fragile notebook based pipelines based on fallible and untrustworthy LLMs. Higher confidence in quality – every chapter passes level, consistency, meaning, and overall reviews—automatic multi-round fixes if needed. Far less toil – no more babysitting fragile notebooks; the pipeline self-checks and fails early. Fast, parallel experimentation – new adapters or prompts run side-by-side with hot-reload; iteration is “super fast and fun.” Cost flexibility – total spend is higher (as there are more LLM calls), but the modular design lets me fall back to cheaper or local models whenever I choose. And the cool thing is that building this tooling was heavily accelerated by a coding assistant/agent, so it was significantly faster and more fun than I would have expected from the scope of the reengineering.