More from Miguel Carranza
2024 has come and gone, and it’s time for my annual post. What a year for startups—like squeezing five regular years into one. Do you remember the Apple Vision Pro, the DMA regulation, founder mode, or the o1 launch? All of that happened in just the last twelve months. It’s also been wild at RevenueCat. My journal is full of stories that could fill a whole book or even a few HBO Silicon Valley seasons. Some are inspiring, some are hilarious, and others are honestly gnarly. Due to limited space, the need for context, and respecting everyone’s privacy, I’ll cover only the most interesting topics at a high level. Looking back, it was a good year for RevenueCat. Actually, a great one. Perhaps our best since 2020. We have plenty to celebrate: We accelerated again, and we hit our C10 revenue plan. We made our first acquisition and welcomed an amazing founder to the team. We signed our first multi-million-dollar contracts. We went to more than 20 events around the world… …including hosting our own conference, featuring its own award ceremony. We became the #1 payments SDK on iOS. Our swag went to 11. We launched over 80 user-facing features. We showed up in Times Square and along Highway 101. We raised a mini Series C and welcomed two new board members. We landed in Japan for the first time. Our API grew beyond 2B requests per day. We are processing nearly twice the TAM we had when we started the company. OpenAI is a friend of the cat. We continued building a winning team. We hired people who were on my to work with bucket list even before we started the company. By all the metrics, 2024 was our year of shipping and selling. We absolutely helped developers make more money. My role If you compare our progress to the goals I wrote about in last year’s blog post, it’s clear we succeeded. But the journey itself was a lot rockier than I had imagined. Many things didn’t go as planned, somes hires did not work out, and some strategy changes were really hard to push through. At the start of the year, besides my usual responsibilities, I also set four personal goals to help me scale with the company. I wanted to: Ship code more consistently. Talk to at least one customer every single day. Get more involved in areas outside of engineering. Stay very close to my co-founder Jacob, giving him my full support. I completely missed goal #1. Honestly, that hurts because I love building software. But I’m not too upset about it—our customers care that the team ships, not that I personally do. And in a way, I did help the team ship. As for the other three goals, I hit them, at least based on the company’s results. Still, my self-perception wasn’t always great. I found myself acting much more like a co-founder/executive than a CTO. Some weeks were brutal, with constant context switching across tasks and teams I didn’t always enjoy. At one point, there were over 60 people in my org, spread all around the world, which is pretty intense for an introverted computer kid. A lot of bullshit escalates to the top, making me question my entire existence some days. And then life threw serious personal emergencies at some of our team members too. Like I said last year, life is what happens when you’re busy building your startup. As we got closer to 100 people, it was statistically unavoidable that we’d face a few life-changing traumas—sometimes several all at once. As a founder, you need to be supportive and empathetic, but also protect your mental health. As a human, it’s tough. Add in a couple of two-year-olds who were often sick and not sleeping, and it felt like a ticking bomb. Did I burn out for the first time in my life? I don’t think so, but it got close. I’m confident being a founder (and having a co-founder) kept me going. I care too much and must stay resilient. If I’d been just an employee, I might have tapped out. But enough about the tough parts. Aside from being a professional BS handler, here’s how I spent most of my time: Lots of travel This was the year I traveled the most in my life—and looking back, I probably should have traveled even more. I visited customers, helped with sales, conducted executive interviews, and spoke at a few conferences. It’s hard being away from two little kids, but each trip turned out to be worth it. Support One of my biggest worries this year was our support function. Everything was fine, but our Support Engineering Manager was going on parental leave, and the bus factor was scary. I’ve seen support crises before—they’re not fun. It wouldn’t have killed the company, but at our size, it might have forced us to pull senior engineers into support and slow down our product velocity. Time was short, so we tried a few things that worked: We hired two new Developer Support Engineers who already knew our product—former customers! Their onboarding was smooth, and they hit the ground running. We split the support team into two pods with their own leads. Each pod handles certain tickets, and they collaborate with each other instead of relying too heavily on one person. We finally set up our first on-call rotation for emergencies. Sales and post-sales Reviewing sales and implementation calls, giving technical input, collecting enterprise customer feedback for product and engineering, joining calls, and even doing some in-person visits. Writing more As our team continues to grow across the globe, not everyone has the same direct interaction with me or Jacob as before. A lot of our culture and collaboration style was once passed through observation, but now needs to be written down to reach everyone faster. These days, my code editor is basically replaced by Google Docs. I’ve been publishing more internal and external documents—like our Engineering Strategy and an updated How to Work with Miguel. Product delivery We still have a few details to refine, but our founder Shipping and Timeline reviews have been valuable. It gives Jacob and me a high-level view of everything in progress, lets us offer feedback, and helps us dig deeper where needed. It’s a great way to see each team’s capacity, find bottlenecks, keep a sense of urgency, and deflate anything that’s not truly important for the customers. Re-orgs This year brought a couple of big reorgs in product and engineering. Overall, they went well, but the puzzle gets more complex with each new piece. We created sub-teams to narrow their focus and keep things running smoothly. For the first time, we had a couple of management layers between me and our ICs. One major change was shutting down our Enterprise/Reactive team. The idea was solid at first, and they delivered plenty of value, but eventually they became the random tasks team. Our new plan is to reinforce the rest of the teams: quick enterprise requests go to the right team to handle them reactively, while bigger projects move to the main roadmap (which keeps us disciplined). The engineers from that old team will help bootstrap new teams as we hire in 2025. Hiring Our engineering hiring goals weren’t super ambitious, but we still reached them. The pace was a bit uneven, and some roles took longer to fill than desired. However, when Hiring Managers took ownership of the process (with recruiting as a support) it made a huge difference in the quality of our candidates. We also brought back the founder interview stage: Jacob or I spoke with every single candidate before making an offer, and we plan to keep doing this for the foreseeable future. Random founder stuff Not my main focus, but I still spent a fair amount of time handling people-related topics, operations, investor, customers and partners relations, fundraising, company policies, planning, etc. Learnings: deepened insights On culture Shipping is king. Yes, deadlines are stressful, but failing to ship and getting stuck in endless debates is far worse. It’s depressing. If someone isn’t a good culture fit, it will be more than a single incident. Over time, it becomes pretty clear to everyone. Top performers tend to measure themselves against the very best in the company. Reassure them they are doing a great job. On the other hand, those who underperform will look to other underperformers to gauge their own progress. Even top performers will struggle if they don’t fully align with the vision. Nothing beats talking to customers. Encourage everyone to do it. Better in person. Most managers don’t have a strong incentive to be strict, so finding the right balance takes a lot of calibration. Only once everyone is aligned, you can truly delegate. On hiring Managers hate hiring because it’s binary: a lot of “no”, but eventually one “yes” can change everything. Staying consistent really helps. Simple things, such as weekly updates keep everyone accountable. People management sucks. If someone’s main motivation is to be a manager for the title, or so they can “lead,” that’s a red flag for me. Best managers end up being those who never planned on becoming one in the first place. They actually roll up their sleeves and can do the work. Big ideas are great, but they need to be executed. This matters even more when it comes to executives. A truly great exec can change your life and it will feel like a brand new company. Given their influence, anything less than great will eventually turn into a big mess. Spend time with executive candidates in person. Watch how they work and make sure they’re real builders. Previous founders and true engineers are usually a little bit de-risked, but they’re still not guaranteed. I also like to write a very detailed onboarding doc, clarifying context, expectations, and what success or failure looks like. Having them write a 30/60/90-day plan helps us all align. I’ve learned not to rely on their past pedigree. The real key is their glass eating endurance. On company building Re-orgs are inevitable at a growing startup, and they will feel scary or emotional for people who aren’t used to them. I find it helpful to be super clear about why we’re doing it, and to share the fallback plan if things don’t work out. But I’ve learned that by the time you think you need a re-org, you’re already behind. We now plan to reassess our team structure twice a year for optimal shipping. A numeric goal (like an SLA or revenue target) is just a proxy. Missing it isn’t the end of the world if you learn in the process. But if you surpass it without recognizing what’s broken underneath, you’ll be masking critical issues. Perfectionists are great, but can have a tough time at startups. Some do fine if they’re focused on one specific thing or working as an IC. But as soon as they have to juggle multiple tasks, the chaos can feel overwhelming. Help them embrace it. Things will break. It’s about continuously reprioritizing, and stopping small cracks from becoming big fires. No agenda == no meeting. Synchronous time is expensive. The only exception is the occasional unscheduled call. After you reach around 50 people (especially in a remote setup), documenting every process change becomes crucial. I’ve learned the hard way that simply talking about a new process isn’t enough. Founders can’t talk to everyone all the time anymore, and misunderstandings or gossip can spread fast. Not everyone will read everything, but at least there’s a single source of truth to reference. Best people can stretch quite a bit—they’ll often rise to the challenge. But it’s wise to keep an eye on their limits before they burn out or become a bottleneck. On scaling as a founder A great EA is life-changing. Family will be supportive, but it’s not fair to offload all the stress on them. They will end up feeling helpless. Having a solid co-founder, a network of peers, or a good executive coach makes a world of difference. Founders are the ultimate guardians of the culture. It’s constant work, and it will feel relentless—especially when things are going well and it’s easy to get entitled. The best team members will help uphold the standard, but you cannot expect them to do all the policing. At this stage, it’s stupid not to level up your lifestyle in ways that reduce stress or save time. This might include childcare support, having a second car, investing in a better mattress, or hiring help with housekeeping. The startup is bigger than its founders, and the goal is to continuously remove yourself from the critical path. Still, it’s easy to forget that, as a founder, you literally brought everything into existence from thin air. Imposter syndrome often creeps in when you step outside your core expertise, but if your gut feeling is strong, it’s worth paying attention to. Stay open-minded, yet remember that no one knows the company quite like you do. The future Next year is going to be another big one. We’ll keep shipping and selling, while finally tackling our design and UX debt. We’ll keep investing heavily in our infrastructure. Not just for reliability, but also for real-time data. We want RevenueCat to feel fast, accurate, and easy to use. We’re also upgrading our self-serve and enterprise support, aiming for a truly world-class experience. In many ways, we’re finally seeing the original vision Jacob and I had back in 2017 come to life. We’ll be launching new product lines too, and if we execute well, we will be just a couple of years away from hitting $100M in revenue. We’ll keep building a winning team. We’ll hire about 45 people, 30 in Engineering, Product, and Design. It’s a challenge, but totally doable. As for me, my personal goals haven’t changed much, but my perspective has. Jacob and I used to joke that being happy and winning can’t happen at the same time. But why not? We’re in a privileged position to shape our own path and change anything we don’t like. After a lot of reflection and coaching, I realized I was simply too hard on myself. I was feeling depressed by the constant BS even though we were winning. A hack that helped was working with my EA to set weekly goals and then sending a public update to the full team. It let me see the real progress behind all the drama and back-to-back meetings and stay transparent with everyone. Next year, I’ll avoid meetings before 9 AM, keep an eye on calendar creep, and hold myself accountable to exercise and doing what helps me decompress. I know, it’s obvious. I also plan to travel more. Especially to the Bay Area, which is clearly back again. Meeting up with other founders and customers is always worthwhile. Another thing that made last year tough was having half my direct reports on parental leave for about half of the year. Now they’re back, and I can feel the momentum returning. Jacob has also taken over product again, now that he’s stepped away from directly owning operations and people. Increased shipping velocity has been noticeable. We have all the pieces in place. All that’s left is to keep pushing forward: shipping, selling and enjoying the ride. If there’s one thing I learned in Silicon Valley, it’s that no goal is too crazy if you refuse to give up. I really hope you enjoyed reading this post. As always, my intention was to share it with complete honesty and transparency, avoiding the hype that often surrounds startups. If you are facing similar challenges and want to connect and share experiences, please do not hesitate to reach out on Twitter or shoot me an email! Special thanks to my co-founder Jacob, my EA Susannah, the whole RevenueCat team, and our valuable customers. I also need to express my eternal gratitude to all the CTOs and leaders who have been kind enough to share their experiences over these years. Shoutout to Dani Lopez, Peter Silberman, Alex Plugaru, Kwindla Hultman Kramer, João Batalha, Karri Saarinen, Miguel Martinez Triviño, Javi Santana, Matias Woloski, Tobias Balling, Jason Warner, and Will Larson. Our investors and early believers Jason Lemkin, Anu Hariharan, Mark Fiorentino, Mark Goldberg, Andrew Maguire, Gustaf Alströmer, Sofia Dolfie, and Nico Wittenborn. I want to convey my deep gratitude to my amazing wife, Marina, who has been my unwavering source of inspiration and support from the very beginning, and for blessing us with our two precious daughters. I cannot close this post without thanking my mom, who made countless sacrifices to mold me into the person I am today. I promise you will look down on us with pride the day we ring the bell in New York. I love you dearly.
I’m back in Spain for my brother’s wedding. I rarely visit during the summer. The heat in my hometown is brutal, around 40 degrees Celsius (over 100 Fahrenheit for my imperial friends). Most people escape to the coast, just like my family did when I was a kid. I haven’t been here in years. As I drive along the coast, I find myself reflecting on a tweet about money and happiness, a vivid memory pulls me back in time. It’s August 22nd, 2007. The iPhone, the first real smartphone, has just been announced. It’s so cool, but of course I cannot afford it. It’s not even going to be released in Spain. I’ve just gotten my driver’s license, and I’m about to dive into my third year of Computer Science. I set up my clunky TomTom navigator knockoff, and hit the road. I’m on my way to meet Marina for our first real date. She’s cool, pretty, and kind. She likes the same music as I do, and even has distant relatives in California, the place we jokingly plan to visit someday (if I ever get enough cash). I’m listening to a pirated Blink-182’s self-titled CD. Pop-punk is pretty niche in the south of Spain, and it’s dying. Blink-182 has split up, and I missed my window to see my favorite band live. The Atlantic Ocean is as flat as a lake. This corner of Huelva’s coast is sheltered from any real waves, a stark contrast to the world-class surf breaks I drool over in magazines. And suddenly, reality hits: my childhood dreams of building a tech company in Silicon Valley, while vacationing in Southern California feel impossibly far. Even getting through my degree feels like a pipe dream. School isn’t fun anymore. It’s grueling, especially the parts I thought I’d enjoy, like algorithms and Data Structures. I might never become a Software Engineer. I feel stuck, trapped by my lack of direction. I am seriously considering quitting. But no degree means no job in the US, and good tech gigs are rare here in Spain. The only cool company is Tuenti, a new startup that is cloning Facebook. I’m nowhere near smart enough to land a job there though. Flash forward to today, 17 years later. It’s almost laughable to think about how hopeless things once seemed. Even now, it doesn’t feel like I’ve “made it.” The path and the results look nothing like what teenage me envisioned, but somehow, I’m realizing I’ve kind of checked off every box. I married Marina, and we live in Southern California with our two beautiful identical kids. We’ve become American citizens, and I’ve lived in the Golden State for nearly a third of my life. I’ve worked as a Software Engineer at a Silicon Valley startup, learned from the best, and found the best co-founder I could ask for. We launched our own company. Smartphones? They’re in everyone’s pockets now. Our product is in a third of all new apps shipped in the US. We’ve helped developers reach millionaire status, and we’ve made more money than I ever thought possible. But I’ve learned that a lot of money is a relative term. Somehow, I managed to hire insanely talented engineers—a bunch of them, ironically, from Tuenti. Blink-182 is back together, and I’ve been fortunate enough to see them live five times. I’ve even bumped into Tom Delonge after surfing world-class waves a few times. I’m literally just realizing how surreal all of this is. I tend to get caught up in the chaos of what’s next—the next big fire, the next goal—but sometimes you’ve got to stop, be present, and reflect on how far you’ve come. It wasn’t easy. It wasn’t without loss, sacrifice, and a fair share of doubts. Am I truly happy? Maybe not in a perfect, all-the-time kind of way. There are external things humans cannot control. But when I look at my life, I realize there’s no real reason not to be. The journey has been was worth it so far: the ups, the downs, the unexpected turns. So, here’s to your journey, whatever it looks like. Keep going, keep dreaming. It might not turn out the way you envisioned it, but it’s only impossible if you quit.
I am drafting this post at 35,000 feet flying back from Japan. I’ve entered the US about 30 times, but this will be the first time I’ll be using my shiny blue passport. No anxiety about aggressive questions, secondary inspection, or the possibility of deportation. A couple of days ago, my wife had her naturalization ceremony, and with her, our whole family is now American. This post is a reflecting on our 11-year immigration journey. My American Dream My story with technology started at 8, with my first computer. I fell in love and decided that one day I would start a computer business. And of course, it would have to be in Silicon Valley, the epicenter of innovation. I grew up influenced by the iconic Californian lifestyle of the 90s, from Tony Hawk to bands like Blink-182 and The Offspring, which only fueled my desire to call the West Coast home. As I finished my Computer Science studies, the reality of achieving my American dream seemed increasingly distant. My enthusiasm for the Californian way of life hadn’t waned — I had started surfing and I was even playing in a pop-punk band. But the immigration complexities were too daunting. It looked impossible. I decided to study a Master’s Degree in the UK and reinforce my English. Right before completing my degree in England, I was offered a six-month internship at a startup in San Francisco. Financially, it was not the smartest decision — I would barely be able to afford rent, despite having better paid options in Europe. However, experiencing Silicon Valley was a lifelong dream. It wasn’t an obvious choice. But I ultimately packed my suitcase, left behind my family, girlfriend and friends, and relocated across the world. Landing in California My journey in the U.S. began with a J1 visa, intended for interns and relatively simple to secure with an employer’s backing. It was a suitable fit for my six-month plan, extendable to a full year, without any ambition for a longer stay. Yet, I worked extremely hard, and the situation changed when I was introduced to the possibility of obtaining an H-1B visa. Unlike the J1, the H-1B visa demands wage parity with U.S. citizens, allows for a stay of up to six years, and paves the way for permanent residency. Most importantly, it meant I could start laying down roots in the US, such as building my credit score, buying a car, or negotiating a long-term lease. First Immigration Problems My living situation dramatically improved when I traded my small, rat-filled room for a two-bedroom apartment in the Outer Sunset, sharing the space with a friend. I got a sizeable salary bump. However, the happiness was short-lived. H-1B visa applications had exceeded available spots for the first time in years, introducing a lottery. My coding skills, education, and value to my employer wouldn’t factor into this gamble. Anxiety mounted for a very long month, as friends celebrated their visa wins. I was left in the dark, bracing for bad news. Against the odds, relief came just a day after a disheartening talk with my immigration lawyer, granting me my first taste of luck. Reuniting with my girlfriend Sick. My H-1B visa approval meant I could make the US my home for six additional years, longer than Marina and I had been dating. As she was about to end her studies, we strategized on ways to reunite in the US. Marrying earlier was an option, but with H-1B restrictions preventing spouses from working, we looked for alternatives. The easiest path forward involved securing a student visa, leading to an 12-month work permit, followed by an H-1B visa application. In the competitive climate of 2015, her work visa acceptance felt nothing short of miraculous, becoming our story’s second lucky strike. Permanent Residency While six years might seem long, they pass quickly when you are busy and having fun. Halfway through, it became clear we needed to strategize for the future when it was time to renew my visa. My employer agreed to initiate the Green Card application, a lengthy and costly process requiring proof that I was indispensable for the company. Despite the complexities, our attorney believed the case would progress smoothly, estimating an 18-month completion time. Surprisingly, the initial phase went way faster than anticipated, prompting our attorney to suggest an immediate wedding for Marina and me, a necessary step to include her in the Green Card application. We quickly scheduled our wedding at the Spanish Consulate in San Francisco, departing from our original plan for a ceremony in Spain. More problems A year and a half in, expecting our Green Cards, we faced an unexpected challenge: our marriage, officiated at a consulate, was not recognized by US Immigration, compelling us to marry again, pay the associated fees again, and start the process over. This development was extremely frustrating, specially as my friend Jacob and I were contemplating founding a company, and the absence of a Green Card meant remaining as an employee. To rectify the situation, we promptly got remarried at San Mateo City Hall, choosing it for its rapid scheduling. Two months later we held another ceremony in Spain with our family and friends (our third marriage overall). The delay in our reapplication, exacerbated by the recent election of Trump and a subsequent slowdown in immigration services, led us into a stressful period of uncertainty. Our inability to make definite plans, from household purchases to housing arrangements left us anxiously awaiting any news on our application. As we navigated this uncertainty, the possibility of dedicating myself fully to our startup (later named RevenueCat) became increasingly dim. Going All In A pivotal moment came when our now startup, RevenueCat, was accepted into Y Combinator, requiring my full-time commitment. A big challenge due to my pending Green Card application. My immigration status became the biggest risk to our startup, even before launching. In searching for a solution, we identified a potential hack: an immigration loophole that allowed for employment changes under specific circumstances. It wasn’t risk free. There were no guarantees, and it involved giving up my H-1B status and the ability to travel. Should my Green Card application face rejection for any reason, I would instantly become an illegal immigrant. Deciding to take the gamble, we prepared the necessary documentation. To support our case, Jacob, my co-founder, had to write a letter stating that although my compensation was on the lower side, his, in the CEO role, was even lower. We also had to declare our company’s annual earnings ($0 at that time), and I suggested Jacob to specify it as less than $1 million. You should never lie to Homeland Security 😅. This leap of faith paid off; eight months later, we received our Green Card interview, where the immigration officer, making fun of our unique circumstances, granted approval on the spot. The Decision to Become American Those with an employment-based Green Card need a five-year stay in the US, compliant with all laws and tax requirements, to qualify for citizenship. While obtaining citizenship is not mandatory, and one might choose to stay a permanent resident indefinitely, citizenship confers full rights and responsibilities. Among these, the requirement to pay federal taxes forever, a big concern for many. If RevenueCat succeeds as we hope, I’m looking at significant tax payments, even if we end up moving back to Europe. The decision to embrace US citizenship came down to a simple reason: we can. We recognize the privilege of this choice, acknowledging the series of fortunate events that brought us here, aware that our journey could have taken decades had we originated from countries like India or China. Our twin daughters are blessed with dual citizenship, offering them a breadth of choices for their future. The opportunities the US has presented to our family are beyond what we once could dream. By becoming citizens, we gain a voice to influence immigration policies positively instead of blocking progress. My tax contributions have already reached the seven-figure mark. The continuation of our tax obligation is a small price to pay. And after all, there are some advantageous double taxation agreements 😉. Special thanks to my family for always supporting me and pushing me to live my dream in San Francisco. To Marina, my now wife, for joining me in this crazy adventure across the globe. To everyone at StepOne for running an amazing internship program. To Jesse, for believing in my potential and tackling the immigration challenges with me. And to my co-founder Jacob, for pushing me to take a leap of faith with my immigration status.
Since reading ‘High Growth Handbook’ by Elad Gil, the value of writing a ‘Working with’ document became crystal clear to me. I am sharing mine externally to inspire other founders and leaders to reflect and write down their own working styles. These documents are incredibly beneficial, especially in a multi-timezone, remote setting like we have at RevenueCat. I’ve spent some time fine-tuning mine, and this is the updated version. Welcome to your go-to manual for understanding how to collaborate effectively with me. My Mindset: Logic-Driven, Plan-Oriented I’m a logical thinker, much like a computer. If A implies B and we have A, I’ll typically conclude B. Sticking to plans and predictability is my comfort zone, yet I value reactivity, especially when customer-related issues arise and are solvable. This company isn’t just a job for me; it’s my life’s work. I’m deeply invested in everything here — our technology, culture, team, and customers. I get inspired and energized by hard-working coworkers who believe in our mission even more than me. As a co-founder, I can offer a wealth of institutional knowledge and guidance. While I may not have all the answers, I’m usually good at pointing you in the right direction. RevenueCat is only a sum of it’s parts. Our teammates drive our culture and I want to make sure we are building a place that people want to be. If you have a suggestion on how to make RevenueCat an even cooler place to work for our teammates I’m always here to talk about it. How We’ll Operate Regular Check-ins: For my direct reports, expect weekly or bi-weekly one-on-one meetings. To make our discussions more focused, I prefer that we establish an agenda before our scheduled time together. Communication Protocols: My schedule doesn’t allow much room for impromptu calls. If something urgent pops up, message me on Slack first. Should it require a call, schedule it through Susannah, please never bypass her. Meeting Preparation: Come to meetings with an agenda to ensure productivity. Without one, I might dominate the conversation, potentially missing your crucial points. Let’s both be responsible for following up on action items. Team Support: I’m open to joining other team meetings, but please share the agenda in advance and mark my attendance as optional unless crucial. Problem-Solving Approach: My engineering background means I love tackling complex problems using a divide and conquer approach: by breaking them down into smaller, manageable chunks, solving each piece, and then combining them for a final solution. If we can improve a completely broken system to 90% functionality, that’s significant progress in my book! Communication Style Note-Taking: While I take meticulous notes, my current preferred tool doesn’t support sharing. If you wish to access these notes, it’s on you to set up a shared document in Google Docs, Notion, or Lattice. Information Filtering: I prefer having complete transparency and the ability to filter out unnecessary details myself. Always explicitly state if you need input from me, or else I’ll assume it’s for my information only. Feedback Style: Expect direct feedback from me. I’ll clearly differentiate between areas for improvement and significant performance concerns. Trust Dynamics: Consider my trust like a metaphorical ‘bucket’ that starts half-full for everyone and adjusts based on your actions. The more you fill this bucket, the more autonomy you’ll have. For Managers Transparency in Challenges: Startups are always broken one way or another. I prefer to hear any bad news about a project or a team member directly from you. Working together through challenges can strengthen our trust and working relationship. Progress and Concerns: During our 1:1’s, I’ll inquire about your team dynamics and direct reports’ progress. I encourage you to include any details in our 1:1 agenda and lead the conversation to address any performance concerns, project delays or notable achievements . Feedback Dynamics: I recognize the weight of my title. To avoid unnecessary tension, I prefer to provide critical feedback about your reports directly to you so you can address privately. On the other hand, if there is any commendable achievement by your team I will do my best to praise publicly. If you feel there is someone on your team that I should connect with or praise, please let me know. Encouraging our team and recognizing their strengths is something that is very important to me. Preferences and Pet Peeves What I like Doing your homework: No question is stupid, but always do your initial research before distracting the team. Being resolutive: Getting things done, unblocking yourself. Readable and consistent code. Proven, boring technology over unproven open source projects that is trending on Hacker News. Proactivity: See a problem? Fix it right away before anyone notices. Made a mistake? Build systems to prevent anyone else making the same one again. Double checking your work: Give your work (documents, presentations, pull requests) a quick self-review before presenting it to the team. Transparency: In a multi-tz, remote environment, over-communication is better than miscommunication. Healthy discussions. When there is a decision to make that is not clear, it’s because all the different approaches have pros and cons. Together we will be able to calibrate and choose the lesser evil. A short call (or loom) is preferred over constant Slack interruptions. What I don’t like Gossip and rumors: They destroy the culture. Be upfront. Cargo cult: Let’s not do something just because BIG CO does it. That’s the beauty of building something from scratch. Unnecessary blockers. You’re all pretty smart here! Always try to unblock yourself first. Lack of context in questions, emails, and discussions. Not speaking up when something isn’t clear. Recurrent mistakes or questions: One time, it’s totally expected. Two times, hmm. Three times, nah. Learn, document, and build systems. Complaining without taking any action to improve the situation. Sarcasm or other ways of communication violence during disagreements: When somebody wins an argument, most of the time, the whole team loses. Acknowledging My Flaws Overcommitment: I tend to take on more than I should, which inevitably affects my focus. While I’m working on this, please understand if I occasionally get sidetracked by emergencies. Communication while Debugging: When addressing issues, I might share unvalidated hypotheses, which can be confusing. I’m learning to communicate more clearly and only after verifying my thoughts. Problem-Solving Obsession: Unsolved problems keep me up at night, which isn’t ideal for my well-being. It’s a habit I’m aware of and trying to balance. Pessimistic Tendencies: In evaluating problems, I often veer towards catastrophic thinking rather than optimism, a trait I’m mindful of and trying to moderate. Office Hours To maximize my availability given my tight schedule, I’ve introduced ‘office hours.’ This time is open for anyone to schedule a 15-minute chat with me about any concerns or ideas you might have. Reach out to Susannah for scheduling details. Thanks for sticking with me till the end! These are my personal preferences, not commandments carved in stone. I’m stoked to collaborate, build awesome stuff, and, above all, have fun together!
Another year as a founder CTO, and let me tell you, it’s been one for the books. I can’t remember a time in my life that was more demanding and emotionally draining. Those early years were filled with hard work, but we were also full of energy, ambition to build, and the sense that we had absolutely nothing to lose. As my dear friend Jacob used to quip, “Worst case scenario nobody dies”. Yet, everything takes on a new perspective when you’re responsible for the monetization infrastructure of over 30,000 apps and ensuring the livelihoods of not only your team but also your own family. While I would typically begin this series of posts with an array of metrics, this year has been a whirlwind of events that has taken precedence over mere numbers. So, dear friend, take a seat and allow me to share with you a deeply personal and epic firefighting tale and the battle to restore the flame of RevenueCat’s core values. The Year of Shipping We embarked on Q1 with big intentions and an ambitious roadmap. With seven new engineering teams, my role was to lead the one responsible for serving our larger customers. My days consisted of collaborating with our (still tiny) sales team, while simultaneously overseeing our support function. While I continued to provide technical guidance to several projects, I made the conscious decision to delegate the bulk of product development to the other teams. In hindsight, this decision proved to be a mistake. The teams were still finding their footing, with a mix of new hires, and there was a disconnect between engineering and product that needed addressing — a topic we’ll get into later. Nevertheless, the team I led was firing on all cylinders, delivering enterprise-level features at a hasty pace. From implementing single-sign-on in a matter of days, to tackling gnarly bugs, this team’s momentum was inspiring. I hoped that this stride would serve as a catalyst, motivating the rest of the teams to achieve a similar velocity. On the home front, my wife’s parental leave came to an end in January, marking the start of a new era where my hours were no longer flexible. Together, we embarked on the demanding task of caring for our twin babies. Little did I know how much the lack of sleep would impact me. Fatigue began to set in and my usual escape valves, such as surfing, turned into distant memories. Caffeine became my closest friend, helping me stay awake as I navigated the demands of work and family. I would spend my entire weekends lying in bed, trying to recover. I wrongly assumed it would be a short phase. Troubles never come alone On the fateful morning of March 9th, while I was soothing one of our crying babies at 4:30 AM, my WhatsApp began buzzing with messages from concerned fellow founders. Rumors were swirling that Silicon Valley Bank, the custodian of most VC-backed startup funds, was about to collapse. Within hours these whispers escalated into a full-blown bank run and we found ourselves unable to access our funds. Without going into the nitty-gritty details, it was an intensely stressful weekend. We were on the verge of a payroll crisis and we didn’t even have a functioning bank to transact with. However, we couldn’t afford to let this situation be what killed RevenueCat. We worked to expedite the opening of new bank accounts and explored multiple liquidity alternatives. Fortunately, we were privileged enough to entertain multiple options. One investor came to help, wiring funds from their personal account, and some of our loyal customers offered to pay in advance. Just as hope seemed to dwindle, the FDIC announced that they would guarantee all deposits on that Sunday afternoon. Crisis averted. Emotionally drained from the weekend’s trauma, we received another unexpected blow just a few days later: Y Combinator was discontinuing its Continuity Fund. This created a dilemma regarding the fate of their board seat, something we would have never anticipated. Just weeks prior, we had suffered a severe outage caused by our cloud provider. We tried to console ourselves, thinking, “At least all these problems are internal; we’re not dealing with downtime”. A Crisis of Reliability Our break was short-lived. Less than a week after the Silicon Valley Bank fiasco we experienced yet another major outage, and this time, it was self-inflicted. When you’re handling hundreds of thousands of API requests per second, even a few seconds of downtime can set off a cascade of alarms and potential sales losses. Our customers were understandably frustrated, and they didn’t hesitate to make their displeasure known through every available channel. It was heartbreaking to witness and I felt a deep sense of personal failure. This marked the third user-facing issue in a matter of months. It wasn’t something to sweep under the rug: our reputation was on the line and it was time to act. By this point, I was utterly sleep-deprived and my Apple Watch had begun sending me concerning health notifications. The day after we resolved the outage, my co-founder Jacob and I came up with an action plan. It was officially wartime. We needed to act decisively. We publicly unveiled our plan: to develop a robust fallback system independent of our current infrastructure. And we promised to deliver it within a week. Simultaneously, Jacob and I embarked on an apology tour, reaching out personally to some of our most worried customers. This was a painful but humbling experience for us, but it proved beneficial on multiple fronts. It allowed us to reiterate our commitment to becoming the best in-app subscription infrastructure provider in the world while gaining invaluable insights into our customers’ pain points. RC Fortress I rallied a small crew of engineers from different teams to build the first version of what we called “RevenueCat Fortress”. This component was designed to make sure end-customers could purchase seamlessly, even when our main servers were unavailable. It was a crazy week because we set ourselves a tight deadline but it helped boost our spirits and proved we could deliver software fast. The initial version of RevenueCat Fortress was quite simple – it operated behind the scenes on the server. But we didn’t stop there. We made it even better in the next iterations by adding SDK improvements such as offline entitlements. When it finally rolled out, it did so with flying colors. We even got to put it to the test during a major Apple outage and it saved the day for RevenueCat customers, making them immune to Apple’s downtime. Turning things around Looking back, the birth of RC Fortress marked the start of a shift in our culture. It got us back to the basics of reliability, fast delivery, and customer obsession. We couldn’t afford to spend months on extensive, untested projects. We had to rapidly build the features our customers valued most and iterate from there. We also realized that keeping things rock-solid wasn’t just the infrastructure team’s job; it was a global effort. Around the same time, we faced a couple of setbacks when we parted ways with two executives – the VP of People and the VP of Engineering. We tried to find a new VP of Engineering but couldn’t find a match that really excited us. So, the board agreed it would be best if I took the reins of the entire engineering organization again. Those days gave me a chance to get closer to the product teams again. Here’s where I spent most of my energy: Performance: I clarified expectations, provided feedback, and coached managers on performance management. Hiring: We tweaked our hiring process and re-calibrated interviewers’ expectations. Reliability and Quality: We were pretty good at doing post-mortems after issues but we had too many of them. They lacked detail and they weren’t followed through with action items. We needed a little bit of a cultural reset. We introduced dedicated incident Slack channels and clearly defined roles. Customer Obsession: Taking over the support team was eye-opening. It gave me a direct line to our product’s weak spots and what confused our customers. We started categorizing support tickets and sending them straight to the right product teams for triaging. Project Management: We focused on breaking projects into smaller chunks to deliver faster, instead of getting lost in never-ending projects. Education and Best Practices: I spent time educating other departments, especially post-sales teams, to avoid recurring mistakes that were slowing down our engineering progress. On top of all that, I reconnected with our customers more than ever. I hopped on planes to visit them at their offices and even worked our booth at a few conferences. It was a refreshing change to chat with users face-to-face, and hearing their unique challenges in person after a long time. Life is what happens when you’re busy running your startup We were fortunate enough to fly in my mother-in-law to assist with our babies, which made my travel plans possible. I finally felt more rested and even managed to squeeze in a few surfing sessions. Things were looking up both personally and professionally. I was eagerly anticipating our annual company-wide offsite, especially since I had missed most of the previous one due to my wife’s high-risk pregnancy. This time around I was geared up to address the entire engineering team, sharing the exciting changes and boosting morale. We were about to start winning again. But then, the day before my flight, I received a call from my father back in Spain. My mother had been rushed to the hospital, and she had been diagnosed with an extremely aggressive form of leukemia. Time seemed to stand still. I boarded the flight as planned, chugged two Red Bulls, and delivered that motivational talk to the whole team, while my mom was in a hospital bed thousands of miles away. This was hands down the toughest thing I’ve ever done as a startup founder. I left the offsite early and headed back to my hometown. Over the following weeks, I traveled back and forth around the globe, coordinating with my family. Sadly, my mother never left the hospital, she passed away merely a few weeks after the diagnosis. These are the things that always lurk in the back of your mind when you’re living 10,000 miles away from home, but you never truly believe they’ll happen. Until they do, and they shatter you. Keep on pushing The days that followed were far from easy. We were dealing with the launch of our biggest customer’s app, something we’d been preparing for a long time. The scale was enormous and the hard work of our team truly paid off. Our systems ran incredibly smoothly. It was a monumental victory, especially after the rocky start to the year. But, mentally, I wasn’t prepared to savor the moment. Yet, the energy post-offsite was infectious. People genuinely enjoyed meeting each other in real life and were fired up to start shipping. We couldn’t let this opportunity slip through our fingers. I paid homage to my mom’s teachings by continuing to press forward. RC Paywalls One of the major product ideas that had always been on our team’s wishlist was paywalls. We hadn’t tackled it because it seemed daunting and we lacked a product team with all the necessary skills. We couldn’t even estimate how long it would take to build. But, fueled by the success of RC Fortress, we decided to take a shot at it. We assembled a small team with members borrowed from different corners of the company. We didn’t mess with any reporting structures but appointed a leader. We went back to our roots, working in a hackathon-style frenzy for a couple of weeks to build a prototype. Just like in the good old days. And, boy, did the team rise to the occasion. We gave them some extra time and, in the end, they delivered one of our biggest product wins of the year. Shipping paywalls felt like a breath of fresh air and a clear sign that we still had our mojo. We were still capable of shipping software at lightning speed and keeping our customers excited. Based on all these lessons, the Head of Product and I started cooking up a brand new way of building products at RevenueCat. Engineering/Product/Design changes The complete process overhaul would warrant a couple of blog posts, but let me highlight the key changes to our workflow: We’ve reviewed all existing teams to determine whether they should continue as is, be replaced, or undergo changes in their mission or structure. We’ve rebalanced and clarified the responsibilities across Product, Engineering, and Design: Engineering takes the lead on feasibility, delivery, and developer experience, managing Linear, and overseeing technical architecture (and debt) roadmaps. Product is dedicated to customer value, business viability, and collaboration with sales and marketing teams. They also provide support to engineering in refining project scopes. Design is responsible for usability. We’ve formalized the role of Tech Lead, assigning them as the Directly Responsible Individuals for specific projects. They’re accountable for project success, with full backing from their Engineering Manager. It’s optional and project-dependent and doesn’t entail a title or salary change. We’ve acknowledged the need for engineering-driven initiatives, where PM and Design are involved on an as-needed basis. For projects with dependencies with other teams, we’ll designate a team member from the collaborating team as a formal interface. Our existing setup of stable product teams will remain the norm for most of our work. Temporary project teams will be established only when there’s a strong need for cross-team collaboration over a limited period. We’ll conduct monthly roadmap and shipping reviews with the founders and Head of Product. These reviews will provide insights into what we’re building, offer feedback opportunities, and help identify cross-functional dependencies and misalignments. 2024: The year of shipping + selling Collaborating with the Product team showcased the immense benefits of working closely together. Traditionally, Product reported directly to our CEO, which introduced unnecessary layers of indirection. In light of this and our recent addition of a VP of Sales, we decided it was time for a reorganization. Currently, Jacob (co-founder, and CEO), is overseeing Go to Market, People, and Operations, while I’m responsible for Engineering, Product, and Customer Engineering. We’ve brought in a VP of Customer Engineering, who reports to me and is in charge of Support and Technical Account Management. With our current headcount at 73 employees, my organization consists of 48 team members. Our executive team developed the most comprehensive planning effort to date. Our goal is to accelerate growth, focusing on sales and product delivery. We will avoid distractions by being extremely strategic at hiring. The past quarter showcased the strength of our engineering and product teams. New team members have been onboarded successfully, contributing meaningfully, and our management structure is finally robust. It took a bit of time for the “year of shipping” to fully materialize, nearly a year later, but customers have taken notice and we’re capitalizing on this momentum. On the enterprise sales front, I’m extremely bullish. We’ve secured the biggest deals in the company’s history. RevenueCat has evolved beyond being a product just for indie developers. However, we acknowledge the need to continue closing the product-market fit gap for enterprise clients. We’ve gained valuable insights into enterprise needs, and we’ll keep developing new products and features tailored to them. Indie developers will also take advantage of them to make more money. In 2024, the collaboration between our go-to-market and engineering teams will be critical. Highlights This year, as you’ve probably noticed, was a tough one. However, besides the challenges, there were several remarkable achievements to celebrate: We truly shipped. Our second hackathon, spanning an entire week, was an epic success. Many of the projects launched immediately, directly benefiting our customers. We had the privilege of collaborating with prominent brands and companies, including none other than Arnold Schwarzenegger himself. We created an astonishing amount of high-quality content. Our SubClub podcast outperformed all my expectations. During the chaos of the bank run, we discovered that one of our idols was not only a RevenueCat customer but also a devoted fan. We experimented with fresh team topologies and processes, and they turned out to be successful. We fine-tuned our vision for building a winning team, offering improved feedback, clearer expectations, and timely performance management. Jacob and I are no longer the sole authorities on Apple and Google subscriptions within the company. We successfully recruited seasoned executives to join our team. Our core infrastructure team accomplished monumental feats. We now support over 1 billion API requests daily, transitioned to our own data platform, and developed our own memcached client. All while maintaining flat costs despite the increase in load. Learnings It’s impossible to achieve peak performance without attention to health, exercise, and sleep. I’m no longer in my twenties. Complexity is the root of all evil. Startups and software are inherently complex, so avoid introducing unnecessary complexity. Begin by building the simplest feature or process, debug it, and then iterate as needed. Starting a business is tough, but launching a remote startup is an even greater challenge. Scaling a remote startup while parenting two under two is a herculean effort. Building a brand takes years but its reputation can be destroyed in an instant. Protect the integrity of your brand at all costs. Every new team member should add value, and especially so in a startup. Some provide immense leverage, while others become bottlenecks. The trickiest are those in the middle, who often end up becoming bottlenecks. Founders usually spot this within the first few weeks. Letting someone go is a taxing task. Even when managers believe it’s the right thing to do, it often requires a significant amount of support and guidance. Transparency in times of crisis pays dividends. Employees want to be treated like adults, and it builds trust. The same applies externally. Early worries often become baseless. By the time they become actual problems, your company might have died, you might have gained experience, or you might have hired the right talent to tackle them. SOC 2 auditors may request unconventional supporting evidence, such as employee performance reviews. Developers love socks. I’m so fortunate to have the world’s best co-founder. At this stage, my role is much more aligned with that of a founder than a traditional CTO or VP of Engineering. I continue to address issues as if they were technical problems, but my responsibilities extend well beyond the technology area. Life keeps moving forward, with its share of highs and lows. Life is too short, so you have to ensure the journey remains fun. For me, that means working with people who inspire me, and serving customers I genuinely care about. I really hope you enjoyed reading this post. I’m aware it’s a lot longer than my previous ones, but there were so many stories to share. As always, my intention was to share it with complete honesty and transparency, avoiding the hype that often surrounds startups. If you are facing similar challenges and want to connect and share experiences, please do not hesitate to reach out on Twitter or shoot me an email! Special thanks to my co-founder Jacob, the whole RevenueCat team, and our valuable customers. I also need to express my eternal gratitude to all the CTOs and leaders who have been kind enough to share their experiences over these years. Shoutout to Dani Lopez, Peter Silberman, Alex Plugaru, Kwindla Hultman Kramer, João Batalha, Karri Saarinen, Miguel Martinez Triviño, Sam Lown, Javi Santana, Pau Ramón, Javier Maestro, Matias Woloski, Tobias Balling, Jason Warner, and Will Larson. Our investors and early believers Jason Lemkin, Anu Hariharan, Mark Fiorentino, Mark Goldberg, Andrew Maguire, Gustaf Alströmer, and Nico Wittenborn. I want to convey my deep gratitude to my amazing wife, Marina, who has been my unwavering source of inspiration and support from the very beginning, and for blessing us with our two precious daughters. I cannot close this post without thanking my mom, who made countless sacrifices to mold me into the person I am today. I promise you will look down on us with pride the day we ring the bell in New York. I love you dearly.
More in programming
This is a re-publishing of a blog post I originally wrote for work, but wanted on my own blog as well. AI is everywhere, and its impressive claims are leading to rapid adoption. At this stage, I’d qualify it as charismatic technology—something that under-delivers on what it promises, but promises so much that the industry still leverages it because we believe it will eventually deliver on these claims. This is a known pattern. In this post, I’ll use the example of automation deployments to go over known patterns and risks in order to provide you with a list of questions to ask about potential AI solutions. I’ll first cover a short list of base assumptions, and then borrow from scholars of cognitive systems engineering and resilience engineering to list said criteria. At the core of it is the idea that when we say we want humans in the loop, it really matters where in the loop they are. My base assumptions The first thing I’m going to say is that we currently do not have Artificial General Intelligence (AGI). I don’t care whether we have it in 2 years or 40 years or never; if I’m looking to deploy a tool (or an agent) that is supposed to do stuff to my production environments, it has to be able to do it now. I am not looking to be impressed, I am looking to make my life and the system better. Another mechanism I want you to keep in mind is something called the context gap. In a nutshell, any model or automation is constructed from a narrow definition of a controlled environment, which can expand as it gains autonomy, but remains limited. By comparison, people in a system start from a broad situation and narrow definitions down and add constraints to make problem-solving tractable. One side starts from a narrow context, and one starts from a wide one—so in practice, with humans and machines, you end up seeing a type of teamwork where one constantly updates the other: The optimal solution of a model is not an optimal solution of a problem unless the model is a perfect representation of the problem, which it never is. — Ackoff (1979, p. 97) Because of that mindset, I will disregard all arguments of “it’s coming soon” and “it’s getting better real fast” and instead frame what current LLM solutions are shaped like: tools and automation. As it turns out, there are lots of studies about ergonomics, tool design, collaborative design, where semi-autonomous components fit into sociotechnical systems, and how they tend to fail. Additionally, I’ll borrow from the framing used by people who study joint cognitive systems: rather than looking only at the abilities of what a single person or tool can do, we’re going to look at the overall performance of the joint system. This is important because if you have a tool that is built to be operated like an autonomous agent, you can get weird results in your integration. You’re essentially building an interface for the wrong kind of component—like using a joystick to ride a bicycle. This lens will assist us in establishing general criteria about where the problems will likely be without having to test for every single one and evaluate them on benchmarks against each other. Questions you'll want to ask The following list of questions is meant to act as reminders—abstracting away all the theory from research papers you’d need to read—to let you think through some of the important stuff your teams should track, whether they are engineers using code generation, SREs using AIOps, or managers and execs making the call to adopt new tooling. Are you better even after the tool is taken away? An interesting warning comes from studying how LLMs function as learning aides. The researchers found that people who trained using LLMs tended to fail tests more when the LLMs were taken away compared to people who never studied with them, except if the prompts were specifically (and successfully) designed to help people learn. Likewise, it’s been known for decades that when automation handles standard challenges, the operators expected to take over when they reach their limits end up worse off and generally require more training to keep the overall system performant. While people can feel like they’re getting better and more productive with tool assistance, it doesn’t necessarily follow that they are learning or improving. Over time, there’s a serious risk that your overall system’s performance will be limited to what the automation can do—because without proper design, people keeping the automation in check will gradually lose the skills they had developed prior. Are you augmenting the person or the computer? Traditionally successful tools tend to work on the principle that they improve the physical or mental abilities of their operator: search tools let you go through more data than you could on your own and shift demands to external memory, a bicycle more effectively transmits force for locomotion, a blind spot alert on your car can extend your ability to pay attention to your surroundings, and so on. Automation that augments users therefore tends to be easier to direct, and sort of extends the person’s abilities, rather than acting based on preset goals and framing. Automation that augments a machine tends to broaden the device’s scope and control by leveraging some known effects of their environment and successfully hiding them away. For software folks, an autoscaling controller is a good example of the latter. Neither is fundamentally better nor worse than the other—but you should figure out what kind of automation you’re getting, because they fail differently. Augmenting the user implies that they can tackle a broader variety of challenges effectively. Augmenting the computers tends to mean that when the component reaches its limits, the challenges are worse for the operator. Is it turning you into a monitor rather than helping build an understanding? If your job is to look at the tool go and then say whether it was doing a good or bad job (and maybe take over if it does a bad job), you’re going to have problems. It has long been known that people adapt to their tools, and automation can create complacency. Self-driving cars that generally self-drive themselves well but still require a monitor are not effectively monitored. Instead, having AI that supports people or adds perspectives to the work an operator is already doing tends to yield better long-term results than patterns where the human learns to mostly delegate and focus elsewhere. (As a side note, this is why I tend to dislike incident summarizers. Don’t make it so people stop trying to piece together what happened! Instead, I prefer seeing tools that look at your summaries to remind you of items you may have forgotten, or that look for linguistic cues that point to biases or reductive points of view.) Does it pigeonhole what you can look at? When evaluating a tool, you should ask questions about where the automation lands: Does it let you look at the world more effectively? Does it tell you where to look in the world? Does it force you to look somewhere specific? Does it tell you to do something specific? Does it force you to do something? This is a bit of a hybrid between “Does it extend you?” and “Is it turning you into a monitor?” The five questions above let you figure that out. As the tool becomes a source of assertions or constraints (rather than a source of information and options), the operator becomes someone who interacts with the world from inside the tool rather than someone who interacts with the world with the tool’s help. The tool stops being a tool and becomes a representation of the whole system, which means whatever limitations and internal constraints it has are then transmitted to your users. Is it a built-in distraction? People tend to do multiple tasks over many contexts. Some automated systems are built with alarms or alerts that require stealing someone’s focus, and unless they truly are the most critical thing their users could give attention to, they are going to be an annoyance that can lower the effectiveness of the overall system. What perspectives does it bake in? Tools tend to embody a given perspective. For example, AIOps tools that are built to find a root cause will likely carry the conceptual framework behind root causes in their design. More subtly, these perspectives are sometimes hidden in the type of data you get: if your AIOps agent can only see alerts, your telemetry data, and maybe your code, it will rarely be a source of suggestions on how to improve your workflows because that isn’t part of its world. In roles that are inherently about pulling context from many disconnected sources, how on earth is automation going to make the right decisions? And moreover, who’s accountable for when it makes a poor decision on incomplete data? Surely not the buyer who installed it! This is also one of the many ways in which automation can reinforce biases—not just based on what is in its training data, but also based on its own structure and what inputs were considered most important at design time. The tool can itself become a keyhole through which your conclusions are guided. Is it going to become a hero? A common trope in incident response is heroes—the few people who know everything inside and out, and who end up being necessary bottlenecks to all emergencies. They can’t go away for vacation, they’re too busy to train others, they develop blind spots that nobody can fix, and they can’t be replaced. To avoid this, you have to maintain a continuous awareness of who knows what, and crosstrain each other to always have enough redundancy. If you have a team of multiple engineers and you add AI to it, having it do all of the tasks of a specific kind means it becomes a de facto hero to your team. If that’s okay, be aware that any outages or dysfunction in the AI agent would likely have no practical workaround. You will essentially have offshored part of your ops. Do you need it to be perfect? What a thing promises to be is never what it is—otherwise AWS would be enough, and Kubernetes would be enough, and JIRA would be enough, and the software would work fine with no one needing to fix things. That just doesn’t happen. Ever. Even if it’s really, really good, it’s gonna have outages and surprises, and it’ll mess up here and there, no matter what it is. We aren’t building an omnipotent computer god, we’re building imperfect software. You’ll want to seriously consider whether the tradeoffs you’d make in terms of quality and cost are worth it, and this is going to be a case-by-case basis. Just be careful not to fix the problem by adding a human in the loop that acts as a monitor! Is it doing the whole job or a fraction of it? We don’t notice major parts of our own jobs because they feel natural. A classic pattern here is one of AIs getting better at diagnosing patients, except the benchmarks are usually run on a patient chart where most of the relevant observations have already been made by someone else. Similarly, we often see AI pass a test with flying colors while it still can’t be productive at the job the test represents. People in general have adopted a model of cognition based on information processing that’s very similar to how computers work (get data in, think, output stuff, rinse and repeat), but for decades, there have been multiple disciplines that looked harder at situated work and cognition, moving past that model. Key patterns of cognition are not just in the mind, but are also embedded in the environment and in the interactions we have with each other. Be wary of acquiring a solution that solves what you think the problem is rather than what it actually is. We routinely show we don’t accurately know the latter. What if we have more than one? You probably know how straightforward it can be to write a toy project on your own, with full control of every refactor. You probably also know how this stops being true as your team grows. As it stands today, a lot of AI agents are built within a snapshot of the current world: one or few AI tools added to teams that are mostly made up of people. By analogy, this would be like everyone selling you a computer assuming it were the first and only electronic device inside your household. Problems arise when you go beyond these assumptions: maybe AI that writes code has to go through a code review process, but what if that code review is done by another unrelated AI agent? What happens when you get to operations and common mode failures impact components from various teams that all have agents empowered to go fix things to the best of their ability with the available data? Are they going to clash with people, or even with each other? Humans also have that ability and tend to solve it via processes and procedures, explicit coordination, announcing what they’ll do before they do it, and calling upon each other when they need help. Will multiple agents require something equivalent, and if so, do you have it in place? How do they cope with limited context? Some changes that cause issues might be safe to roll back, some not (maybe they include database migrations, maybe it is better to be down than corrupting data), and some may contain changes that rolling back wouldn’t fix (maybe the workload is controlled by one or more feature flags). Knowing what to do in these situations can sometimes be understood from code or release notes, but some situations can require different workflows involving broader parts of the organization. A risk of automation without context is that if you have situations where waiting or doing little is the best option, then you’ll need to either have automation that requires input to act, or a set of actions to quickly disable multiple types of automation as fast as possible. Many of these may exist at the same time, and it becomes the operators’ jobs to not only maintain their own context, but also maintain a mental model of the context each of these pieces of automation has access to. The fancier your agents, the fancier your operators’ understanding and abilities must be to properly orchestrate them. The more surprising your landscape is, the harder it can become to manage with semi-autonomous elements roaming around. After an outage or incident, who does the learning and who does the fixing? One way to track accountability in a system is to figure out who ends up having to learn lessons and change how things are done. It’s not always the same people or teams, and generally, learning will happen whether you want it or not. This is more of a rhetorical question right now, because I expect that in most cases, when things go wrong, whoever is expected to monitor the AI tool is going to have to steer it in a better direction and fix it (if they can); if it can’t be fixed, then the expectation will be that the automation, as a tool, will be used more judiciously in the future. In a nutshell, if the expectation is that your engineers are going to be doing the learning and tweaking, your AI isn’t an independent agent—it’s a tool that cosplays as an independent agent. Do what you will—just be mindful All in all, none of the above questions flat out say you should not use AI, nor where exactly in the loop you should put people. The key point is that you should ask that question and be aware that just adding whatever to your system is not going to substitute workers away. It will, instead, transform work and create new patterns and weaknesses. Some of these patterns are known and well-studied. We don’t have to go rushing to rediscover them all through failures as if we were the first to ever automate something. If AI ever gets so good and so smart that it’s better than all your engineers, it won’t make a difference whether you adopt it only once it’s good. In the meanwhile, these things do matter and have real impacts, so please design your systems responsibly. If you’re interested to know more about the theoretical elements underpinning this post, the following references—on top of whatever was already linked in the text—might be of interest: Books: Joint Cognitive Systems: Foundations of Cognitive Systems Engineering by Erik Hollnagel Joint Cognitive Systems: Patterns in Cognitive Systems Engineering by David D. Woods Cognition in the Wild by Edwin Hutchins Behind Human Error by David D. Woods, Sydney Dekker, Richard Cook, Leila Johannesen, Nadine Sarter Papers: Ironies of Automation by Lisanne Bainbridge The French-Speaking Ergonomists’ Approach to Work Activity by Daniellou How in the World Did We Ever Get into That Mode? Mode Error and Awareness in Supervisory Control by Nadine Sarter Can We Ever Escape from Data Overload? A Cognitive Systems Diagnosis by David D. Woods Ten Challenges for Making Automation a “Team Player” in Joint Human-Agent Activity by Gary Klein and David D. Woods MABA-MABA or Abracadabra? Progress on Human–Automation Co-ordination by Sidney Dekker Managing the Hidden Costs of Coordination by Laura Maguire Designing for Expertise by David D. Woods The Impact of Generative AI on Critical Thinking by Lee et al.
AMD is sending us the two MI300X boxes we asked for. They are in the mail. It took a bit, but AMD passed my cultural test. I now believe they aren’t going to shoot themselves in the foot on software, and if that’s true, there’s absolutely no reason they should be worth 1/16th of NVIDIA. CUDA isn’t really the moat people think it is, it was just an early ecosystem. tiny corp has a fully sovereign AMD stack, and soon we’ll port it to the MI300X. You won’t even have to use tinygrad proper, tinygrad has a torch frontend now. Either NVIDIA is super overvalued or AMD is undervalued. If the petaflop gets commoditized (tiny corp’s mission), the current situation doesn’t make any sense. The hardware is similar, AMD even got the double throughput Tensor Cores on RDNA4 (NVIDIA artificially halves this on their cards, soon they won’t be able to). I’m betting on AMD being undervalued, and that the demand for AI has barely started. With good software, the MI300X should outperform the H100. In for a quarter million. Long term. It can always dip short term, but check back in 5 years.
Earlier this weekGuileWhippet But now I do! Today’s note is about how we can support untagged allocations of a few different kinds in Whippet’s .mostly-marking collector Why bother supporting untagged allocations at all? Well, if I had my way, I wouldn’t; I would just slog through Guile and fix all uses to be tagged. There are only a finite number of use sites and I could get to them all in a month or so. The problem comes for uses of from outside itself, in C extensions and embedding programs. These users are loathe to adapt to any kind of change, and garbage-collection-related changes are the worst. So, somehow, we need to support these users if we are not to break the Guile community.scm_gc_malloclibguile The problem with , though, is that it is missing an expression of intent, notably as regards tagging. You can use it to allocate an object that has a tag and thus can be traced precisely, or you can use it to allocate, well, anything else. I think we will have to add an API for the tagged case and assume that anything that goes through is requesting an untagged, conservatively-scanned block of memory. Similarly for : you could be allocating a tagged object that happens to not contain pointers, or you could be allocating an untagged array of whatever. A new API is needed there too for pointerless untagged allocations.scm_gc_mallocscm_gc_mallocscm_gc_malloc_pointerless Recall that the mostly-marking collector can be built in a number of different ways: it can support conservative and/or precise roots, it can trace the heap precisely or conservatively, it can be generational or not, and the collector can use multiple threads during pauses or not. Consider a basic configuration with precise roots. You can make tagged pointerless allocations just fine: the trace function for that tag is just trivial. You would like to extend the collector with the ability to make pointerless allocations, for raw data. How to do this?untagged Consider first that when the collector goes to trace an object, it can’t use bits inside the object to discriminate between the tagged and untagged cases. Fortunately though . Of those 8 bits, 3 are used for the mark (five different states, allowing for future concurrent tracing), two for the , one to indicate whether the object is pinned or not, and one to indicate the end of the object, so that we can determine object bounds just by scanning the metadata byte array. That leaves 1 bit, and we can use it to indicate untagged pointerless allocations. Hooray!the main space of the mostly-marking collector has one metadata byte for each 16 bytes of payloadprecise field-logging write barrier However there is a wrinkle: when Whippet decides the it should evacuate an object, it tracks the evacuation state in the object itself; the embedder has to provide an implementation of a , allowing the collector to detect whether an object is forwarded or not, to claim an object for forwarding, to commit a forwarding pointer, and so on. We can’t do that for raw data, because all bit states belong to the object, not the collector or the embedder. So, we have to set the “pinned” bit on the object, indicating that these objects can’t move.little state machine We could in theory manage the forwarding state in the metadata byte, but we don’t have the bits to do that currently; maybe some day. For now, untagged pointerless allocations are pinned. You might also want to support untagged allocations that contain pointers to other GC-managed objects. In this case you would want these untagged allocations to be scanned conservatively. We can do this, but if we do, it will pin all objects. Thing is, conservative stack roots is a kind of a sweet spot in language run-time design. You get to avoid constraining your compiler, you avoid a class of bugs related to rooting, but you can still support compaction of the heap. How is this, you ask? Well, consider that you can move any object for which we can precisely enumerate the incoming references. This is trivially the case for precise roots and precise tracing. For conservative roots, we don’t know whether a given edge is really an object reference or not, so we have to conservatively avoid moving those objects. But once you are done tracing conservative edges, any live object that hasn’t yet been traced is fair game for evacuation, because none of its predecessors have yet been visited. But once you add conservatively-traced objects back into the mix, you don’t know when you are done tracing conservative edges; you could always discover another conservatively-traced object later in the trace, so you have to pin everything. The good news, though, is that we have gained an easier migration path. I can now shove Whippet into Guile and get it running even before I have removed untagged allocations. Once I have done so, I will be able to allow for compaction / evacuation; things only get better from here. Also as a side benefit, the mostly-marking collector’s heap-conservative configurations are now faster, because we have metadata attached to objects which allows tracing to skip known-pointerless objects. This regains an optimization that BDW has long had via its , used in Guile since time out of mind.GC_malloc_atomic With support for untagged allocations, I think I am finally ready to start getting Whippet into Guile itself. Happy hacking, and see you on the other side! inside and outside on intent on data on slop fin
I’ve been working on a project where I need to plot points on a map. I don’t need an interactive or dynamic visualisation – just a static map with coloured dots for each coordinate. I’ve created maps on the web using Leaflet.js, which load map data from OpenStreetMap (OSM) and support zooming and panning – but for this project, I want a standalone image rather than something I embed in a web page. I want to put in coordinates, and get a PNG image back. This feels like it should be straightforward. There are lots of Python libraries for data visualisation, but it’s not an area I’ve ever explored in detail. I don’t know how to use these libraries, and despite trying I couldn’t work out how to accomplish this seemingly simple task. I made several attempts with libraries like matplotlib and plotly, but I felt like I was fighting the tools. Rather than persist, I wrote my own solution with “lower level” tools. The key was a page on the OpenStreetMap wiki explaining how to convert lat/lon coordinates into the pixel system used by OSM tiles. In particular, it allowed me to break the process into two steps: Get a “base map” image that covers the entire world Convert lat/lon coordinates into xy coordinates that can be overlaid on this image Let’s go through those steps. Get a “base map” image that covers the entire world Let’s talk about how OpenStreetMap works, and in particular their image tiles. If you start at the most zoomed-out level, OSM represents the entire world with a single 256×256 pixel square. This is the Web Mercator projection, and you don’t get much detail – just a rough outline of the world. We can zoom in, and this tile splits into four new tiles of the same size. There are twice as many pixels along each edge, and each tile has more detail. Notice that country boundaries are visible now, but we can’t see any names yet. We can zoom in even further, and each of these tiles split again. There still aren’t any text labels, but the map is getting more detailed and we can see small features that weren’t visible before. You get the idea – we could keep zooming, and we’d get more and more tiles, each with more detail. This tile system means you can get detailed information for a specific area, without loading the entire world. For example, if I’m looking at street information in Britain, I only need the detailed tiles for that part of the world. I don’t need the detailed tiles for Bolivia at the same time. OpenStreetMap will only give you 256×256 pixels at a time, but we can download every tile and stitch them together, one-by-one. Here’s a Python script that enumerates all the tiles at a particular zoom level, downloads them, and uses the Pillow library to combine them into a single large image: #!/usr/bin/env python3 """ Download all the map tiles for a particular zoom level from OpenStreetMap, and stitch them into a single image. """ import io import itertools import httpx from PIL import Image zoom_level = 2 width = 256 * 2**zoom_level height = 256 * (2**zoom_level) im = Image.new("RGB", (width, height)) for x, y in itertools.product(range(2**zoom_level), range(2**zoom_level)): resp = httpx.get(f"https://tile.openstreetmap.org/{zoom_level}/{x}/{y}.png", timeout=50) resp.raise_for_status() im_buffer = Image.open(io.BytesIO(resp.content)) im.paste(im_buffer, (x * 256, y * 256)) out_path = f"map_{zoom_level}.png" im.save(out_path) print(out_path) The higher the zoom level, the more tiles you need to download, and the larger the final image will be. I ran this script up to zoom level 6, and this is the data involved: Zoom level Number of tiles Pixels File size 0 1 256×256 17.1 kB 1 4 512×512 56.3 kB 2 16 1024×1024 155.2 kB 3 64 2048×2048 506.4 kB 4 256 4096×4096 2.7 MB 5 1,024 8192×8192 13.9 MB 6 4,096 16384×16384 46.1 MB I can just about open that zoom level 6 image on my computer, but it’s struggling. I didn’t try opening zoom level 7 – that includes 16,384 tiles, and I’d probably run out of memory. For most static images, zoom level 3 or 4 should be sufficient – I ended up a base map from zoom level 4 for my project. It takes a minute or so to download all the tiles from OpenStreetMap, but you only need to request it once, and then you have a static image you can use again and again. This is a particularly good approach if you want to draw a lot of maps. OpenStreetMap is provided for free, and we want to be a respectful user of the service. Downloading all the map tiles once is more efficient than making repeated requests for the same data. Overlay lat/lon coordinates on this base map Now we have an image with a map of the whole world, we need to overlay our lat/lon coordinates as points on this map. I found instructions on the OpenStreetMap wiki which explain how to convert GPS coordinates into a position on the unit square, which we can in turn add to our map. They outline a straightforward algorithm, which I implemented in Python: import math def convert_gps_coordinates_to_unit_xy( *, latitude: float, longitude: float ) -> tuple[float, float]: """ Convert GPS coordinates to positions on the unit square, which can be plotted on a Web Mercator projection of the world. This expects the coordinates to be specified in **degrees**. The result will be (x, y) coordinates: - x will fall in the range (0, 1). x=0 is the left (180° west) edge of the map. x=1 is the right (180° east) edge of the map. x=0.5 is the middle, the prime meridian. - y will fall in the range (0, 1). y=0 is the top (north) edge of the map, at 85.0511 °N. y=1 is the bottom (south) edge of the map, at 85.0511 °S. y=0.5 is the middle, the equator. """ # This is based on instructions from the OpenStreetMap Wiki: # https://wiki.openstreetmap.org/wiki/Slippy_map_tilenames#Example:_Convert_a_GPS_coordinate_to_a_pixel_position_in_a_Web_Mercator_tile # (Retrieved 16 January 2025) # Convert the coordinate to the Web Mercator projection # (https://epsg.io/3857) # # x = longitude # y = arsinh(tan(latitude)) # x_webm = longitude y_webm = math.asinh(math.tan(math.radians(latitude))) # Transform the projected point onto the unit square # # x = 0.5 + x / 360 # y = 0.5 - y / 2π # x_unit = 0.5 + x_webm / 360 y_unit = 0.5 - y_webm / (2 * math.pi) return x_unit, y_unit Their documentation includes a worked example using the coordinates of the Hachiko Statue. We can run our code, and check we get the same results: >>> convert_gps_coordinates_to_unit_xy(latitude=35.6590699, longitude=139.7006793) (0.8880574425, 0.39385379958274735) Most users of OpenStreetMap tiles will use these unit positions to select the tiles they need, and then dowload those images – but we can also position these points directly on the global map. I wrote some more Pillow code that converts GPS coordinates to these unit positions, scales those unit positions to the size of the entire map, then draws a coloured circle at each point on the map. Here’s the code: from PIL import Image, ImageDraw gps_coordinates = [ # Hachiko Memorial Statue in Tokyo {"latitude": 35.6590699, "longitude": 139.7006793}, # Greyfriars Bobby in Edinburgh {"latitude": 55.9469224, "longitude": -3.1913043}, # Fido Statue in Tuscany {"latitude": 43.955101, "longitude": 11.388186}, ] im = Image.open("base_map.png") draw = ImageDraw.Draw(im) for coord in gps_coordinates: x, y = convert_gps_coordinates_to_unit_xy(**coord) radius = 32 draw.ellipse( [ x * im.width - radius, y * im.height - radius, x * im.width + radius, y * im.height + radius, ], fill="red", ) im.save("map_with_dots.png") and here’s the map it produces: The nice thing about writing this code in Pillow is that it’s a library I already know how to use, and so I can customise it if I need to. I can change the shape and colour of the points, or crop to specific regions, or add text to the image. I’m sure more sophisticated data visualisation libraries can do all this, and more – but I wouldn’t know how. The downside is that if I need more advanced features, I’ll have to write them myself. I’m okay with that – trading sophistication for simplicity. I didn’t need to learn a complex visualization library – I was able to write code I can read and understand. In a world full of AI-generating code, writing something I know I understand feels more important than ever. [If the formatting of this post looks odd in your feed reader, visit the original article]
This website has a new section: blogroll.opml! A blogroll is a list of blogs - a lightweight way of people recommending other people’s writing on the indieweb. What it includes The blogs that I included are just sampled from my many RSS subscriptions that I keep in my Feedbin reader. I’m subscribed to about 200 RSS feeds, the majority of which are dead or only publish once a year. I like that about blogs, that there’s no expectation of getting a post out every single day, like there is in more algorithmically-driven media. If someone who I interacted with on the internet years ago decides to restart their writing, that’s great! There’s no reason to prune all the quiet feeds. The picks are oriented toward what I’m into: niches, blogs that have a loose topic but don’t try to be general-interest, people with distinctive writing. If you import all of the feeds into your RSS reader, you’ll probably end up unsubscribing from some of them because some of the experimental electric guitar design or bonsai news is not what you’re into. Seems fine, or you’ll discover a new interest! How it works Ruben Schade figured out a brilliant way to show blogrolls and I copied him. Check out his post on styling OPML and RSS with XSLT to XHTML for how it works. My only additions to that scheme were making the blogroll page blend into the rest of the website by using an include tag with Jekyll to add the basic site skeleton, and adding a link with the download attribute to provide a simple way to download the OPML file. Oddly, if you try to save the OPML page using Save as… in Firefox, Firefox will save the transformed output via the XSLT, rather than the raw source code. XSLT is such an odd and rare part of the web ecosystem, I had to use it.