More from Miguel Carranza
2024 has come and gone, and it’s time for my annual post. What a year for startups—like squeezing five regular years into one. Do you remember the Apple Vision Pro, the DMA regulation, founder mode, or the o1 launch? All of that happened in just the last twelve months. It’s also been wild at RevenueCat. My journal is full of stories that could fill a whole book or even a few HBO Silicon Valley seasons. Some are inspiring, some are hilarious, and others are honestly gnarly. Due to limited space, the need for context, and respecting everyone’s privacy, I’ll cover only the most interesting topics at a high level. Looking back, it was a good year for RevenueCat. Actually, a great one. Perhaps our best since 2020. We have plenty to celebrate: We accelerated again, and we hit our C10 revenue plan. We made our first acquisition and welcomed an amazing founder to the team. We signed our first multi-million-dollar contracts. We went to more than 20 events around the world… …including hosting our own conference, featuring its own award ceremony. We became the #1 payments SDK on iOS. Our swag went to 11. We launched over 80 user-facing features. We showed up in Times Square and along Highway 101. We raised a mini Series C and welcomed two new board members. We landed in Japan for the first time. Our API grew beyond 2B requests per day. We are processing nearly twice the TAM we had when we started the company. OpenAI is a friend of the cat. We continued building a winning team. We hired people who were on my to work with bucket list even before we started the company. By all the metrics, 2024 was our year of shipping and selling. We absolutely helped developers make more money. My role If you compare our progress to the goals I wrote about in last year’s blog post, it’s clear we succeeded. But the journey itself was a lot rockier than I had imagined. Many things didn’t go as planned, somes hires did not work out, and some strategy changes were really hard to push through. At the start of the year, besides my usual responsibilities, I also set four personal goals to help me scale with the company. I wanted to: Ship code more consistently. Talk to at least one customer every single day. Get more involved in areas outside of engineering. Stay very close to my co-founder Jacob, giving him my full support. I completely missed goal #1. Honestly, that hurts because I love building software. But I’m not too upset about it—our customers care that the team ships, not that I personally do. And in a way, I did help the team ship. As for the other three goals, I hit them, at least based on the company’s results. Still, my self-perception wasn’t always great. I found myself acting much more like a co-founder/executive than a CTO. Some weeks were brutal, with constant context switching across tasks and teams I didn’t always enjoy. At one point, there were over 60 people in my org, spread all around the world, which is pretty intense for an introverted computer kid. A lot of bullshit escalates to the top, making me question my entire existence some days. And then life threw serious personal emergencies at some of our team members too. Like I said last year, life is what happens when you’re busy building your startup. As we got closer to 100 people, it was statistically unavoidable that we’d face a few life-changing traumas—sometimes several all at once. As a founder, you need to be supportive and empathetic, but also protect your mental health. As a human, it’s tough. Add in a couple of two-year-olds who were often sick and not sleeping, and it felt like a ticking bomb. Did I burn out for the first time in my life? I don’t think so, but it got close. I’m confident being a founder (and having a co-founder) kept me going. I care too much and must stay resilient. If I’d been just an employee, I might have tapped out. But enough about the tough parts. Aside from being a professional BS handler, here’s how I spent most of my time: Lots of travel This was the year I traveled the most in my life—and looking back, I probably should have traveled even more. I visited customers, helped with sales, conducted executive interviews, and spoke at a few conferences. It’s hard being away from two little kids, but each trip turned out to be worth it. Support One of my biggest worries this year was our support function. Everything was fine, but our Support Engineering Manager was going on parental leave, and the bus factor was scary. I’ve seen support crises before—they’re not fun. It wouldn’t have killed the company, but at our size, it might have forced us to pull senior engineers into support and slow down our product velocity. Time was short, so we tried a few things that worked: We hired two new Developer Support Engineers who already knew our product—former customers! Their onboarding was smooth, and they hit the ground running. We split the support team into two pods with their own leads. Each pod handles certain tickets, and they collaborate with each other instead of relying too heavily on one person. We finally set up our first on-call rotation for emergencies. Sales and post-sales Reviewing sales and implementation calls, giving technical input, collecting enterprise customer feedback for product and engineering, joining calls, and even doing some in-person visits. Writing more As our team continues to grow across the globe, not everyone has the same direct interaction with me or Jacob as before. A lot of our culture and collaboration style was once passed through observation, but now needs to be written down to reach everyone faster. These days, my code editor is basically replaced by Google Docs. I’ve been publishing more internal and external documents—like our Engineering Strategy and an updated How to Work with Miguel. Product delivery We still have a few details to refine, but our founder Shipping and Timeline reviews have been valuable. It gives Jacob and me a high-level view of everything in progress, lets us offer feedback, and helps us dig deeper where needed. It’s a great way to see each team’s capacity, find bottlenecks, keep a sense of urgency, and deflate anything that’s not truly important for the customers. Re-orgs This year brought a couple of big reorgs in product and engineering. Overall, they went well, but the puzzle gets more complex with each new piece. We created sub-teams to narrow their focus and keep things running smoothly. For the first time, we had a couple of management layers between me and our ICs. One major change was shutting down our Enterprise/Reactive team. The idea was solid at first, and they delivered plenty of value, but eventually they became the random tasks team. Our new plan is to reinforce the rest of the teams: quick enterprise requests go to the right team to handle them reactively, while bigger projects move to the main roadmap (which keeps us disciplined). The engineers from that old team will help bootstrap new teams as we hire in 2025. Hiring Our engineering hiring goals weren’t super ambitious, but we still reached them. The pace was a bit uneven, and some roles took longer to fill than desired. However, when Hiring Managers took ownership of the process (with recruiting as a support) it made a huge difference in the quality of our candidates. We also brought back the founder interview stage: Jacob or I spoke with every single candidate before making an offer, and we plan to keep doing this for the foreseeable future. Random founder stuff Not my main focus, but I still spent a fair amount of time handling people-related topics, operations, investor, customers and partners relations, fundraising, company policies, planning, etc. Learnings: deepened insights On culture Shipping is king. Yes, deadlines are stressful, but failing to ship and getting stuck in endless debates is far worse. It’s depressing. If someone isn’t a good culture fit, it will be more than a single incident. Over time, it becomes pretty clear to everyone. Top performers tend to measure themselves against the very best in the company. Reassure them they are doing a great job. On the other hand, those who underperform will look to other underperformers to gauge their own progress. Even top performers will struggle if they don’t fully align with the vision. Nothing beats talking to customers. Encourage everyone to do it. Better in person. Most managers don’t have a strong incentive to be strict, so finding the right balance takes a lot of calibration. Only once everyone is aligned, you can truly delegate. On hiring Managers hate hiring because it’s binary: a lot of “no”, but eventually one “yes” can change everything. Staying consistent really helps. Simple things, such as weekly updates keep everyone accountable. People management sucks. If someone’s main motivation is to be a manager for the title, or so they can “lead,” that’s a red flag for me. Best managers end up being those who never planned on becoming one in the first place. They actually roll up their sleeves and can do the work. Big ideas are great, but they need to be executed. This matters even more when it comes to executives. A truly great exec can change your life and it will feel like a brand new company. Given their influence, anything less than great will eventually turn into a big mess. Spend time with executive candidates in person. Watch how they work and make sure they’re real builders. Previous founders and true engineers are usually a little bit de-risked, but they’re still not guaranteed. I also like to write a very detailed onboarding doc, clarifying context, expectations, and what success or failure looks like. Having them write a 30/60/90-day plan helps us all align. I’ve learned not to rely on their past pedigree. The real key is their glass eating endurance. On company building Re-orgs are inevitable at a growing startup, and they will feel scary or emotional for people who aren’t used to them. I find it helpful to be super clear about why we’re doing it, and to share the fallback plan if things don’t work out. But I’ve learned that by the time you think you need a re-org, you’re already behind. We now plan to reassess our team structure twice a year for optimal shipping. A numeric goal (like an SLA or revenue target) is just a proxy. Missing it isn’t the end of the world if you learn in the process. But if you surpass it without recognizing what’s broken underneath, you’ll be masking critical issues. Perfectionists are great, but can have a tough time at startups. Some do fine if they’re focused on one specific thing or working as an IC. But as soon as they have to juggle multiple tasks, the chaos can feel overwhelming. Help them embrace it. Things will break. It’s about continuously reprioritizing, and stopping small cracks from becoming big fires. No agenda == no meeting. Synchronous time is expensive. The only exception is the occasional unscheduled call. After you reach around 50 people (especially in a remote setup), documenting every process change becomes crucial. I’ve learned the hard way that simply talking about a new process isn’t enough. Founders can’t talk to everyone all the time anymore, and misunderstandings or gossip can spread fast. Not everyone will read everything, but at least there’s a single source of truth to reference. Best people can stretch quite a bit—they’ll often rise to the challenge. But it’s wise to keep an eye on their limits before they burn out or become a bottleneck. On scaling as a founder A great EA is life-changing. Family will be supportive, but it’s not fair to offload all the stress on them. They will end up feeling helpless. Having a solid co-founder, a network of peers, or a good executive coach makes a world of difference. Founders are the ultimate guardians of the culture. It’s constant work, and it will feel relentless—especially when things are going well and it’s easy to get entitled. The best team members will help uphold the standard, but you cannot expect them to do all the policing. At this stage, it’s stupid not to level up your lifestyle in ways that reduce stress or save time. This might include childcare support, having a second car, investing in a better mattress, or hiring help with housekeeping. The startup is bigger than its founders, and the goal is to continuously remove yourself from the critical path. Still, it’s easy to forget that, as a founder, you literally brought everything into existence from thin air. Imposter syndrome often creeps in when you step outside your core expertise, but if your gut feeling is strong, it’s worth paying attention to. Stay open-minded, yet remember that no one knows the company quite like you do. The future Next year is going to be another big one. We’ll keep shipping and selling, while finally tackling our design and UX debt. We’ll keep investing heavily in our infrastructure. Not just for reliability, but also for real-time data. We want RevenueCat to feel fast, accurate, and easy to use. We’re also upgrading our self-serve and enterprise support, aiming for a truly world-class experience. In many ways, we’re finally seeing the original vision Jacob and I had back in 2017 come to life. We’ll be launching new product lines too, and if we execute well, we will be just a couple of years away from hitting $100M in revenue. We’ll keep building a winning team. We’ll hire about 45 people, 30 in Engineering, Product, and Design. It’s a challenge, but totally doable. As for me, my personal goals haven’t changed much, but my perspective has. Jacob and I used to joke that being happy and winning can’t happen at the same time. But why not? We’re in a privileged position to shape our own path and change anything we don’t like. After a lot of reflection and coaching, I realized I was simply too hard on myself. I was feeling depressed by the constant BS even though we were winning. A hack that helped was working with my EA to set weekly goals and then sending a public update to the full team. It let me see the real progress behind all the drama and back-to-back meetings and stay transparent with everyone. Next year, I’ll avoid meetings before 9 AM, keep an eye on calendar creep, and hold myself accountable to exercise and doing what helps me decompress. I know, it’s obvious. I also plan to travel more. Especially to the Bay Area, which is clearly back again. Meeting up with other founders and customers is always worthwhile. Another thing that made last year tough was having half my direct reports on parental leave for about half of the year. Now they’re back, and I can feel the momentum returning. Jacob has also taken over product again, now that he’s stepped away from directly owning operations and people. Increased shipping velocity has been noticeable. We have all the pieces in place. All that’s left is to keep pushing forward: shipping, selling and enjoying the ride. If there’s one thing I learned in Silicon Valley, it’s that no goal is too crazy if you refuse to give up. I really hope you enjoyed reading this post. As always, my intention was to share it with complete honesty and transparency, avoiding the hype that often surrounds startups. If you are facing similar challenges and want to connect and share experiences, please do not hesitate to reach out on Twitter or shoot me an email! Special thanks to my co-founder Jacob, my EA Susannah, the whole RevenueCat team, and our valuable customers. I also need to express my eternal gratitude to all the CTOs and leaders who have been kind enough to share their experiences over these years. Shoutout to Dani Lopez, Peter Silberman, Alex Plugaru, Kwindla Hultman Kramer, João Batalha, Karri Saarinen, Miguel Martinez Triviño, Javi Santana, Matias Woloski, Tobias Balling, Jason Warner, and Will Larson. Our investors and early believers Jason Lemkin, Anu Hariharan, Mark Fiorentino, Mark Goldberg, Andrew Maguire, Gustaf Alströmer, Sofia Dolfie, and Nico Wittenborn. I want to convey my deep gratitude to my amazing wife, Marina, who has been my unwavering source of inspiration and support from the very beginning, and for blessing us with our two precious daughters. I cannot close this post without thanking my mom, who made countless sacrifices to mold me into the person I am today. I promise you will look down on us with pride the day we ring the bell in New York. I love you dearly.
I am drafting this post at 35,000 feet flying back from Japan. I’ve entered the US about 30 times, but this will be the first time I’ll be using my shiny blue passport. No anxiety about aggressive questions, secondary inspection, or the possibility of deportation. A couple of days ago, my wife had her naturalization ceremony, and with her, our whole family is now American. This post is a reflecting on our 11-year immigration journey. My American Dream My story with technology started at 8, with my first computer. I fell in love and decided that one day I would start a computer business. And of course, it would have to be in Silicon Valley, the epicenter of innovation. I grew up influenced by the iconic Californian lifestyle of the 90s, from Tony Hawk to bands like Blink-182 and The Offspring, which only fueled my desire to call the West Coast home. As I finished my Computer Science studies, the reality of achieving my American dream seemed increasingly distant. My enthusiasm for the Californian way of life hadn’t waned — I had started surfing and I was even playing in a pop-punk band. But the immigration complexities were too daunting. It looked impossible. I decided to study a Master’s Degree in the UK and reinforce my English. Right before completing my degree in England, I was offered a six-month internship at a startup in San Francisco. Financially, it was not the smartest decision — I would barely be able to afford rent, despite having better paid options in Europe. However, experiencing Silicon Valley was a lifelong dream. It wasn’t an obvious choice. But I ultimately packed my suitcase, left behind my family, girlfriend and friends, and relocated across the world. Landing in California My journey in the U.S. began with a J1 visa, intended for interns and relatively simple to secure with an employer’s backing. It was a suitable fit for my six-month plan, extendable to a full year, without any ambition for a longer stay. Yet, I worked extremely hard, and the situation changed when I was introduced to the possibility of obtaining an H-1B visa. Unlike the J1, the H-1B visa demands wage parity with U.S. citizens, allows for a stay of up to six years, and paves the way for permanent residency. Most importantly, it meant I could start laying down roots in the US, such as building my credit score, buying a car, or negotiating a long-term lease. First Immigration Problems My living situation dramatically improved when I traded my small, rat-filled room for a two-bedroom apartment in the Outer Sunset, sharing the space with a friend. I got a sizeable salary bump. However, the happiness was short-lived. H-1B visa applications had exceeded available spots for the first time in years, introducing a lottery. My coding skills, education, and value to my employer wouldn’t factor into this gamble. Anxiety mounted for a very long month, as friends celebrated their visa wins. I was left in the dark, bracing for bad news. Against the odds, relief came just a day after a disheartening talk with my immigration lawyer, granting me my first taste of luck. Reuniting with my girlfriend Sick. My H-1B visa approval meant I could make the US my home for six additional years, longer than Marina and I had been dating. As she was about to end her studies, we strategized on ways to reunite in the US. Marrying earlier was an option, but with H-1B restrictions preventing spouses from working, we looked for alternatives. The easiest path forward involved securing a student visa, leading to an 12-month work permit, followed by an H-1B visa application. In the competitive climate of 2015, her work visa acceptance felt nothing short of miraculous, becoming our story’s second lucky strike. Permanent Residency While six years might seem long, they pass quickly when you are busy and having fun. Halfway through, it became clear we needed to strategize for the future when it was time to renew my visa. My employer agreed to initiate the Green Card application, a lengthy and costly process requiring proof that I was indispensable for the company. Despite the complexities, our attorney believed the case would progress smoothly, estimating an 18-month completion time. Surprisingly, the initial phase went way faster than anticipated, prompting our attorney to suggest an immediate wedding for Marina and me, a necessary step to include her in the Green Card application. We quickly scheduled our wedding at the Spanish Consulate in San Francisco, departing from our original plan for a ceremony in Spain. More problems A year and a half in, expecting our Green Cards, we faced an unexpected challenge: our marriage, officiated at a consulate, was not recognized by US Immigration, compelling us to marry again, pay the associated fees again, and start the process over. This development was extremely frustrating, specially as my friend Jacob and I were contemplating founding a company, and the absence of a Green Card meant remaining as an employee. To rectify the situation, we promptly got remarried at San Mateo City Hall, choosing it for its rapid scheduling. Two months later we held another ceremony in Spain with our family and friends (our third marriage overall). The delay in our reapplication, exacerbated by the recent election of Trump and a subsequent slowdown in immigration services, led us into a stressful period of uncertainty. Our inability to make definite plans, from household purchases to housing arrangements left us anxiously awaiting any news on our application. As we navigated this uncertainty, the possibility of dedicating myself fully to our startup (later named RevenueCat) became increasingly dim. Going All In A pivotal moment came when our now startup, RevenueCat, was accepted into Y Combinator, requiring my full-time commitment. A big challenge due to my pending Green Card application. My immigration status became the biggest risk to our startup, even before launching. In searching for a solution, we identified a potential hack: an immigration loophole that allowed for employment changes under specific circumstances. It wasn’t risk free. There were no guarantees, and it involved giving up my H-1B status and the ability to travel. Should my Green Card application face rejection for any reason, I would instantly become an illegal immigrant. Deciding to take the gamble, we prepared the necessary documentation. To support our case, Jacob, my co-founder, had to write a letter stating that although my compensation was on the lower side, his, in the CEO role, was even lower. We also had to declare our company’s annual earnings ($0 at that time), and I suggested Jacob to specify it as less than $1 million. You should never lie to Homeland Security 😅. This leap of faith paid off; eight months later, we received our Green Card interview, where the immigration officer, making fun of our unique circumstances, granted approval on the spot. The Decision to Become American Those with an employment-based Green Card need a five-year stay in the US, compliant with all laws and tax requirements, to qualify for citizenship. While obtaining citizenship is not mandatory, and one might choose to stay a permanent resident indefinitely, citizenship confers full rights and responsibilities. Among these, the requirement to pay federal taxes forever, a big concern for many. If RevenueCat succeeds as we hope, I’m looking at significant tax payments, even if we end up moving back to Europe. The decision to embrace US citizenship came down to a simple reason: we can. We recognize the privilege of this choice, acknowledging the series of fortunate events that brought us here, aware that our journey could have taken decades had we originated from countries like India or China. Our twin daughters are blessed with dual citizenship, offering them a breadth of choices for their future. The opportunities the US has presented to our family are beyond what we once could dream. By becoming citizens, we gain a voice to influence immigration policies positively instead of blocking progress. My tax contributions have already reached the seven-figure mark. The continuation of our tax obligation is a small price to pay. And after all, there are some advantageous double taxation agreements 😉. Special thanks to my family for always supporting me and pushing me to live my dream in San Francisco. To Marina, my now wife, for joining me in this crazy adventure across the globe. To everyone at StepOne for running an amazing internship program. To Jesse, for believing in my potential and tackling the immigration challenges with me. And to my co-founder Jacob, for pushing me to take a leap of faith with my immigration status.
Since reading ‘High Growth Handbook’ by Elad Gil, the value of writing a ‘Working with’ document became crystal clear to me. I am sharing mine externally to inspire other founders and leaders to reflect and write down their own working styles. These documents are incredibly beneficial, especially in a multi-timezone, remote setting like we have at RevenueCat. I’ve spent some time fine-tuning mine, and this is the updated version. Welcome to your go-to manual for understanding how to collaborate effectively with me. My Mindset: Logic-Driven, Plan-Oriented I’m a logical thinker, much like a computer. If A implies B and we have A, I’ll typically conclude B. Sticking to plans and predictability is my comfort zone, yet I value reactivity, especially when customer-related issues arise and are solvable. This company isn’t just a job for me; it’s my life’s work. I’m deeply invested in everything here — our technology, culture, team, and customers. I get inspired and energized by hard-working coworkers who believe in our mission even more than me. As a co-founder, I can offer a wealth of institutional knowledge and guidance. While I may not have all the answers, I’m usually good at pointing you in the right direction. RevenueCat is only a sum of it’s parts. Our teammates drive our culture and I want to make sure we are building a place that people want to be. If you have a suggestion on how to make RevenueCat an even cooler place to work for our teammates I’m always here to talk about it. How We’ll Operate Regular Check-ins: For my direct reports, expect weekly or bi-weekly one-on-one meetings. To make our discussions more focused, I prefer that we establish an agenda before our scheduled time together. Communication Protocols: My schedule doesn’t allow much room for impromptu calls. If something urgent pops up, message me on Slack first. Should it require a call, schedule it through Susannah, please never bypass her. Meeting Preparation: Come to meetings with an agenda to ensure productivity. Without one, I might dominate the conversation, potentially missing your crucial points. Let’s both be responsible for following up on action items. Team Support: I’m open to joining other team meetings, but please share the agenda in advance and mark my attendance as optional unless crucial. Problem-Solving Approach: My engineering background means I love tackling complex problems using a divide and conquer approach: by breaking them down into smaller, manageable chunks, solving each piece, and then combining them for a final solution. If we can improve a completely broken system to 90% functionality, that’s significant progress in my book! Communication Style Note-Taking: While I take meticulous notes, my current preferred tool doesn’t support sharing. If you wish to access these notes, it’s on you to set up a shared document in Google Docs, Notion, or Lattice. Information Filtering: I prefer having complete transparency and the ability to filter out unnecessary details myself. Always explicitly state if you need input from me, or else I’ll assume it’s for my information only. Feedback Style: Expect direct feedback from me. I’ll clearly differentiate between areas for improvement and significant performance concerns. Trust Dynamics: Consider my trust like a metaphorical ‘bucket’ that starts half-full for everyone and adjusts based on your actions. The more you fill this bucket, the more autonomy you’ll have. For Managers Transparency in Challenges: Startups are always broken one way or another. I prefer to hear any bad news about a project or a team member directly from you. Working together through challenges can strengthen our trust and working relationship. Progress and Concerns: During our 1:1’s, I’ll inquire about your team dynamics and direct reports’ progress. I encourage you to include any details in our 1:1 agenda and lead the conversation to address any performance concerns, project delays or notable achievements . Feedback Dynamics: I recognize the weight of my title. To avoid unnecessary tension, I prefer to provide critical feedback about your reports directly to you so you can address privately. On the other hand, if there is any commendable achievement by your team I will do my best to praise publicly. If you feel there is someone on your team that I should connect with or praise, please let me know. Encouraging our team and recognizing their strengths is something that is very important to me. Preferences and Pet Peeves What I like Doing your homework: No question is stupid, but always do your initial research before distracting the team. Being resolutive: Getting things done, unblocking yourself. Readable and consistent code. Proven, boring technology over unproven open source projects that is trending on Hacker News. Proactivity: See a problem? Fix it right away before anyone notices. Made a mistake? Build systems to prevent anyone else making the same one again. Double checking your work: Give your work (documents, presentations, pull requests) a quick self-review before presenting it to the team. Transparency: In a multi-tz, remote environment, over-communication is better than miscommunication. Healthy discussions. When there is a decision to make that is not clear, it’s because all the different approaches have pros and cons. Together we will be able to calibrate and choose the lesser evil. A short call (or loom) is preferred over constant Slack interruptions. What I don’t like Gossip and rumors: They destroy the culture. Be upfront. Cargo cult: Let’s not do something just because BIG CO does it. That’s the beauty of building something from scratch. Unnecessary blockers. You’re all pretty smart here! Always try to unblock yourself first. Lack of context in questions, emails, and discussions. Not speaking up when something isn’t clear. Recurrent mistakes or questions: One time, it’s totally expected. Two times, hmm. Three times, nah. Learn, document, and build systems. Complaining without taking any action to improve the situation. Sarcasm or other ways of communication violence during disagreements: When somebody wins an argument, most of the time, the whole team loses. Acknowledging My Flaws Overcommitment: I tend to take on more than I should, which inevitably affects my focus. While I’m working on this, please understand if I occasionally get sidetracked by emergencies. Communication while Debugging: When addressing issues, I might share unvalidated hypotheses, which can be confusing. I’m learning to communicate more clearly and only after verifying my thoughts. Problem-Solving Obsession: Unsolved problems keep me up at night, which isn’t ideal for my well-being. It’s a habit I’m aware of and trying to balance. Pessimistic Tendencies: In evaluating problems, I often veer towards catastrophic thinking rather than optimism, a trait I’m mindful of and trying to moderate. Office Hours To maximize my availability given my tight schedule, I’ve introduced ‘office hours.’ This time is open for anyone to schedule a 15-minute chat with me about any concerns or ideas you might have. Reach out to Susannah for scheduling details. Thanks for sticking with me till the end! These are my personal preferences, not commandments carved in stone. I’m stoked to collaborate, build awesome stuff, and, above all, have fun together!
Another year as a founder CTO, and let me tell you, it’s been one for the books. I can’t remember a time in my life that was more demanding and emotionally draining. Those early years were filled with hard work, but we were also full of energy, ambition to build, and the sense that we had absolutely nothing to lose. As my dear friend Jacob used to quip, “Worst case scenario nobody dies”. Yet, everything takes on a new perspective when you’re responsible for the monetization infrastructure of over 30,000 apps and ensuring the livelihoods of not only your team but also your own family. While I would typically begin this series of posts with an array of metrics, this year has been a whirlwind of events that has taken precedence over mere numbers. So, dear friend, take a seat and allow me to share with you a deeply personal and epic firefighting tale and the battle to restore the flame of RevenueCat’s core values. The Year of Shipping We embarked on Q1 with big intentions and an ambitious roadmap. With seven new engineering teams, my role was to lead the one responsible for serving our larger customers. My days consisted of collaborating with our (still tiny) sales team, while simultaneously overseeing our support function. While I continued to provide technical guidance to several projects, I made the conscious decision to delegate the bulk of product development to the other teams. In hindsight, this decision proved to be a mistake. The teams were still finding their footing, with a mix of new hires, and there was a disconnect between engineering and product that needed addressing — a topic we’ll get into later. Nevertheless, the team I led was firing on all cylinders, delivering enterprise-level features at a hasty pace. From implementing single-sign-on in a matter of days, to tackling gnarly bugs, this team’s momentum was inspiring. I hoped that this stride would serve as a catalyst, motivating the rest of the teams to achieve a similar velocity. On the home front, my wife’s parental leave came to an end in January, marking the start of a new era where my hours were no longer flexible. Together, we embarked on the demanding task of caring for our twin babies. Little did I know how much the lack of sleep would impact me. Fatigue began to set in and my usual escape valves, such as surfing, turned into distant memories. Caffeine became my closest friend, helping me stay awake as I navigated the demands of work and family. I would spend my entire weekends lying in bed, trying to recover. I wrongly assumed it would be a short phase. Troubles never come alone On the fateful morning of March 9th, while I was soothing one of our crying babies at 4:30 AM, my WhatsApp began buzzing with messages from concerned fellow founders. Rumors were swirling that Silicon Valley Bank, the custodian of most VC-backed startup funds, was about to collapse. Within hours these whispers escalated into a full-blown bank run and we found ourselves unable to access our funds. Without going into the nitty-gritty details, it was an intensely stressful weekend. We were on the verge of a payroll crisis and we didn’t even have a functioning bank to transact with. However, we couldn’t afford to let this situation be what killed RevenueCat. We worked to expedite the opening of new bank accounts and explored multiple liquidity alternatives. Fortunately, we were privileged enough to entertain multiple options. One investor came to help, wiring funds from their personal account, and some of our loyal customers offered to pay in advance. Just as hope seemed to dwindle, the FDIC announced that they would guarantee all deposits on that Sunday afternoon. Crisis averted. Emotionally drained from the weekend’s trauma, we received another unexpected blow just a few days later: Y Combinator was discontinuing its Continuity Fund. This created a dilemma regarding the fate of their board seat, something we would have never anticipated. Just weeks prior, we had suffered a severe outage caused by our cloud provider. We tried to console ourselves, thinking, “At least all these problems are internal; we’re not dealing with downtime”. A Crisis of Reliability Our break was short-lived. Less than a week after the Silicon Valley Bank fiasco we experienced yet another major outage, and this time, it was self-inflicted. When you’re handling hundreds of thousands of API requests per second, even a few seconds of downtime can set off a cascade of alarms and potential sales losses. Our customers were understandably frustrated, and they didn’t hesitate to make their displeasure known through every available channel. It was heartbreaking to witness and I felt a deep sense of personal failure. This marked the third user-facing issue in a matter of months. It wasn’t something to sweep under the rug: our reputation was on the line and it was time to act. By this point, I was utterly sleep-deprived and my Apple Watch had begun sending me concerning health notifications. The day after we resolved the outage, my co-founder Jacob and I came up with an action plan. It was officially wartime. We needed to act decisively. We publicly unveiled our plan: to develop a robust fallback system independent of our current infrastructure. And we promised to deliver it within a week. Simultaneously, Jacob and I embarked on an apology tour, reaching out personally to some of our most worried customers. This was a painful but humbling experience for us, but it proved beneficial on multiple fronts. It allowed us to reiterate our commitment to becoming the best in-app subscription infrastructure provider in the world while gaining invaluable insights into our customers’ pain points. RC Fortress I rallied a small crew of engineers from different teams to build the first version of what we called “RevenueCat Fortress”. This component was designed to make sure end-customers could purchase seamlessly, even when our main servers were unavailable. It was a crazy week because we set ourselves a tight deadline but it helped boost our spirits and proved we could deliver software fast. The initial version of RevenueCat Fortress was quite simple – it operated behind the scenes on the server. But we didn’t stop there. We made it even better in the next iterations by adding SDK improvements such as offline entitlements. When it finally rolled out, it did so with flying colors. We even got to put it to the test during a major Apple outage and it saved the day for RevenueCat customers, making them immune to Apple’s downtime. Turning things around Looking back, the birth of RC Fortress marked the start of a shift in our culture. It got us back to the basics of reliability, fast delivery, and customer obsession. We couldn’t afford to spend months on extensive, untested projects. We had to rapidly build the features our customers valued most and iterate from there. We also realized that keeping things rock-solid wasn’t just the infrastructure team’s job; it was a global effort. Around the same time, we faced a couple of setbacks when we parted ways with two executives – the VP of People and the VP of Engineering. We tried to find a new VP of Engineering but couldn’t find a match that really excited us. So, the board agreed it would be best if I took the reins of the entire engineering organization again. Those days gave me a chance to get closer to the product teams again. Here’s where I spent most of my energy: Performance: I clarified expectations, provided feedback, and coached managers on performance management. Hiring: We tweaked our hiring process and re-calibrated interviewers’ expectations. Reliability and Quality: We were pretty good at doing post-mortems after issues but we had too many of them. They lacked detail and they weren’t followed through with action items. We needed a little bit of a cultural reset. We introduced dedicated incident Slack channels and clearly defined roles. Customer Obsession: Taking over the support team was eye-opening. It gave me a direct line to our product’s weak spots and what confused our customers. We started categorizing support tickets and sending them straight to the right product teams for triaging. Project Management: We focused on breaking projects into smaller chunks to deliver faster, instead of getting lost in never-ending projects. Education and Best Practices: I spent time educating other departments, especially post-sales teams, to avoid recurring mistakes that were slowing down our engineering progress. On top of all that, I reconnected with our customers more than ever. I hopped on planes to visit them at their offices and even worked our booth at a few conferences. It was a refreshing change to chat with users face-to-face, and hearing their unique challenges in person after a long time. Life is what happens when you’re busy running your startup We were fortunate enough to fly in my mother-in-law to assist with our babies, which made my travel plans possible. I finally felt more rested and even managed to squeeze in a few surfing sessions. Things were looking up both personally and professionally. I was eagerly anticipating our annual company-wide offsite, especially since I had missed most of the previous one due to my wife’s high-risk pregnancy. This time around I was geared up to address the entire engineering team, sharing the exciting changes and boosting morale. We were about to start winning again. But then, the day before my flight, I received a call from my father back in Spain. My mother had been rushed to the hospital, and she had been diagnosed with an extremely aggressive form of leukemia. Time seemed to stand still. I boarded the flight as planned, chugged two Red Bulls, and delivered that motivational talk to the whole team, while my mom was in a hospital bed thousands of miles away. This was hands down the toughest thing I’ve ever done as a startup founder. I left the offsite early and headed back to my hometown. Over the following weeks, I traveled back and forth around the globe, coordinating with my family. Sadly, my mother never left the hospital, she passed away merely a few weeks after the diagnosis. These are the things that always lurk in the back of your mind when you’re living 10,000 miles away from home, but you never truly believe they’ll happen. Until they do, and they shatter you. Keep on pushing The days that followed were far from easy. We were dealing with the launch of our biggest customer’s app, something we’d been preparing for a long time. The scale was enormous and the hard work of our team truly paid off. Our systems ran incredibly smoothly. It was a monumental victory, especially after the rocky start to the year. But, mentally, I wasn’t prepared to savor the moment. Yet, the energy post-offsite was infectious. People genuinely enjoyed meeting each other in real life and were fired up to start shipping. We couldn’t let this opportunity slip through our fingers. I paid homage to my mom’s teachings by continuing to press forward. RC Paywalls One of the major product ideas that had always been on our team’s wishlist was paywalls. We hadn’t tackled it because it seemed daunting and we lacked a product team with all the necessary skills. We couldn’t even estimate how long it would take to build. But, fueled by the success of RC Fortress, we decided to take a shot at it. We assembled a small team with members borrowed from different corners of the company. We didn’t mess with any reporting structures but appointed a leader. We went back to our roots, working in a hackathon-style frenzy for a couple of weeks to build a prototype. Just like in the good old days. And, boy, did the team rise to the occasion. We gave them some extra time and, in the end, they delivered one of our biggest product wins of the year. Shipping paywalls felt like a breath of fresh air and a clear sign that we still had our mojo. We were still capable of shipping software at lightning speed and keeping our customers excited. Based on all these lessons, the Head of Product and I started cooking up a brand new way of building products at RevenueCat. Engineering/Product/Design changes The complete process overhaul would warrant a couple of blog posts, but let me highlight the key changes to our workflow: We’ve reviewed all existing teams to determine whether they should continue as is, be replaced, or undergo changes in their mission or structure. We’ve rebalanced and clarified the responsibilities across Product, Engineering, and Design: Engineering takes the lead on feasibility, delivery, and developer experience, managing Linear, and overseeing technical architecture (and debt) roadmaps. Product is dedicated to customer value, business viability, and collaboration with sales and marketing teams. They also provide support to engineering in refining project scopes. Design is responsible for usability. We’ve formalized the role of Tech Lead, assigning them as the Directly Responsible Individuals for specific projects. They’re accountable for project success, with full backing from their Engineering Manager. It’s optional and project-dependent and doesn’t entail a title or salary change. We’ve acknowledged the need for engineering-driven initiatives, where PM and Design are involved on an as-needed basis. For projects with dependencies with other teams, we’ll designate a team member from the collaborating team as a formal interface. Our existing setup of stable product teams will remain the norm for most of our work. Temporary project teams will be established only when there’s a strong need for cross-team collaboration over a limited period. We’ll conduct monthly roadmap and shipping reviews with the founders and Head of Product. These reviews will provide insights into what we’re building, offer feedback opportunities, and help identify cross-functional dependencies and misalignments. 2024: The year of shipping + selling Collaborating with the Product team showcased the immense benefits of working closely together. Traditionally, Product reported directly to our CEO, which introduced unnecessary layers of indirection. In light of this and our recent addition of a VP of Sales, we decided it was time for a reorganization. Currently, Jacob (co-founder, and CEO), is overseeing Go to Market, People, and Operations, while I’m responsible for Engineering, Product, and Customer Engineering. We’ve brought in a VP of Customer Engineering, who reports to me and is in charge of Support and Technical Account Management. With our current headcount at 73 employees, my organization consists of 48 team members. Our executive team developed the most comprehensive planning effort to date. Our goal is to accelerate growth, focusing on sales and product delivery. We will avoid distractions by being extremely strategic at hiring. The past quarter showcased the strength of our engineering and product teams. New team members have been onboarded successfully, contributing meaningfully, and our management structure is finally robust. It took a bit of time for the “year of shipping” to fully materialize, nearly a year later, but customers have taken notice and we’re capitalizing on this momentum. On the enterprise sales front, I’m extremely bullish. We’ve secured the biggest deals in the company’s history. RevenueCat has evolved beyond being a product just for indie developers. However, we acknowledge the need to continue closing the product-market fit gap for enterprise clients. We’ve gained valuable insights into enterprise needs, and we’ll keep developing new products and features tailored to them. Indie developers will also take advantage of them to make more money. In 2024, the collaboration between our go-to-market and engineering teams will be critical. Highlights This year, as you’ve probably noticed, was a tough one. However, besides the challenges, there were several remarkable achievements to celebrate: We truly shipped. Our second hackathon, spanning an entire week, was an epic success. Many of the projects launched immediately, directly benefiting our customers. We had the privilege of collaborating with prominent brands and companies, including none other than Arnold Schwarzenegger himself. We created an astonishing amount of high-quality content. Our SubClub podcast outperformed all my expectations. During the chaos of the bank run, we discovered that one of our idols was not only a RevenueCat customer but also a devoted fan. We experimented with fresh team topologies and processes, and they turned out to be successful. We fine-tuned our vision for building a winning team, offering improved feedback, clearer expectations, and timely performance management. Jacob and I are no longer the sole authorities on Apple and Google subscriptions within the company. We successfully recruited seasoned executives to join our team. Our core infrastructure team accomplished monumental feats. We now support over 1 billion API requests daily, transitioned to our own data platform, and developed our own memcached client. All while maintaining flat costs despite the increase in load. Learnings It’s impossible to achieve peak performance without attention to health, exercise, and sleep. I’m no longer in my twenties. Complexity is the root of all evil. Startups and software are inherently complex, so avoid introducing unnecessary complexity. Begin by building the simplest feature or process, debug it, and then iterate as needed. Starting a business is tough, but launching a remote startup is an even greater challenge. Scaling a remote startup while parenting two under two is a herculean effort. Building a brand takes years but its reputation can be destroyed in an instant. Protect the integrity of your brand at all costs. Every new team member should add value, and especially so in a startup. Some provide immense leverage, while others become bottlenecks. The trickiest are those in the middle, who often end up becoming bottlenecks. Founders usually spot this within the first few weeks. Letting someone go is a taxing task. Even when managers believe it’s the right thing to do, it often requires a significant amount of support and guidance. Transparency in times of crisis pays dividends. Employees want to be treated like adults, and it builds trust. The same applies externally. Early worries often become baseless. By the time they become actual problems, your company might have died, you might have gained experience, or you might have hired the right talent to tackle them. SOC 2 auditors may request unconventional supporting evidence, such as employee performance reviews. Developers love socks. I’m so fortunate to have the world’s best co-founder. At this stage, my role is much more aligned with that of a founder than a traditional CTO or VP of Engineering. I continue to address issues as if they were technical problems, but my responsibilities extend well beyond the technology area. Life keeps moving forward, with its share of highs and lows. Life is too short, so you have to ensure the journey remains fun. For me, that means working with people who inspire me, and serving customers I genuinely care about. I really hope you enjoyed reading this post. I’m aware it’s a lot longer than my previous ones, but there were so many stories to share. As always, my intention was to share it with complete honesty and transparency, avoiding the hype that often surrounds startups. If you are facing similar challenges and want to connect and share experiences, please do not hesitate to reach out on Twitter or shoot me an email! Special thanks to my co-founder Jacob, the whole RevenueCat team, and our valuable customers. I also need to express my eternal gratitude to all the CTOs and leaders who have been kind enough to share their experiences over these years. Shoutout to Dani Lopez, Peter Silberman, Alex Plugaru, Kwindla Hultman Kramer, João Batalha, Karri Saarinen, Miguel Martinez Triviño, Sam Lown, Javi Santana, Pau Ramón, Javier Maestro, Matias Woloski, Tobias Balling, Jason Warner, and Will Larson. Our investors and early believers Jason Lemkin, Anu Hariharan, Mark Fiorentino, Mark Goldberg, Andrew Maguire, Gustaf Alströmer, and Nico Wittenborn. I want to convey my deep gratitude to my amazing wife, Marina, who has been my unwavering source of inspiration and support from the very beginning, and for blessing us with our two precious daughters. I cannot close this post without thanking my mom, who made countless sacrifices to mold me into the person I am today. I promise you will look down on us with pride the day we ring the bell in New York. I love you dearly.
More in programming
The concept of Product-Market Fit (PMF) collapse has gained renewed attention with the rise of large language models (LLMs), as highlighted in a recent Reforge article. The article argues we’re witnessing unprecedented market disruption, in this post, I propose we’re experiencing an acceleration of a familiar pattern rather than a fundamentally new phenomenon. Adoption Curves […] The post The Exodus Curve appeared first on Marc Astbury.
In 1940, President Roosevelt tapped William S. Knudsen to run the government's production of military equipment. Knudsen had spent a pivotal decade at Ford during the mass-production revolution, and was president of General Motors, when he was drafted as a civilian into service as a three-star general. Not bad for a Dane, born just ten minutes on bike from where I'm writing this in Copenhagen! Knudsen's leadership raised the productive capacity of the US war machine by a 100x in areas like plane production, where it went from producing 3,000 planes in 1939 to over 300,000 by 1945. He was quoted on his achievement: "We won because we smothered the enemy in an avalanche of production, the like of which he had never seen, nor dreamed possible". Knudsen wasn't an elected politician. He wasn't even a military man. But Roosevelt saw that this remarkable Dane had the skills needed to reform a puny war effort into one capable of winning the Second World War. Do you see where I'm going with this? Elon Musk is a modern day William S. Knudsen. Only even more accomplished in efficiency management, factory optimization, and first-order systems thinking. No, America isn't in a hot war with the Axis powers, but for the sake of the West, it damn well better be prepared for one in the future. Or better still, be so formidable that no other country or alliance would even think to start one. And this requires a strong, confident, and sound state with its affairs in order. If you look at the government budget alone, this is direly not so. The US was knocking on a two-trillion-dollar budget deficit in 2024! Adding to a towering debt that's now north of 36 trillion. A burden that's already consuming $881 billion in yearly interest payments. More than what's spent on the military or Medicare. Second to only Social Security on the list of line items. Clearly, this is not sustainable. This is the context of DOGE. The program, lead by Musk, that's been deputized by Trump to turn the ship around. History doesn't repeat, but it rhymes, and Musk is dropping beats that Knudsen would have surely been tapping his foot to. And just like Knudsen in his time, it's hard to think of any other American entrepreneur more qualified to tackle exactly this two-trillion dollar problem. It is through The Musk Algorithm that SpaceX lowered the cost of sending a kilo of goods into lower orbit from the US by well over a magnitude. And now America's share of worldwide space transit has risen from less than 30% in 2010 to about 85%. Thanks to reusable rockets and chopstick-catching landing towers. Thanks to Musk. Or to take a more earthly example with Twitter. Before Musk took over, Twitter had revenues of $5 billion and earned $682 million. After the take over, X has managed to earn $1.25 billion on $2.7 billion in revenue. Mostly thank to the fact that Musk cut 80% of the staff out of the operation, and savaged the cloud costs of running the service. This is not what people expected at the time of the take over! Not only did many commentators believe that Twitter was going to collapse from the drastic costs in staff, they also thought that the financing for the deal would implode. Chiefly as a result of advertisers withdrawing from the platform under intense media pressure. But that just didn't happen. Today, the debt used to take over Twitter and turn it into X is trading at 97 cents on the dollar. The business is twice as profitable as it was before, and arguably as influential as ever. All with just a fifth of the staff required to run it. Whatever you think of Musk and his personal tweets, it's impossible to deny what an insane achievement of efficiency this has been! These are just two examples of Musk's incredible ability to defy the odds and deliver the most unbelievable efficiency gains known to modern business records. And we haven't even talked about taking Tesla from producing 35,000 cars in 2014 to making 1.7 million in 2024. Or turning xAI into a major force in AI by assembling a 100,000 H100 cluster at "superhuman" pace. Who wouldn't want such a capacity involved in finding the waste, sloth, and squander in the US budget? Well, his political enemies, of course! And I get it. Musk's magic is balanced with mania and even a dash of madness. This is usually the case with truly extraordinary humans. The taller they stand, the longer the shadow. Expecting Musk to do what he does and then also be a "normal, chill dude" is delusional. But even so, I think it's completely fair to be put off by his tendency to fire tweets from the hip, opine on world affairs during all hours of the day, and offer his support to fringe characters in politics, business, and technology. I'd be surprised if even the most ardent Musk super fans don't wince a little every now and then at some of the antics. And yet, I don't have any trouble weighing those antics against the contributions he's made to mankind, and finding an easy and overwhelming balance in favor of his positive achievements. Musk is exactly the kind of formidable player you want on your team when you're down two trillion to nothing, needing a Hail Mary pass for the destiny of America, and eager to see the West win the future. He's a modern-day Knudsen on steroids (or Ketamine?). Let him cook.
Last week, James Truitt asked a question on Mastodon: James Truitt (he/him) @linguistory@code4lib.social Mastodon #digipres folks happen to have a handy repo of small invalid bags for testing purposes? I'm trying to automate our ingest process, and want to make sure I'm accounting for as many broken expectations as possible. Jan 31, 2025 at 07:49 PM The “bags” he’s referring to are BagIt bags. BagIt is an open format developed by the Library of Congress for packaging digital files. Bags include manifests and checksums that describe their contents, and they’re often used by libraries and archives to organise files before transfering them to permanent storage. Although I don’t use BagIt any more, I spent a lot of time working with it when I was a software developer at Wellcome Collection. We used BagIt as the packaging format for files saved to our cloud storage service, and we built a microservice very similar to what James is describing. The “bag verifier” would look for broken bags, and reject them before they were copied to long-term storage. I wrote a lot of bag verifier test cases to confirm that it would spot invalid or broken bags, and that it would give a useful error message when it did. All of the code for Wellcome’s storage service is shared on GitHub under an MIT license, including the bag verifier tests. They’re wrapped in a Scala test framework that might not be the easiest thing to read, so I’m going to describe the test cases in a more human-friendly way. Before diving into specific examples, it’s worth remembering: context is king. BagIt is described by RFC 8493, and you could create invalid bags by doing a line-by-line reading and deliberately ignoring every “MUST” or “SHOULD” but I wouldn’t recommend this aproach. You’d get a long list of test cases, but you’d be overwhelmed by examples, and you might miss specific requirements for your system. The BagIt RFC is written for the most general case, but if you’re actually building a storage service, you’ll have more concrete requirements and context. It’s helpful to look at that context, and how it affects the data you want to store. Who’s creating the bags? How will they name files? Where are you going to store bags? How do bags fit into your wider systems? And so on. Understanding your context will allow you to skip verification steps that you don’t need, and to add verification steps that are important to you. I doubt any two systems implement the exact same set of checks, because every system has different context. Here are examples of potential validation issues drawn from the BagIt specification and my real-world experience. You won’t need to check for everything on this list, and this list isn’t exhaustive – but it should help you think about bag validation in your own context. The Bag Declaration bagit.txt This file declares that this is a BagIt bag, and the version of BagIt you’re using (RFC 8493 §2.1.1). It looks the same in almost every bag, for example: BagIt-Version: 1.0 Tag-File-Character-Encoding: UTF-8 This tightly prescribed format means it can only be invalid in a few ways: What if the bag doesn’t have a bag declaration? It’s a required element of every BagIt bag; it has to be there. What if the bag declaration is the wrong format? It should contain exactly two lines: a version number and a character encoding, in that order. What if the bag declaration has an unexpected version number? If you see a BagIt version that you’ve not seen before, the bag might have a different structure than what you expect. The Payload Files and Payload Manifest The payload files are the actual content you want to save and preserve. They get saved in the payload directory data/ (RFC 8493 §2.1.2), and there’s a payload manifest manifest-algorithm.txt that lists them, along with their checksums (RFC 8493 §2.1.3). Here’s an example of a payload manifest with MD5 checksums: 37d0b74d5300cf839f706f70590194c3 data/waterfall.jpg This tells us that the bag contains a single file data/waterfall.jpg, and it has the MD5 checksum 37d0…. These checksums can be used to verify that the files have transferred correctly, and haven’t been corrupted in the process. There are lots of ways a payload manifest could be invalid: What if the bag doesn’t have a payload manifest? Every BagIt bag must have at least one Payload Manifest file. What if the payload manifest is the wrong format? These files have a prescribed format – one file per line, with a checksum and file path. What if the payload manifest refers to a file that isn’t in the bag? Either one of the files in the bag has been deleted, or the manifest has an erroneous entry. What if the bag has a file that isn’t listed in the payload manifest? The manifest should be a complete listing of all the payload files in the bag. If the bag has a file which isn’t in the payload manifest, either that file isn’t meant to be there, or the manifest is missing an entry. Checking for unlisted files is how I spotted unwanted .DS_Store and Thumbs.db files. What if the checksum in the payload manifest doesn’t match the checksum of the file? Either the file has been corrupted, or the checksum is incorrect. What if there are payload files outside the data/ directory? All the payload files should be stored in data/. Anything outside that is an error. What if there are duplicate entries in the payload manifest? Every payload file must be listed exactly once in the manifest. This avoids ambiguity – suppose a file is listed twice, with two different checksums. Is the bag valid if one of those checksums is correct? Requiring unique entries avoids this sort of issue. What if the payload directory is empty? This is perfectly acceptable in the BagIt RFC, but it may not be what you want. If you know that you will always be sending bags that contain files, you should flag empty payload directories as an error. What if the payload manifest contains paths outside data/, or relative paths that try to escape the bag? (e.g. ../file.txt) Now we’re into “malicious bag” territory – a bag uploaded by somebody who’s trying to compromise your ingest pipeline. Any such bags should be treated with suspicion and rejected. If you’re concerned about malicious bags, you need a more thorough test suite to catch other shenanigans. We never went this far at Wellcome Collection, because we didn’t ingest bags from arbitrary sources. The bags only came from internal systems, and our verification was mainly about spotting bugs in those systems, not defending against malicious actors. A bag can contain multiple payload manifests – for example, it might contain both MD5 and SHA1 checksums. Every payload manifest must be valid for the overall bag to be valid. Payload filenames There are lots of gotchas around filenames and paths. It’s a complicated problem, and I definitely don’t understand all of it. It’s worth understanding the filename rules of any filesystem where you will be storing bags. For example, Azure Blob Storage has a number of rules around how you can name files, and Amazon S3 has different rules. We stored files in both at Wellcome Collection, and so the storage service had to enforce the superset of these rules. I’ve listed some edge cases of filenames you might want to consider, but it’s not a comlpete list. There are lots of ways that unexpected filenames could cause you issues, but whether you care depends on the source of your bags. If you control the bags and you know you’re not going to include any weird filenames, you can probably skip most of these. We only checked for one of these conditions at Wellcome Collection, because we had a pre-ingest step that normalised filenames. It converted filenames to ASCII, and saved a mapping between original and normalised filename in the bag. However, the normalisation was only designed for one filesystem, and produced filenames with trailing dots that were still disallowed in Azure Blob. What if a filename is too long? Some systems have a maximum path length, and an excessively deep directory structure or long filename could cause issues. What if a filename contains special characters? Spaces, emoji, or special characters (\, :, *, etc.) can cause problems for some tools. You should also think about characters that need to be URL-encoded. What if a filename has trailing spaces or dots? Some filesystems can’t support filenames ending in a dot or a space. What happens if your bag contains such a file, and you try to save it to the filesystem? This caused us issues at Wellcome Collection. We initially stored bags just in Amazon S3, which is happy to take filenames with a trailing dot – then we added backups to Azure Blob, which doesn’t. One of the bags we’d stored in Amazon S3 had a trailing dot in the filename, and caused us headaches when we tried to copy it to Azure. What if a filename contains a mix of path separators? The payload manifest uses a forward slash (/) as a path separator. If you have a filename with an alternative path separator, it might behave differently on different systems. For example, consider the payload file a\b\c. This would be a single file on macOS or Linux, but it would be nested inside two folders on Windows. What if the filenames are a mix of uppercase and lowercase characters? Some fileystems are case-sensitive, others aren’t. This can cause issues when you move bags between systems. For example, suppose a bag contains two different files Macrodata.txt and macrodata.txt. When you save that bag on a case-insensitive filesystem, only one file will be saved. What if the same filename appears twice with different Unicode normalisations? This is similar to filenames which only differ in upper/lowercase. They might be treated as two files on one filesystem, but collapsed into one file on another. The classic example is the word “café”: this can be encoded as caf\xc3\xa9 (UTF-8 encoded é) or cafe\xcc\x81 (e + combining acute accent). What if a filename contains a directory reference? A directory reference is /./ (current directory) or /../ (parent directory). It’s used on both Unix and Windows-like systems, and it’s another case of two filenames that look different but can resolve to the same path. For example: a/b, a/./b and a/subdir/../b all resolve to the same path under these rules. This can cause particular issues if you’re moving between local filesystems and cloud storage. Local filesystems treat filenames as hierarchical paths, where cloud storage like Amazon S3 often treats them as opaque strings. This can cause issues if you try to copy files from cloud storage to a local system – if you’re not careful, you could lose files in the process. The Tag Manifest tagmanifest-algorithm.txt Similar to the payload manifest, the tag manifest lists the tag files and their checksums. A “tag file” is the BagIt term for any metadata file that isn’t part of the payload (RFC 8493 §2.2.1). Unlike the payload manifest, the tag manifest is optional. A bag without a tag manifest can still be a valid bag. If the tag manifest is present, then many of the ways that a payload manifest can invalidate a bag – malformed contents, unreferenced files, or incorrect checksums – can also apply to tag manifests. There are some additional things to consider: What if a tag manifest lists payload files? The tag manifest lists tag files; the payload manifest lists payload files in the data/ directory. A tag manifest that lists files in the data/ directory is incorrect. What if the bag has a file that isn’t listed in either manifest? Every file in a bag (except the tag manifests) should be listed in either a payload or a tag manifest. A file that appears in neither could mean an unexpected file, or a missing manifest entry. Although the tag manifest is optional in the BagIt spec, at Wellcome Collection we made it a required file. Every bag had to have at least one tag manifest file, or our storage service would refuse to ingest it. The Bag Metadata bag-info.txt This is an optional metadata file that describes the bag and its contents (RFC 8493 §2.2.2). It’s a list of metadata elements, as simple label-value pairs, one per line. Here’s an example of a bag metadata file: Source-Organization: Lumon Industries Organization-Address: 100 Main Street, Kier, PE, 07043 Contact-Name: Harmony Cobel Unlike the manifest files, this is primarily intended for human readers. You can put arbitrary metadata in here, so you can add fields specific to your organisation. Although this file is more flexible, there are still ways it can be invalid: What if the bag metadata is the wrong format? It should have one metadata entry per line, with a label-value pair that’s separated by a colon. What if the Payload-Oxum is incorrect? The Payload-Oxum contains some concise statistics about the payload files: their total size in bytes, and how many there are. For example: Payload-Oxum: 517114.42 This tells us that the bag contains 42 payload files, and their total size is 517,114 bytes. If these stats don’t match the rest of the bag, something is wrong. What if non-repeatable metadata element names are repeated? The BagIt RFC defines a small number of reserved metadata element names which have a standard meaning. Although most metadata element names can be repeated, there are some which can’t, because they can only have one value. In particular: Bagging-Date, Bag-Size, Payload-Oxum and Bag-Group-Identifier. Although the bag metadata file is optional in a general BagIt bag, you may want to add your own rules based on how you use it. For example, at Wellcome Collection, we required all bags to have an External-Identifier value, that matched a specific schema. This allowed us to link bags to records in other databases, and our bag verifier would reject bags that didn’t include it. The Fetch File fetch.txt This is an optional element that allows you to reference files stored elsewhere (RFC 8493 §2.2.3). It tells the person reading the bag that a file hasn’t been included in this copy of the bag; they have to go and fetch it from somewhere else. The file is still recorded in the payload manifest (with a checksum you can verify), but you don’t have a complete bag until you’ve downloaded all the files. Here’s an example of a fetch.txt: https://topekastar.com/~daria/article.txt 1841 data/article.txt This tells us that data/article.txt isn’t included in this copy of the bag, but we we can download it from https://topekastar.com/~daria/article.txt. (The number 1841 is the size of the file in bytes. It’s optional.) Using fetch.txt allows you to send a bag with “holes”, which saves disk space and network bandwidth, but at a cost – we’re now relying on the remote location to remain available. From a preservation standpoint, this is scary! If topekastar.com goes away, this bag will be broken. I know some people don’t use fetch.txt for precisely this reason. If you do use fetch.txt, here are some things to consider: What if the fetch file is the wrong format? There’s a prescribed format – one file per line, with a URL, optional file size, and file path. What if the fetch file lists a file which isn’t in the payload manifest? The fetch.txt should only tell us that a file is stored elsewhere, and shouldn’t be introducing otherwise unreferenced files. If a file appears in fetch.txt but not the payload manifest, then we can’t verify the remote file because we don’t have a checksum for it. There’s either an erroneous fetch file entry or a missing manifest entry. What if the fetch file points to a file at an unusable URL? The URL is only useful if the person who receives the bag can use it to download the file. If they can’t, the bag might technically be valid, but it’s functionally broken. For example, you might reject URLs that don’t start with http:// or https://. What if the fetch file points to a file with the wrong length? The fetch.txt can optionally specify the size of a file, so you know how much storage you need to download it. If you download the file, the actual size should match the stated size. What if the fetch files points to a file that’s already included in the bag? Now you have two ways to get this file: you can read it from the bag, or from the remote URL. If a file is listed in both fetch.txt and included in the bag, either that file isn’t meant to be in the bag, or the fetch file has an erroneous entry. We used fetch files at Wellcome Collection to implement versioning, and we added extra rules about what remote URLs were allowed. In particular, we didn’t allow fetching a file from just anywhere – you could fetch from our S3 buckets, but not the general Internet. The bag verifier would reject a fetch file entry that pointed elsewhere. These examples illustrate just how many ways a BagIt bag can be invalid, from simple structural issues to complex edge cases. Remember: the key is to understand your specific needs and requirements. By considering your context – who creates your bags, where they’ll be stored, and how they fit into your wider systems – you can build a validation process to catch the issues that matter to you, while avoiding unnecessary complexity. I can give you my ideas, but only you can build your system. [If the formatting of this post looks odd in your feed reader, visit the original article]
We bought sixty-one servers for the launch of Basecamp 3 back in 2015. Dell R430s and R630s, packing thousands of cores and terabytes of RAM. Enough to fill all the app, job, cache, and database duties we needed. The entire outlay for this fleet was about half a million dollars, and it's only now, almost a decade later, that we're finally retiring the bulk of them for a full hardware refresh. What a bargain! That's over 3,500 days of service from this fleet, at a fully amortized cost of just $142/day. For everything needed to run Basecamp. A software service that has grossed hundreds of millions of dollars in that decade. We've of course had other expenses beyond hardware from operating Basecamp over the past decade. The ops team, the bandwidth, the power, and the cabinet rental across both our data centers. But none the less, owning our own iron has been a fantastically profitable proposition. Millions of dollars saved over renting in the cloud. And we aren't even done deriving value from this venerable fleet! The database servers, Dell R630s w/ Xeon E5-2699 CPUs and 768G of RAM, are getting handed down to some of our heritage apps. They will keep on trucking until they give up the ghost. When we did the public accounting for our cloud exit, it was based on five years of useful life from the hardware. But as this example shows, that's pretty conservative. Most servers can easily power your applications much longer than that. Owning your own servers has easily been one of our most effective cost advantages. Together with running a lean team. And managing our costs remains key to reaping the profitable fruit from the business. The dollar you keep at the end of the year is just as real whether you earn it or save it. So you just might want to run those cloud-exit numbers once more with a longer server lifetime value. It might just tip the equation, and motivate you to become a server owner rather than a renter.
At some point in a startup’s lifecycle, they decide that they need to be ready to go public in 18 months, and a flurry of IPO-readiness activity kicks off. This strategy focuses on a company working on IPO readiness, which has identified a gap in their internal controls for managing access to their users’ data. It’s a company that wants to meaningfully improve their security posture around user data access, but which has had a number of failed security initiatives over the years. Most of those initiatives have failed because they significantly degraded internal workflows for teams like customer support, such that the initial progress was reverted and subverted over time, to little long-term effect. This strategy represents the Chief Information Security Officer’s (CISO) attempt to acknowledge and overcome those historical challenges while meeting their IPO readiness obligations, and–most importantly–doing right by their users. This is an exploratory, draft chapter for a book on engineering strategy that I’m brainstorming in #eng-strategy-book. As such, some of the links go to other draft chapters, both published drafts and very early, unpublished drafts. Reading this document To apply this strategy, start at the top with Policy. To understand the thinking behind this strategy, read sections in reverse order, starting with Explore, then Diagnose and so on. Relative to the default structure, this document has been refactored in two ways to improve readability: first, Operation has been folded into Policy; second, Refine has been embedded in Diagnose. More detail on this structure in Making a readable Engineering Strategy document. Policy & Operations Our new policies, and the mechanisms to operate them are: Controls for accessing user data must be significantly stronger prior to our IPO. Senior leadership, legal, compliance and security have decided that we are not comfortable accepting the status quo of our user data access controls as a public company, and must meaningfully improve the quality of resource-level access controls as part of our pre-IPO readiness efforts. Our Security team is accountable for the exact mechanisms and approach to addressing this risk. We will continue to prioritize a hybrid solution to resource-access controls. This has been our approach thus far, and the fastest available option. Directly expose the log of our resource-level accesses to our users. We will build towards a user-accessible log of all company accesses of user data, and ensure we are comfortable explaining each and every access. In addition, it means that each rationale for access must be comprehensible and reasonable from a user perspective. This is important because it aligns our approach with our users’ perspectives. They will be able to evaluate how we access their data, and make decisions about continuing to use our product based on whether they agree with our use. Good security discussions don’t frame decisions as a compromise between security and usability. We will pursue multi-dimensional tradeoffs to simultaneously improve security and efficiency. Whenever we frame a discussion on trading off between security and utility, it’s a sign that we are having the wrong discussion, and that we should rethink our approach. We will prioritize mechanisms that can both automatically authorize and automatically document the rationale for accesses to customer data. The most obvious example of this is automatically granting access to a customer support agent for users who have an open support ticket assigned to that agent. (And removing that access when that ticket is reassigned or resolved.) Measure progress on percentage of customer data access requests justified by a user-comprehensible, automated rationale. This will anchor our approach on simultaneously improving the security of user data and the usability of our colleagues’ internal tools. If we only expand requirements for accessing customer data, we won’t view this as progress because it’s not automated (and consequently is likely to encourage workarounds as teams try to solve problems quickly). Similarly, if we only improve usability, charts won’t represent this as progress, because we won’t have increased the number of supported requests. As part of this effort, we will create a private channel where the security and compliance team has visibility into all manual rationales for user-data access, and will directly message the manager of any individual who relies on a manual justification for accessing user data. Expire unused roles to move towards principle of least privilege. Today we have a number of roles granted in our role-based access control (RBAC) system to users who do not use the granted permissions. To address that issue, we will automatically remove roles from colleagues after 90 days of not using the role’s permissions. Engineers in an active on-call rotation are the exception to this automated permission pruning. Weekly reviews until we see progress; monthly access reviews in perpetuity. Starting now, there will be a weekly sync between the security engineering team, teams working on customer data access initiatives, and the CISO. This meeting will focus on rapid iteration and problem solving. This is explicitly a forum for ongoing strategy testing, with CISO serving as the meeting’s sponsor, and their Principal Security Engineer serving as the meeting’s guide. It will continue until we have clarity on the path to 100% coverage of user-comprehensible, automated rationales for access to customer data. Separately, we are also starting a monthly review of sampled accesses to customer data to ensure the proper usage and function of the rationale-creation mechanisms we build. This meeting’s goal is to review access rationales for quality and appropriateness, both by reviewing sampled rationales in the short-term, and identifying more automated mechanisms for identifying high-risk accesses to review in the future. Exceptions must be granted in writing by CISO. While our overarching Engineering Strategy states that we follow an advisory architecture process as described in Facilitating Software Architecture, the customer data access policy is an exception and must be explicitly approved, with documentation, by the CISO. Start that process in the #ciso channel. Diagnose We have a strong baseline of role-based access controls (RBAC) and audit logging. However, we have limited mechanisms for ensuring assigned roles follow the principle of least privilege. This is particularly true in cases where individuals change teams or roles over the course of their tenure at the company: some individuals have collected numerous unused roles over five-plus years at the company. Similarly, our audit logs are durable and pervasive, but we have limited proactive mechanisms for identifying anomalous usage. Instead they are typically used to understand what occurred after an incident is identified by other mechanisms. For resource-level access controls, we rely on a hybrid approach between a 3rd-party platform for incoming user requests, and approval mechanisms within our own product. Providing a rationale for access across these two systems requires manual work, and those rationales are later manually reviewed for appropriateness in a batch fashion. There are two major ongoing problems with our current approach to resource-level access controls. First, the teams making requests view them as a burdensome obligation without much benefit to them or on behalf of the user. Second, because the rationale review steps are manual, there is no verifiable evidence of the quality of the review. We’ve found no evidence of misuse of user data. When colleagues do access user data, we have uniformly and consistently found that there is a clear, and reasonable rationale for that access. For example, a ticket in the user support system where the user has raised an issue. However, the quality of our documented rationales is consistently low because it depends on busy people manually copying over significant information many times a day. Because the rationales are of low quality, the verification of these rationales is somewhat arbitrary. From a literal compliance perspective, we do provide rationales and auditing of these rationales, but it’s unclear if the majority of these audits increase the security of our users’ data. Historically, we’ve made significant security investments that caused temporary spikes in our security posture. However, looking at those initiatives a year later, in many cases we see a pattern of increased scrutiny, followed by a gradual repeal or avoidance of the new mechanisms. We have found that most of them involved increased friction for essential work performed by other internal teams. In the natural order of performing work, those teams would subtly subvert the improvements because it interfered with their immediate goals (e.g. supporting customer requests). As such, we have high conviction from our track record that our historical approach can create optical wins internally. We have limited conviction that it can create long-term improvements outside of significant, unlikely internal changes (e.g. colleagues are markedly less busy a year from now than they are today). It seems likely we need a new approach to meaningfully shift our stance on these kinds of problems. Explore Our experience is that best practices around managing internal access to user data are widely available through our networks, and otherwise hard to find. The exact rationale for this is hard to determine, but it seems possible that it’s a topic that folks are generally uncomfortable discussing in public on account of potential future liability and compliance issues. In our exploration, we found two standardized dimensions (role-based access controls, audit logs), and one highly divergent dimension (resource-specific access controls): Role-based access controls (RBAC) are a highly standardized approach at this point. The core premise is that users are mapped to one or more roles, and each role is granted a certain set of permissions. For example, a role representing the customer support agent might be granted permission to deactivate an account, whereas a role representing the sales engineer might be able to configure a new account. Audit logs are similarly standardized. All access and mutation of resources should be tied in a durable log to the human who performed the action. These logs should be accumulated in a centralized, queryable solution. One of the core challenges is determining how to utilize these logs proactively to detect issues rather than reactively when an issue has already been flagged. Resource-level access controls are significantly less standardized than RBAC or audit logs. We found three distinct patterns adopted by companies, with little consistency across companies on which is adopted. Those three patterns for resource-level access control were: 3rd-party enrichment where access to resources is managed in a 3rd-party system such as Zendesk. This requires enriching objects within those systems with data and metadata from the product(s) where those objects live. It also requires implementing actions on the platform, such as archiving or configuration, allowing them to live entirely in that platform’s permission structure. The downside of this approach is tight coupling with the platform vendor, any limitations inherent to that platform, and the overhead of maintaining engineering teams familiar with both your internal technology stack and the platform vendor’s technology stack. 1st-party tool implementation where all activity, including creation and management of user issues, is managed within the core product itself. This pattern is most common in earlier stage companies or companies whose customer support leadership “grew up” within the organization without much exposure to the approach taken by peer companies. The advantage of this approach is that there is a single, tightly integrated and infinitely extensible platform for managing interactions. The downside is that you have to build and maintain all of that work internally rather than pushing it to a vendor that ought to be able to invest more heavily into their tooling. Hybrid solutions where a 3rd-party platform is used for most actions, and is further used to permit resource-level access within the 1st-party system. For example, you might be able to access a user’s data only while there is an open ticket created by that user, and assigned to you, in the 3rd-party platform. The advantage of this approach is that it allows supporting complex workflows that don’t fit within the platform’s limitations, and allows you to avoid complex coupling between your product and the vendor platform. Generally, our experience is that all companies implement RBAC, audit logs, and one of the resource-level access control mechanisms. Most companies pursue either 3rd-party enrichment with a sizable, long-standing team owning the platform implementation, or rely on a hybrid solution where they are able to avoid a long-standing dedicated team by lumping that work into existing teams.