Full Width [alt+shift+f] Shortcuts [alt+shift+k]
Sign Up [alt+shift+s] Log In [alt+shift+l]
17
IE11 is not dead yet, and our library is supposed to run there and make russian grandmas happy. As you can guess, we rely on babel's preset-env a lot. We also don't want our code to be 55% babel helpers, so we use babel's transform-runtime — it should make babel import someHelper from '@babel/runtime/some-helper' instead of inlining it into every file. After making some build chain updates I went to see if the transpiled version was OK. And guess what I noticed? Some babel helpers were still there, inlined: import _defineProperty from "@babel/runtime/helpers/defineProperty"; // go away transform do you have any idea who I am? function ownKeys(object, enumerableOnly) { /* blah-blah-blah */ } function _objectSpread(target) { /* more blah-blah-blah */ } var copy = function copy(obj) { return _objectSpread({}, options); }; WTF? I want to import _objectSpread, you lazy code! What's wrong with you? A leak from an external library? An unexpected interaction with preset-react or...
over a year ago

Improve your reading experience

Logged in users get linked directly to articles resulting in a better reading experience. Please login for free, it takes less than 1 minute.

More from Vladimir Klepov as a Coder

Growing my team 4x has been a pain. Can we do better?

My name is Vladimir, and I'm an engineering manager of a team building a banking app. Following the success of our core banking product, we've decided to expand to other financial services. In the last four months my team has grown 4x, going from a 4-person team to 4 teams totalling 15 people. It was, frankly, a shitshow, and now I see many things we could have done better. At first sight, increasing your team is a perfect way to speed up your product development. In practice, scaling is very challenging. Today, I'll share the pains of growth we've ran into: Disbalanced growth across functions. Teams becoming too large to manage. Processes breaking down. Mass onboarding challenges. Seniority skew. Now that it's all done, I can't say these problems were unexpected, or that our solutions have been incredibly inventive, but a first-hand account is worth writing down. Whether you're a leader expecting your team to scale, or just curious about the daily challenges of an engineering manager, hope you'll find something for yourself. Let's go! Scale evenly across functions You lead a 5-man engineering team. You'd like to build 2x the stuff you're building. The obvious solution is to hire 5 more engineers. Problem solved? Not so fast. Anything you build flows across several stages, each handled by a certain role, for example: product manager -> design -> engineering -> QA -> product analytics -> (back to) product manager. If you have a balanced flow with your current team, enlarging just the engineering will not make you more productive: Downstream functions (QA / analytics) can't keep up with the increased production — they must either put in overtime or downgrade quality standards. Upstream functions (product / design) can't fill the backlog fast enough, and the eng team has nothing to focus on, either slacking or refactoring the refactorings. With rapid growth, some temporary disbalances are unavoidable. It's fine to spend a few weeks or months in a disproportionate state, but overall aim for balanced scaling. Here are some tips to smooth the transition and even use the disbalance to your advantage: If you have any control over it, give product & design a hiring head-start. You can get their artifacts ready for development with a limited series of grooming meetings, and once the new engineers are on board, you'll have some great useful tasks to feed them. Catch up on your tech debt. Have too many engineers and not enough product tasks? Don't despair: use the time to clean up some old bugs, do the overdue refactorings, and prepare the codebase for the speed-up. Expand the area of responsibility. If you lack QA specialists, it might be time for the engineers to practice their testing skills. Oversized non-technical component (product / design) is more problematic, but you can give them some no-code tools to replace the eng team in some scenarios — e.g. build an admin UI where PM can edit the texts, create new banners, and so on, without involving your team. Update team structure Say you have a normal-sized team (4–7 engineers) with your average meeting structure (whole-team planning, grooming, and retro + weekly 1x1 with every team member). Making it a 15-person team won't work at all. An hour-long 4-person retro has 10 minutes of speaking time per member — enough to make a point. In a 15-person retro, it's 3 minutes — not practical. You either exclude some people, or extend the meetings — both poor options. It's harder to agree on any decision, because you now have 3x the possible objections. The 1x1s alone eat up 7 hours (almost a full day!) of your time a week. Managing communication of 15 people plus all the external stakeholders is time-consuming. The team must be split. What does it mean to be a separate team, anyways? Ever heard of "high cohesion and low coupling" principle in software architecture? I think this also applies to teams: Shared information space. Team members know what you're working on, where you're going, who your peers are, the system structure and so on. Own a well-defined part of the product and the codebase. A look at a random feature is enough to guess the responsible team. Control your processes — meeting structure, releases, etc. You can't be responsible for what you can't control. Have all the capabilities needed for your day-to-day work. Begging someone each time you need to deploy, change the API or add a banner is not very effective. It's best to split by product domain: customer acquisition team owns the landing page and signup, daily banking team owns the main app, and so on. You could split by layer (product + infra) or by function (backend + frontend + mobile) — I feel these compromise points 2 and 4, but let's not die on this hill today. At any rate, a team over 8–10 people must be split into sane-sized chunks to keep going. A note on grouping You can split a team by (A) building the new team out of newbies or (B) mixing newbies and oldies in each new part. Prefer mixing: it distributes the knowledge across the organization, and the social connections from the original team prevent siloing. You could argue that (A) keeps the original high-performing team intact, but it does so by slowing down the new team and undermining your long-term flexibility. Where to get new leaders? Splitting a team into 3 parts creates 2–4 leadership positions, depending on your place in the new structure. Ideally, you have senior members of the original team to lead the new teams, because it's a rare opportunity for career growth on a management track, and they can easily hire and onboard new members of their teams. If you don't have a suitable candidate (everyone is either very junior, or hates management), it's fine to hire externally — following a few hiccups, I recommend hiring people with prior leadership experience, because adjusting to a new product and a new role at once can be too much. Split iteratively You don't have to produce a fully separate team right away. As usual, move step by step — you get faster results, and can adapt to the issues that arise. Here's one possible sequence: Assemble a domain team, appoint one as the lead (you can call it trial-lead, to give them a chance to cop out). See how they like their new roles, and if the headcount needs tweaking. Run a retro for the new team to catch communication issues or cross-team dependencies early. Separate the domain backlog and kanban boards (or wherever you track the tasks). You'll need it for further process splitting, and to assess the team's load and velocity. Split planning and daily meetings, so that the teams don't waste each other's time on discussing irrelevant tasks. Gradually transfer the remaining processes (1x1s, onboarding, postmortems) to the new lead. Split the codebase, so that the new team can fully own its service. Update your processes Just like your single-team structure, the processes you have will likely fail for a larger team, especially if you have many newbies. Example: we had a liberal release process — if you want some feature in production, you deploy it. As the team grew, the release frequency dropped — the newbies were afraid to touch prod, the oldies were hoping one of the other 14 people would do it. Before I give you my solution to this puzzle, let's look at the general advice for process scaling. Localize processes to the new teams. Owning processes makes a team more effective. Retros, plannings, kickoffs, demos, daily stand-ups, releases, documenting, on-call duty, whatever you can split, do split. Yes, your overall team loses sight of the stuff going on across the system, causing duplication and poor decisions, but in return we can focus on a specific business area, and maximize productive time instead of drowning in discussions. If you need to offset the downsides, introduce a cross-team sync here and there (still working on this one). Let the new teams experiment with processes. What worked for your original team doesn't matter, that team is dead and gone. What works for one of your teams won't necessarily work for the other, because they work in different conditions. For example, our CA team has many time-bound tasks from marketing. The core banking team focuses on building quality software, and fixing the bugs as they arise. Very different teams. Start with a copy of your current processes (just to start somewhere!), and introduce team retros as early as possible to tweak the process as needed. Stricten the centralized processes. Back to our problem with deployment — we couldn't isolate the release process to sub-teams, because splitting a monolithic front-end into independently deployable parts is technically challenging. We introduced a more structured release process: The releases are automatically built and ready to deploy every morning. No more decisions to make. Daily rotation of release managers responsible for getting the release to production. With regular training, you get better at releasing. The release process is clearly documented. The newbies have a clear path to follow, making it less stressful. This works for other centralized processes — writing documentation, debugging with customer support, maintaining shared libs. Overall: hand over as much process as you can to the sub-teams, and introduce clear rules for the remaining centralized processes. The onboarding valley Surprisingly, fast hiring can reduce your team's productivity in the short term. The newbies are not yet up to speed, and the oldies now spend time explaining your codebase and reviewing code. This will fix itself over time, but here are a few strategies to get past the bump faster. Prefer slower growth. Adding one person every few weeks is much better than adding 6 people simultaneously, because: The "onboarding load" stretched over time occupies a smaller share of your team's resource. A few weeks in, new hires can already help onboard someone else. In some cases, they'll do a better job than any oldie, because their memories of one-off tasks like setting up the dev environment are fresh. Every onboarding exposes new roadblocks in your process, helping you smooth the next ones. On paper, batch onboarding might seem like a time-saver, as you can make a lecture explaining the basics to many people at once. In practice, unless your product is very small, or the tasks are very repetitive, every newbie faces very different challenges, drowning you with a wave of questions. Encourage peer-to-peer onboarding. As a leader, you might think onboarding is your personal responsibility. I call BS — peer-to-peer onboarding is clearly better: More "onboarding resource" leads to faster, better onboarding. The load on you, personally, decreases, freeing time to do other impactful things and, you know, live. Team members get a safe environment to practice their mentorship skills. People get to know each other, instead of only talking to you. ICs with recent hands-on experience do a better job at explaining the specifics than you. You can pair a formal "mentor" to every newbie, or direct questions to a team group chat. If you want to control the overall onboarding, at least route specific questions to team members experienced in that area instead of trying to come up with all the answers yourself. Write the docs. The best way to make onboardings cheaper is writing stuff down instead of explaining it over and over again, with your mouth. Some particular things to focus on: Onboarding checklist — the things every new team member must do: get a VPN certificate from the security dude, join this and that chat, clone a repo here and there, boom you're done. Only include essential steps — adding somewhat useful stuff obscures the actually important things. Document your existing business processes, system architecture, technical conventions, team and communication structure. It's better than explaining in real-time, because you get higher-quality charts, relevant links, and you can collaborate to put the knowledge of multiple team members in one place. Tooling and automation. The more automated a process or convention is, the less onboarding you need. Example: if you build your releases locally and upload somewhere via FTP using the keys you get from Piotr the devops, it's time to set up decent CD instead of documenting the current state of affairs. Pro tip: encourage newbies to improve and update the docs as they follow along — it's a great first contribution to your team! Senior vs junior hires It seems sensible to focus exclusively on senior hires. Experienced engineers get up to speed quicker, because they're already familiar with the basics, have a lower risk of making catastrophically poor decisions, and can bring good practices and ideas from across the industry to your team. Not so fast — here are some reasons to hire junior developers. With little exposure to the industry, they can easily adapt to whatever culture and processes you have. Anyone on your team can mentor a junior hire, while getting e.g. a junior engineer with decent knowledge of the product to mentor a newly hired senior engineer might be awkward, not very productive, or even taken as an insult. And of course, you can hire more junior engineers on the same budget. Overall, aim for a balanced team composition in the mid-term. You don't want your team to be a kindergarten, but a nursing home is no good either. Remember that people tend to gain experience, so the junior engineers you hire will become middle in no time. Today, we've discussed the challenges of rapid team growth — and ways to address them: Hiring more engineers won't speed you up unless product, design and QA grow to match. Start by growing product & design. If product lags behind, use the spare time to clean up the tech debt. Expand the area of responsibility of the oversized roles. Teams over 8–10 people are hard to manage. Split into chunks of 3–7 people, preferably by business domain. Mix old and new members in each team. Split step-by-step instead of going all in. The processes of your original team won't accommodate a larger team. Localize the processes to the sub-teams as much as possible. Let the teams tweak their processes to suit their needs. Stricten the remaining centralized processes. Onboarding is time-consuming, and can slow you down. Go slow: onboarding a person every week is easier than 6 people at once. Write the docs instead of explaining stuff over and over. Peer-to-peer knowledge transfer is better than onboarding everyone personally. Hiring only senior engineers is not a silver bullet. Aim for a healthy experience distribution in the mid-term. Hope these tips help you get past the scaling issues and up to speed in no time.

11 months ago 28 votes
From engineer to manager: what I love, what I hate

It's been almost 2 years since I moved to a team lead role, then to a full-time engineering management position after the expansion of our team. I've been a front-end developer for 7 years before that, and initially I took the "advanced individual contributor" career track before doing the management turnaround. How's it been? Bumpy, but fun. In this article, I'll share the things I love and hate about my current job. Love Let's start with the positive side of management positions. There's plenty to love, honestly. Impact First things first, I adore the power to improve the product we're building, and the overall team well-being that comes with a management position. As an engineer, you'll sometimes find yourself in a tough spot with little to no power to change things. Early morning standups are a chore? Code quality sucks? The new feature makes no sense? As a manager, the power to change is yours: you get both the formal and informal authority to change things for the better, and to make yourself and your team happier. Your words have weight. If you, an IC, say "guys, we really should write tests", everybody goes "oh, crazy old Vladimir, all grumpy again, haha, where do you see tests fit here?". If you, an EM, casually say "guys, we really should write tests", you might be surprised to find the tests unexpectedly growing in different places. Pleasant. Career opportunities Being an engineering manager is a more promising career opportunity than an engineering track. This might be controversial, but hear me out: The EM is normally higher-paid than the basic team-level engineering grades (junior / middle / senior). There are higher IC grades (staff, principal, president of code, whatever) in the EM+ salary bands. These staff+ IC jobs are concentrated in larger tech companies, because those have tougher technical challenges. Almost every software team in every company has a leadership position. The management career ladder is "taller" than that of ICs. Yes, the chances of becoming a CTO are slim, but it's an opportunity that's just not there for a pure IC with no management experience. All in all, I believe the demand for EMs to be steadier than for staff+ engineers, and this path gives you more opportunities at the later stages of your career. On a related note... Transferable skills Management skills are more widely useful than an IC engineering role. A decent front-end engineer with React experience won't have trouble moving to another front-end framework, and can probably transition to a back-end / mobile engineering role with -1 grade (a year handicap or so). That's not bad. What roles are available to someone with engineering management experience? First, you can easily take on a team with a wildly different focus — mobile developers, infrastructure, ML engineers. You'd need some time to get up to speed on the big-picture technical struggles of your new team, but most companies would take this shot. If you don't want to be an EM any more, you're well-positioned to move to a project or product management role. If the entire tech market falls into decline, many management skills would still work for other industries. While I don't see a big flow of tech managers moving into construction business (tech does pay well), there's one alternate path to consider — entrepreneurship. Involvement with people and business decisions makes for great training before starting your own business. So, being a manager gives you quite a bit of career flexibility, and makes you less vulnerable to future technological shifts. Less knowledge rot Suffering from front-end fatigue? Can't keep up with the newest shiniest frameworks and tools? Management's got you covered! The "hot new" agile / kanban / scrum methodologies are 20–30 years old. The basic meeting types (demos, dailies, 1-on-1s) have been developing for centuries. At the core, you have teamwork and human interactions, which haven't changed that much since the beginning of humanity. My grandfather was a big railroad boss in the 70s, and we can sensibly discuss some of my work challenges. "Oh, you have this talented slacker? Give him some big important task, let's see what he's worth." When it comes to computers, he's more like "I'd like a shovel big enough to throw all your silly gadgets into stratosphere." So, if you're tired of keeping up with the latest hot thing in tech, a management role can provide a well-deserved relief. Do keep an eye on what's happening on the tech side of things, but there's no urgency, and no need to get real deep. New challenges Frankly, after 4–5 years of working in a particular tech area, you can solve the vast majority of practical problems well enough. If you want some work challenge, you can: Slightly alter your stack — say, a new FE framework. But it's unlikely to keep you engaged very long. Make a broader career shift — e.g. frontend to backend. This would probably give you another couple years of fun, but such transitions are, in my experience, either random (e.g. your BE dev quits and someone has to fill the role), or hit your salary. Invent problems out of thin air — rewrite everything using a new library, or handle 9000 RPS "for the future". Fun, but most of the time it's more harm than good for your team and business. Of all the possible career moves a seasoned engineer can make, switching to management gives you the most new challenges (years worth of new stuff to learn) without hitting your salary. Hate As much as I like the challenges and impact of my new role, and the practical career benefits, I'll be the first one to admit it has downsides as well. Corporate BS As an engineer, I hated bloody corporate BS: individual performance reviews, useless deadlines, company-enforced restrictions on processes and tech stack. Well, congratulations, as a manager you are the sheriff of these practices, whether you believe in them or not. As a leader of a team in an org with performance calibrations, I must nominate 1 person who hasn't been working hard enough every 6 months. This human chess is soul-sucking, but I can't make it go away — if I don't offer a sacrificial teammate, someone will be picked randomly further down the process. Crazy shit. Sometimes you can negotiate a bit, or hack the process, e.g. assign "below-expected performance" on a round-robin basis, but to your team you'll sometimes be the corporate monster. Sigh. And I haven't even been through the real tough stuff like layoffs, closures and reorganizations. Awkward social situations I've made it to an engineering management position by being good at building stuff. I've been prepared to help with technical decisions, give career guidance, tune processes and set up automation as needed. In fact, a large portion of my job is debugging social tensions and psychological insecurities of people. Your junior engineer comments out a few tests to deploy a feature preview. The QA person sees this, and is very pissed because your whole team apparently does not respect the QA role and the value they provide. Restore trust. A project manager makes an unsuccessful joke that hurts your designer, who's now crying. Make the PM apologize. Am I a kindergarten teacher or something? Boy, I'm no psychologist, and I can't say I'm exceptionally good with people. This part of my job is quite hard, trying to fake it til I make it here. Office hours Life of an engineer is relatively relaxed. If you don't have anything urgent, you can go lay on the grass for half a day, thinking about the future of your project or something. You can miss a few meetings on short notice, no questions asked. Now, you're an EM. Try going and lying on the grass for a few hours. You come back to a messenger full of problems: your intern can't work because she forgot how to npm install; a senior manager wants to discuss some potential feature; release has derailed. Also, you can't really skip a meeting you're supposed to facilitate / organize without some up-front preparation. This might improve as your team matures and builds better processes, but in general you feel office hours much more as a manager, and your work-life balance directly depends on how good you are at your job. Long feedback loop The final thing I hate about management is the long feedback loop of your actions. Most engineering tasks show the result quite fast: new features take weeks to months, and if that's too long for you — fix a bug and see happy users the next day, or refactor some code and watch complexity decrease in a few hours. Amazing! You're a manager? Well, very few of your actions produce a visible result in under a month. Suppose your team has grown too large, and you want to split it up. You must pick a well-rounded set of engineers for the new team, talk to everybody involved to see how they feel about such a change, arrange new regular meetings, set up processes and communications, do some jira magic, maybe isolate the codebases of sub-products. If you think it can be done in a week, well, you're wrong. Then, even the right changes can make things get worse before they get better. Say you're understaffed, and you decide to hire. In the short term, you spend hours and hours interviewing, and a new team member won't get up to speed right away, sucking out precious time for onboarding. It's sometimes hard to see the long-term goal behind the short-term inconvenience. So, while engineering problem-solving is often fairly straightforward, management changes are more similar to large-scale refactorings. You won't see any quick improvements, which can be frustrating. To sum up, moving from an IC engineering role to a management position has been a rollercoaster ride for me, with both bright and bleak spots. Here's what I love: The wider impact on the product and team. Management is a great long-term career track: it gives you more job opportunities than a staff+ IC, the flexibility to move between different technical areas and roles, and skills that will be relevant across various industries for years to come. If you're bored with your field of tech expertise, moving to a management role is a great way to bring the challenge back into your job. And here's what I hate: Enforcing corporate decisions and policies can be soul-sucking. Dealing with social tensions and psychological insecurities of people isn't something I was ready for. It's hard to go offline even for a few hours without preparing in advance. Your actions have long and non-linear feedback loops with very delayed gratification. Now, is this career move the right one for you? If you enjoy challenge and responsibility, and you get an opportunity — I'd say go for it! Yes, management is not a fit for everybody (I'm not even sure it fits me TBH), but it's a great experience that would surely expand your skill set and make you see engineering work from a new angle. If you totally hate it, you have plenty of time to go back into coding =)

a year ago 18 votes
The most useful programming language

Aspiring developers often ask me what's the best programming language to learn. Personally, I mostly work with JS — solid choice, but everyone and their dog learns JS these days, so it might be time to add some diversity. I'm curious — which single programming language covers the most bases for you, and gives you most career opportunities for years to come? That's the question we'll try to answer today. Here's the plan. I made a list of 8 tech specializations: 2 web development areas: back- and front-end. Both pretty big areas, and ones I have most experience with. Mobile and desktop native app development. Native app development (especially desktop apps) seems to have fallen out of favor, but there's still enough work in these areas. Quality assurance automation. QA grows along with engineering, and increasingly relies on automated tests. Embedded systems. We'll focus on microcontroller programming, not fat boxes with a full windows / linux OS. Quite a promising area with the growth of IoT. Game development. Granted, I don't know much about this area, but I'll do my best to cover it as well, as many developers dream of building a fun game someday. Data analysis and Machine Learning. One of the most hyped areas of the last decade. The contenders are the usual suspects from TIOBE top 20: python, C, C++, Java (grouped with Kotlin and other JVM languages), C# (again, throw in VB and other .NET languages), JavaScript (and TypeScript), PHP, Go, Swift, Ruby, Rust. I left out SQL and Scratch, because they're not general-purpose languages, and Fortan with Matlab, because they aren't really used outside of scientific / engineering computing. A language scores 1 point by being the industry standard in the area — vast community and ecosystem, abundant jobs. Being useful for certain tasks in the area gets you 0.5 points. So, let's see what languages will make you the most versatile engineer, shall we? Backend Let's start with the simple one — Java, C#, Python, PHP, Go and Ruby are all excellent back-end programming languages. Of these, I'd say PHP is slightly more useful as many low-code solutions rely on it, and Ruby is steadily declining. Still, all these languages have earned 1 point. Next, 0.5 points go to: C++, used in high-load and time-critical scenarios, JS — node.js is often used to support front-end, but there aren't that many strictly back-end jobs for JS developers. Rust — still not that widely used, but growing fast. The only languages to fail here are Swift (technically usable on server via e.g. vapor, but I couldn't find any jobs in this stack) and C. Frontend Obviously, JavaScript is the language for front-end developers, which runs natively in browsers. But, surprise, other languages still qualify! All solid back-end languages (Java, C#, Python, PHP, Go, Ruby) get 0.5 points, because you can solve many UI problems by rendering HTML server-side the old-school way. C# has a slight edge here, since blazor is quite smart and popular. C, C++, and Rust score 0.5 points because they can be compiled to WebAssembly and run in the browser — just look at figma. Rust also powers some cool JS tooling, like biome and swc The only language to fail here is, again, Swift. QA automation The topic of QA automation is really simple. Java and python get the cake — Allure, Selenium, JUnit, and pytest are the most sought-after automation tools on the market right now. JS gets 0.5 points for playwright and cypress — the preferred tools for testing complex web front-ends. A few automation tools support C# — worth 0.2 points. Mobile apps Another straightforward area. Android apps are written in JVM languages (Java / Kotlin), iOS is integrated with Swift (finally). JS scores 0.5 points, because you can effectively build apps with React Native, and you can get pretty far with PWA or a good old WebView. Another 0.5 point for C#, thanks to Xamarin and MAUI. Desktop apps (windows / linux / MacOS) The three kings here are C++, C#, and Java. JS gets 0.5 points, again, for electron — disgusting or not, it's widely used. Another 0.5 points for Swift, because that's what you build MacOS apps with, but MacOS computers are relatively niche. Rust has the highly-hyped Tauri project for building desktop apps, but it's not that widespread, and I'm not aware of any high-profile apps using it. Let's give each 0.2 points for the effort and check back later. Embedded systems Embedded systems are usually tight on resources, so compiled languages are the way to go here. Basically any embedded job requires C and C++. Rust is, as usual, very promising, but not that popular yet, so 0.5 points. Another half-point for Python — used for edge computer vision and prototyping, but struggling with high memory requirements. Game development The primary languages in big gamedev are C++ (used in Unreal Engine) and C# (for Unity). Since mobile games are a thing, Java and Swift get 0.5 points each, because that's what you'll likely use here. Another 0.5 points for JS (browser games). Rust should be quite a good fit for games, but (as expected by now) it's not quite there yet. Data Analysis & Machine Learning It's no secret that Python is the language of choice for anything data-related, and most of the cutting edge stuff happens, well deserved 1 point here. But do you know there's another top language to get your piece of Data & ML hype? Big companies have a lot of data, right? And big companies love Java. So, many big data tools (especially coming from Apache — Hadoop, Spark, Jena) work with Java, and most data jobs require experience with python or java, so another 1 point for java. On to more surprises. Large chunks of data-heavy python libraries are actually written in C / C++ — e.g. over a third of numpy, or most of LlamaCPP — which earns both half-a-point. As you'd expect, Rust is also gaining traction for this use case with stuff like pola.rs, so another 0.2 points! The final half-a-point goes to JS for powering much of the UI / visualization stuff (see e.g. bokeh). Before we reveal our final ranking, let's weigh the categories, because they're not the same size. I've used some back-of-the-napkin analysis of job postings and sizing of reddit / linkedin groups and my personal experience. With backend as our reference, I'd say frontend is roughly the same size. Mobile development is surprisingly sizable — let's give it a 0.6 weight. For QA, I'd say 0.2 makes sense, as 1 QA per 3–5 devs is a normal ratio, and manual QA is still a thing. Desktop is easily the smallest area, looks like a 0.1 to me. For gamedev, 0.5 is just my random guess. Finally, there are surprisingly many data people — with the good salaries, let's make it a 0.6. Putting it all together: Java takes the first spot by a good margin by topping 5 categories, and having some gamedev / frontend capabilities. Place other JVM languages (especially Kotlin) around here, but with a discount since they're not as widely used. The next three are really close, but JS gets slighly ahead by being average at everything except embedded, even though it's only the top choice for front-end development. Python and C# tie for the third place. Both are top-tier backend languages with other strong areas (QA / ML for python, desktop and gamedev for C#). C++ is not that far behind either, as it's still the top language when it comes to efficiency. It also steps into other languages' realms when they need some speedup (WebAssembly / ML). Next come "three backend friends" — Go, PHP, and Ruby. All top-notch languages for building web backends, but not much else beyond that. Of these, Ruby is on the decline, and PHP and Go both have their separate niches. Rust does not score that well, but still makes it into the top 10 — not bad for such a new language. It has great growth potential by eating at the traditional C++ areas, super excited to see where it gets in 3–5 years. We all love good old C, but C++ looks like a better fit for complex systems. Swift comes in last — fair enough for a language that's only useful for the products of one single company. Perhaps surprisingly, the single most useful language is Java. Python and JS, beginner favorites, come strong, with a very different focus. C# perhaps deserves a bit more attention. Overall, today we've learnt about many amazing technologies that allow languages to sneak into each other's territory. If you were to start anew, what language would you learn?

a year ago 19 votes
I conducted 60 interviews in 2 months — here's what I learned

It's hard to believe, but, starting mid-october 2023 I conducted 60 technical interviews and hired 10 people into our team. It's been extremely tiring: around 80 hours of active interviewing, plus writing interview reports, plus screening CVs and take-home assignments, plus onboarding new members — all while doing my normal work stuff. Still, I feel like I learnt a lot in the process — things that would help me as a candidate in the future, and might help you land your next job. Note that I'm a fairly relaxed interviewer, and, as an internal startup of a large tech company, we generally have a more humane hiring process, so your mileage may vary. Still, I've done my best to pick the tips that I feel are universally applicable. Here are nine insights I took out of this experience, in no particular order: Be generous with your "expected income". Say you're a solid higher-middle engineer, and you ask for a senior salary. My thought process: OK buddy, it's a bit more than reasonable now, but I won't have to fight for your promotion 8–12 months from now when you get there, and I don't have to spend another 12 hours of my own time (and leave my team understaffed for another few weeks) looking for a real hardcore senior, so I'll let you have it. Now suppose you ask for a junior salary. It's suspicious — why is your bar so low? Is there someting about your work performance you're not telling us? So, do your research on reasonable salaries for your level of experience, and aim slightly above that. Ask the right questions. I always leave time for the candidate to ask me questions — obviously, this lets the candidate probe what it's like to work at our team, but it's also the best opportunity for me to learn what really matters to the candidate. I've never been much of an asker myself, but now I see that "Thanks, I have no questions" does not look good — if anything, it paints you as someone who doesn't care. Here's a short list of good questions: What does the daily work in this role look like? Harmless. What features are you building next? Caring about the overall product, nice. Sometimes the answer is "I can't disclose this secret", but not that often. Anyting about processes or team structure: how many people are on the team? How often do you release? What regular meetings do you have? Interested in organization, might want to be a team lead someday, great. Anything tech-related: which framework do you use? Why did you pick framework X? How do you test your app? Especially suitable for junior- to middle developers who are most involved in hands-on work. What kind of tasks do you see me doing? Again, just a good neutral question, because responsibilities for any role differ wildly between companies. What growth / promotion opportunities does this position have? Cool trick, flipping the feared "where do you see yourself in 5 years" question against the hiring manager. Here are a few questions that are not very good: Do you use jira and github? It's a minor detail, won't you be able to work with youtrack and gitlab? Do you sometimes work late? Only if something breaks, but overall this question makes you seem a bit lazy. People on poor teams that routinely overtime aren't likely to answer this question honestly, at any rate. Social skills matter. I understand that not everybody is super outgoing, but if we already feel awkward 1 hour into our acquaintance, why work together — to feel awkward for months to come? Just a few tips anyone can follow: Be energetic. You're tired, I'm tired, we're all tired of endless interviews. Are you just tired today, or generally always too tired to get anything done? I know it's easier said than done, but try and show me all the energy you have left. Show respect. People enjoy being respected. Very easy one: you have a great product. Sounds like you have a great engineering culture. This is one of the most interesting interviews I've ever seen. Like, I know you don't necessarily mean that, but subconsciously I'm very pleased: "oh yes, I'm very proud of my interview process, thanks for noticing" On a related note... Provide conversation opportunities. Q: Do you use TDD? Bad answer: "no". Good answer: "no, but I've heard of it. Interesting approach. Does your team use TDD?" Now you get to spend 5 minutes talking on your terms instead of being bombarded with random questions, and you come off as someone curious about stuff. On another related note... It's easy to hurt people. People normally ask you about stuff because they care about it. So, again, the interviewer askning "do you use TDD?", presumably, likes TDD and uses it. So, the worst answer: "no, TDD sucks, it's pure waste of time for idiots." A rare interviewer might appreciate you having a strong opinion on a topic, but to most this just paints you as a jerk, kinda like "Here's a photo of my children — "I hate children, and yours are especialy horrible". Not smart. Smart talk is not your friend. Saying stuff like "our front-end guild evaluated several cutting-edge approaches to testing universal applications" only makes you seem smart if you can elaborate on that topic: what these approaches were, the pros and cons you found, what tradeoffs you made for your final decision. If you can't answer a follow-up question beside "we settled on jest, not sure why", it was better to stay away from that topic altogether. Related: "in code reviews, I always consider the optimality of the algorithm selected" (proceeds to estimate the time complexity of comparison-based sorting as O(1). I never ask this unless the candidate boasts about her algo skillz). Admit your mistakes. Don't know an answer? Your code has a bug? It's always better to admit it and then try to come up with something at the spot than trying to talk your way out of it. Event loop? Sure thing, I'm an expert on loops. It's the way events are looped. Uses logarithmic weighing. Again, this makes you look like a candidate with big mouth and small hands. I have seen a couple of people who could talk their way out of any situation, but I honestly think with such skills you'd do better in a different line of work, like international relations, or selling financial services. Note that you really should give it your best shot — giving up at the first sign of trouble is not a good impression. If you genuinely have no idea — see conversation opportunity: "Event delegation? Tough luck, never heard of it. Would you tell me about it so that I learn something new today?" Make yourself memorable. It's hard to keep detailed profiles of 10 candidates in mind — after a good interview streak all I remember is the general impression (great / OK / horrible) and a few truly notable things. This guy worked for some crypto scam that went bust, that girl had a cute dog that was trying to eat the camera. The worst you can do is be a totally neutral candidate — we've had an interview, but I can't remember any details. So try and sneak some anecdote, or wear a silly scarf — something to remember. This point is especially important for intern / junior positions — online JS bootcamps do a good job of covering the basics, and it's really hard to differentiate these candidates. The memorable thing doesn't have to be professional, or even positive (even though it sure won't hurt) — your best bet would be some original personal project. Ask for feedback on the spot. Asking how you did at the end of the interview doesn't hurt. Yes, some interviewers will be hesitant to answer — at large companies, the feedback is normally sent through the recruiter, and you're never sure if sidestepping this process would get you into trouble. Besides, if the feedback is not complimentary, you're essentially asking for conflict at the spot, and people normally avoid conflict when possible. Still, it's a chance to adjust your expectations (if the interviewer says, looking you in the eyes, that you've done great, it's a good sign), and you might get actually useful tips that would probably get lost passing through the written report, and then through the non-technical recruiter.

a year ago 18 votes
Svelte stores: the curious parts

We've already learnt a lot about svelte's reactivity system — the primary way to work with state in svelte components. But not all state belongs in components — sometimes we want app-global state (think state manager), sometimes we just want to reuse logic between components. React has hooks, Vue has composables. For svelte, the problem is even harder — reactive state only works inside component files, so the rest is handled by a completely separate mechanism — stores. The tutorial does a decent job of covering the common use cases, but I still had questions: What's the relationship between the stores? Are they built on some common base? Is it safe to use { set } = store as a free function? How does get(store) receive the current value if it's not exposed on the object? Does set() trigger subscribers when setting the current value? What's the order of subscriber calls if you set() inside a subscriber? Does derived listen to the base stores when it's not observed? Will changing two dervied dependencies trigger one or two derived computations? Why does subscribe() have a second argument? What is $store sytax compiled to? In this article, I explore all these questions (and find a few svelte bugs in the process). writable is the mother store Svelte has 3 built-in store types: writable, readable, and derived. However, they are neatly implemented in terms of one another, taking only 236 lines, over half of which is TS types and comments. The implementation of readable is remarkably simple — it creates a writable, and only returns its subscribe method. Let me show it in its entirety: const readable = (value, start) => ({ subscribe: writable(value, start).subscribe }); Moreover, derived is just a special way of constructing readable: export function derived(stores, fn, initial_value) { // ...some normalization return readable(initial_value, /* some complex code */); } While we're at it, note that update method of a writable store is a very thin wrapper over set: fn => set(fn(value)). All in all: writable is the OG store, readable just removes set & update methods from a writable, derived is just a predefined readable setup, update is just a wrapper over set. This greatly simplifies our analysis — we can just investigate writable arguments, subscribe, and set — and our findings also hold for other store types. Well done, svelte! Store methods don't rely on this Writable (and, by extension, readable and derived) is implemented with objects and closures, and does not rely on this, so you can safely pass free methods around without dancing with bind: const { subscribe, set } = writable(false); const toggle = { subscribe, activate: () => set(true) }; However, arbitrary custom stores are not guaranteed to have this trait, so it's best to stay safe working with an unknown store-shaped argument — like svelte itself does with readonly: function readonly(store) { return { subscribe: store.subscribe.bind(store), }; } Subscriber is invoked immediately As svelte stores implement observable value pattern, you'd expect them to have a way to access current value via store.get() or store.value — but it's not there! Instead, you use the special get() helper function: import { get } from 'svelte/store' const value = get(store); But, if the store does not expose a value, how can get(store) synchronously access it? Normally, the subscribers are only called on change, which can occur whenever. Well, svelte subscribe is not your average subscribe — calling subscribe(fn) not only starts listening to changes, but also synchronously calls fn with the current value. get subscribes to the store, extracts the value from this immediate invocation, and immediately unsubscribes — like this: let value; const unsub = store.subscribe(v => value = v); unsub(); The official svelte tutorial section on custom stores says: as long as an object correctly implements the subscribe method, it's a store. This might bait you into writing "custom stores" with subscribe method, not based off of writable. The trick word here is correctly implements — even based on the tricky subscribe self-invocation it's not an easy feat, so please stick to manipulations with readable / writable / derived. set() is pure for primitives writable stores are pure in the same sense as svelte state — notifications are skipped when state is primitive, and the next value is equal to the current one: const s = writable(9); // logs 9 because immediate self-invocation s.subscribe(console.log); // does not log s.set(9); Object state disables this optimization — you can pass a shallow equal object, or the same (by reference) object, the subscribers will be called in any case: const s = writable({ value: 9 }); s.subscribe(console.log); // each one logs s.update(s => s); s.set(get(s)); s.set({ value: 9 }); On the bright side, you can mutate the state in update, and it works: s.update(s => { s.value += 1; return s }); Subscriber consistency Normally, store.set(value) synchronously calls all subscribers with value. However, a naive implementation will shoot you in the foot when updating a store from within a subscriber (if you think it's a wild corner case — it's not, it's how derived stores work): let currentValue = null; const store = naiveWritable(1); store.subscribe(v => { // let's try to avoid 0 if (v === 0) store.set(1); }) store.subscribe(v => currentValue = v); If we now call set(0), we intuitively expect both the store's internal value and currentValue to be 1 after all callbacks settle. But in practice it can fail: Store value becomes 0; First subscriber sees 0, calls set(1), then: Store value becomes 1; set(1) synchronously invokes all subscribers with 1; First subscriber sees 1, does nothing; Second subscriber is called with 1, sets currentValue to 1; First subscriber run for 0 is completed, continuing with the initial updates triggered by set(0) Second subscriber is called with 0, setting currentValue to 0; Bang, inconsistent state! This is very dangerous territory — you're bound to either skip some values, get out-of-order updates, or have subscribers called with different values. Rich Harris has taken a lot of effort to provide the following guarantees, regardless of where you set the value: Every subscriber always runs for every set() call (corrected for primitive purity). Subscribers for one set() run, uninterrupted, after one another (in insertion order, but I wouldn't rely on this too much). Subscribers are invoked globally (across all svelte stores) in the same order as set calls, even when set calls are nested (called from within a subscriber). All subscribers are called synchronously within the outermost set call (the one outside any subscriber). So, in our example, the actual callback order is: subscriber 1 sees 0, calls set(1) subscribers calls with 1 are enqueued subscriber 2 sets currentValue = 0 subscriber 1 runs with 1, does nothing subscriber 2 sets currentValue = 1 Since the callback queue is global, this holds even when updating store B from a subscriber to store A. One more reason to stick with svelte built-in stores instead of rolling your own. Derived is lazy derived looks simple on the surface — I thought it just subscribes to all the stores passed, and keeps an up-to-date result of the mapper function. In reality, it's smarter than that — subscription and unsubscription happens in the start / stop handler, which yields some nice properties: Subscriptions to base stores are automatically removed once you stop listening to the derived store, no leaks. Derived value and subscriptions are reused no matter how many times you subscribe to a derived store. When nobody is actively listening to a derived store, the mapper does not run. The value is automatically updated when someone first subscribes to the derived store (again, courtesy of subscribe self-invocation). Very, very tastefully done. Derived is not transactional While lazy, derived is not transactional, and not batched — synchronously changing 2 dependencies will trigger 2 derivations, and 2 subscriber calls — one after the first update, and one after the second one. In this code sample, we'd expect left + right to always be 200 (we synchronously move 10 from left to right), there's a glimpse of 190 (remember, the subscribers are synchronously called during set): const left = writable(100); const right = writable(100); const total = derived([left, right], ([x, y]) => { console.log('derive', x, y); return x + y; }); total.subscribe(t => console.log('total', t)); const update = () => { // try to preserve total = 200 left.update(l => l - 10); // ^^ derives, and logs "total 190" right.update(r => r + 10); // ^^ derives, and logs "total 200" }; This isn't a deal breaker, svelte won't render the intermediate state, but it's something to keep in mind, or you get hurt: const obj = writable({ me: { total: 0 } }); const key = writable('me'); const value = derived([obj, key], ([obj, key]) => obj[key].total); // throws, because { me: ... } has no 'order' field key.set('order'); obj.set({ order: { total: 100 } }); The mysteryous subscriber-invalidator Looking at subscribe() types, you may've noticed the mysterious second argument — invalidate callback. Unlike the subscriber, it's not queued, and is always called synchronously during set(). The only place I've seen an invalidator used in svelte codebase is inside derived — and, TBH, I don't understand its purpose. I expected it to stabilize derived chains, but it's not working. Also, the TS types are wrong — the value is never passed to invalidator as an argument. Verdict: avoid. $-dereference internals As you probably know, svelte components have a special syntax sugar for accessing stores — just prefix the store name with a $, and you can read and even assign it like a regular reactive variable — very convenient: import { writable } from 'svelte/store'; const value = writable(0); const add = () => $value += 1; <button on:click={add}> {$value} </button> I always thought that $value is compiled to get, $value = v to value.set(v), and so on, with a subscriber triggering a re-render in some smart way, but it's not the case. Instead, $value becomes a regular svelte reactive variable, synchronized to the store, and the rest is handled by the standard svelte update mechanism. Here's the compilation result: // the materialized $-variable let $value; // the store const value = writable(0); // auto-subscription const unsub = value.subscribe(value, value => { $$invalidate(0, $value = value) }); onDestroy(unsub); const add = () => { // assign to variable $value += 1; // update store value.set($value); }; In plain English: $store is a real actual svelte reactive variable. store.subscribe updates the variable and triggers re-render. The unsubscriber is stored and called onDestroy. AFAIK, store.update is never used by svelte. Assignments to $store simultaneously mutate $store variable without invalidating and triggering re-render and call store.set, which in turn enqueues the update via $$invalidate The last point puts us in a double-source-of-truth situation: the current store value lives both in the $store reactive variable, and inside store itself. I expected this to cause some havok in an edge case, and so it does — if you patch store.set method to skip some updates, the $-variable updates before your custom set runs, and the two values go out of sync as of svelte@3.59.1: const value = { ...writable(0), // prevent updates set: () => {} }; const add = () => $value += 1; let rerender = {}; $: total = $value + (rerender ? 0 : 1); {total} <button on:click={add}>increment</button> <button on:click={() => rerender = {}}> rerender </button> To summarize: Both readable and derived are built on top of writable — readable only picks subscribe method, derived is a readable with a smart start / stop notifier. Built-in stores don't rely on this, so you can safely use their methods as free functions. Calling subscribe(fn) immediately invokes fn with the current value — used in get(store) to get the current value. Calling set() with the current value of the store will skip notifying subscribers if the value is primitive. set() on object state always notifies, even if the object is same, by reference, as the current state. The subscribers for a single set() run after one another. If a subscriber calls set, this update will be processed once the first set() is fully flushed. derived only subscribes to the base stores and maps the value when someone's actively listening to it. When synchronously changing two dependencies of derived, the mapper will be called after the first change. There's no way to batch these updates. subscribe() has a second argument — a callback that's called synchronously during set(). I can't imagine a use case for it. $store syntax generates a regular svelte reactive variable called $store, and synchronizes it with the store in a subscriber. If you learn one thing from this article — svelte stores are thoughtfully done and help you with quite a few corner-cases. Please avoid excessive trickery, and build on top of the svelte primitives. In the next part of my svelte series, I'll show you some neat tricks with stores — stay tuned on twitter!

a year ago 16 votes

More in programming

ChatGPT Would be a Decent Policy Advisor

Revealed: How the UK tech secretary uses ChatGPT for policy advice by Chris Stokel-Walker for the New Scientist

12 hours ago 3 votes
Setting policy for strategy.

This book’s introduction started by defining strategy as “making decisions.” Then we dug into exploration, diagnosis, and refinement: three chapters where you could argue that we didn’t decide anything at all. Clarifying the problem to be solved is the prerequisite of effective decision making, but eventually decisions do have to be made. Here in this chapter on policy, and the following chapter on operations, we finally start to actually make some decisions. In this chapter, we’ll dig into: How we define policy, and how setting policy differs from operating policy as discussed in the next chapter The structured steps for setting policy How many policies should you set? Is it preferable to have one policy, many policies, or does it not matter much either way? Recurring kinds of policies that appear frequently in strategies Why it’s valuable to be intentional about your strategy’s altitude, and how engineers and executives generally maintain different altitudes in their strategies Criteria to use for evaluating whether your policies are likely to be impactful How to develop novel policies, and why it’s rare Why having multiple bundles of alternative policies is generally a phase in strategy development that indicates a gap in your diagnosis How policies that ignore constraints sound inspirational, but accomplish little Dealing with ambiguity and uncertainty created by missing strategies from cross-functional stakeholders By the end, you’ll be ready to evaluate why an existing strategy’s policies are struggling to make an impact, and to start iterating on policies for strategy of your own. This is an exploratory, draft chapter for a book on engineering strategy that I’m brainstorming in #eng-strategy-book. As such, some of the links go to other draft chapters, both published drafts and very early, unpublished drafts. What is policy? Policy is interpreting your diagnosis into a concrete plan. That plan will be a collection of decisions, tradeoffs, and approaches. They’ll range from coding practices, to hiring mandates, to architectural decisions, to guidance about how choices are made within your organization. An effective policy solves the entirety of the strategy’s diagnosis, although the diagnosis itself is encouraged to specify which aspects can be ignored. For example, the strategy for working with private equity ownership acknowledges in its diagnosis that they don’t have clear guidance on what kind of reduction to expect: Based on general practice, it seems likely that our new Private Equity ownership will expect us to reduce R&D headcount costs through a reduction. However, we don’t have any concrete details to make a structured decision on this, and our approach would vary significantly depending on the size of the reduction. Faced with that uncertainty, the policy simply acknowledges the ambiguity and commits to reconsider when more information becomes available: We believe our new ownership will provide a specific target for Research and Development (R&D) operating expenses during the upcoming financial year planning. We will revise these policies again once we have explicit targets, and will delay planning around reductions until we have those numbers to avoid running two overlapping processes. There are two frequent points of confusion when creating policies that are worth addressing directly: Policy is a subset of strategy, rather than the entirety of strategy, because policy is only meaningful in the context of the strategy’s diagnosis. For example, the “N-1 backfill policy” makes sense in the context of new, private equity ownership. The policy wouldn’t work well in a rapidly expanding organization. Any strategy without a policy is useless, but you’ll also find policies without context aren’t worth much either. This is particularly unfortunate, because so often strategies are communicated without those critical sections. Policy describes how tradeoffs should be made, but it doesn’t verify how the tradeoffs are actually being made in practice. The next chapter on operations covers how to inspect an organization’s behavior to ensure policies are followed. When reworking a strategy to be more readable, it often makes sense to merge policy and operation sections together. However, when drafting strategy it’s valuable to keep them separate. Yes, you might use a weekly meeting to review whether the policy is being followed, but whether it’s an effective policy is independent of having such a meeting, and what operational mechanisms you use will vary depending on the number of policies you intend to implement. With this definition in mind, now we can move onto the more interesting discussion of how to set policy. How to set policy Every part of writing a strategy feels hard when you’re doing it, but I personally find that writing policy either feels uncomfortably easy or painfully challenging. It’s never a happy medium. Fortunately, the exploration and diagnosis usually come together to make writing your policy simple: although sometimes that simple conclusion may be a difficult one to swallow. The steps I follow to write a strategy’s policy are: Review diagnosis to ensure it captures the most important themes. It doesn’t need to be perfect, but it shouldn’t have omissions so obvious that you can immediately identify them. Select policies that address the diagnosis. Explicitly match each policy to one or more diagnoses that it addresses. Continue adding policies until every diagnosis is covered. This is a broad instruction, but it’s simpler than it sounds because you’ll typically select from policies identified during your exploration phase. However, there certainly is space to tweak those policies, and to reapply familiar policies to new circumstances. If you do find yourself developing a novel policy, there’s a later section in this chapter, Developing novel policies, that addresses that topic in more detail. Consolidate policies in cases where they overlap or adjoin. For example, two policies about specific teams might be generalized into a policy about all teams in the engineering organization. Backtest policy against recent decisions you’ve made. This is particularly effective if you maintain a decision log in your organization. Mine for conflict once again, much as you did in developing your diagnosis. Emphasize feedback from teams and individuals with a different perspective than your own, but don’t wholly eliminate those that you agree with. Just as it’s easy to crowd out opposing views in diagnosis if you don’t solicit their input, it’s possible to accidentally crowd out your own perspective if you anchor too much on others’ perspectives. Consider refinement if you finish writing, and you just aren’t sure your approach works – that’s fine! Return to the refinement phase by deploying one of the refinement techniques to increase your conviction. Remember that we talk about strategy like it’s done in one pass, but almost all real strategy takes many refinement passes. The steps of writing policy are relatively pedestrian, largely because you’ve done so much of the work already in the exploration, diagnosis, and refinement steps. If you skip those phases, you’d likely follow the above steps for writing policy, but the expected quality of the policy itself would be far lower. How many policies? Addressing the entirety of the diagnosis is often complex, which is why most strategies feature a set of policies rather than just one. The strategy for decomposing a monolithic application is not one policy deciding not to decompose, but a series of four policies: Business units should always operate in their own code repository and monolith. New integrations across business unit monoliths should be done using gRPC. Except for new business unit monoliths, we don’t allow new services. Merge existing services into business-unit monoliths where you can. Four isn’t universally the right number either. It’s simply the number that was required to solve that strategy’s diagnosis. With an excellent diagnosis, your policies will often feel inevitable, and perhaps even boring. That’s great: what makes a policy good is that it’s effective, not that it’s novel or inspiring. Kinds of policies While there are so many policies you can write, I’ve found they generally fall into one of four major categories: approvals, allocations, direction, and guidance. This section introduces those categories. Approvals define the process for making a recurring decision. This might require invoking an architecture advice process, or it might require involving an authority figure like an executive. In the Index post-acquisition integration strategy, there were a number of complex decisions to be made, and the approval mechanism was: Escalations come to paired leads: given our limited shared context across teams, all escalations must come to both Stripe’s Head of Traffic Engineering and Index’s Head of Engineering. This allowed the acquired and acquiring teams to start building trust between each other by ensuring both were consulted before any decision was finalized. On the other hand, the user data access strategy’s approval strategy was more focused on managing corporate risk: Exceptions must be granted in writing by CISO. While our overarching Engineering Strategy states that we follow an advisory architecture process as described in Facilitating Software Architecture, the customer data access policy is an exception and must be explicitly approved, with documentation, by the CISO. Start that process in the #ciso channel. These two different approval processes had different goals, so they made tradeoffs differently. There are so many ways to tweak approval, allowing for many different tradeoffs between safety, productivity, and trust. Allocations describe how resources are split across multiple potential investments. Allocations are the most concrete statement of organizational priority, and also articulate the organization’s belief about how productivity happens in teams. Some companies believe you go fast by swarming more people onto critical problems. Other companies believe you go fast by forcing teams to solve problems without additional headcount. Both can work, and teach you something important about the company’s beliefs. The strategy on Uber’s service migration has two concrete examples of allocation policies. The first describes the Infrastructure engineering team’s allocation between manual provision tasks and investing into creating a self-service provisioning platform: Constrain manual provisioning allocation to maximize investment in self-service provisioning. The service provisioning team will maintain a fixed allocation of one full time engineer on manual service provisioning tasks. We will move the remaining engineers to work on automation to speed up future service provisioning. This will degrade manual provisioning in the short term, but the alternative is permanently degrading provisioning by the influx of new service requests from newly hired product engineers. The second allocation policy is implicitly noted in this strategy’s diagnosis, where it describes the allocation policy in the Engineering organization’s higher altitude strategy: Within infrastructure engineering, there is a team of four engineers responsible for service provisioning today. While our organization is growing at a similar rate as product engineering, none of that additional headcount is being allocated directly to the team working on service provisioning. We do not anticipate this changing. Allocation policies often create a surprising amount of clarity for the team, and I include them in almost every policy I write either explicitly, or implicitly in a higher altitude strategy. Direction provides explicit instruction on how a decision must be made. This is the right tool when you know where you want to go, and exactly the way that you want to get there. Direction is appropriate for problems you understand clearly, and you value consistency more than empowering individual judgment. Direction works well when you need an unambiguous policy that doesn’t leave room for interpretation. For example, Calm’s policy for working in the monolith: We write all code in the monolith. It has been ambiguous if new code (especially new application code) should be written in our JavaScript monolith, or if all new code must be written in a new service outside of the monolith. This is no longer ambiguous: all new code must be written in the monolith. In the rare case that there is a functional requirement that makes writing in the monolith implausible, then you should seek an exception as described below. In that case, the team couldn’t agree on what should go into the monolith. Individuals would often make incompatible decisions, so creating consistency required removing personal judgment from the equation. Sometimes judgment is the issue, and sometimes consistency is difficult due to misaligned incentives. A good example of this comes in strategy on working with new Private Equity ownership: We will move to an “N-1” backfill policy, where departures are backfilled with a less senior level. We will also institute a strict maximum of one Principal Engineer per business unit. It’s likely that hiring managers would simply ignore this backfill policy if it was stated more softly, although sometimes less forceful policies are useful. Guidance provides a recommendation about how a decision should be made. Guidance is useful when there’s enough nuance, ambiguity, or complexity that you can explain the desired destination, but you can’t mandate the path to reaching it. One example of guidance comes from the Index acquisition integration strategy: Minimize changes to tokenization environment: because point-of-sale devices directly work with customer payment details, the API that directly supports the point-of-sale device must live within our secured environment where payment details are stored. However, any other functionality must not be added to our tokenization environment. This might read like direction, but it’s clarifying the desired outcome of avoiding unnecessary complexity in the tokenization environment. However, it’s not able to articulate what complexity is necessary, so ultimately it’s guidance because it requires significant judgment to interpret. A second example of guidance comes in the strategy on decomposing a monolithic codebase: Merge existing services into business-unit monoliths where you can. We believe that each choice to move existing services back into a monolith should be made “in the details” rather than from a top-down strategy perspective. Consequently, we generally encourage teams to wind down their existing services outside of their business unit’s monolith, but defer to teams to make the right decision for their local context. This is another case of knowing the desired outcome, but encountering too much uncertainty to direct the team on how to get there. If you ask five engineers about whether it’s possible to merge a given service back into a monolithic codebase, they’ll probably disagree. That’s fine, and highlights the value of guidance: it makes it possible to make incremental progress in areas where more concrete direction would cause confusion. When you’re working on a strategy’s policy section, it’s important to consider all of these categories. Which feel most natural to use will vary depending on your team and role, but they’re all usable: If you’re a developer productivity team, you might have to lean heavily on guidance in your policies and increased support for that guidance within the details of your platform. If you’re an executive, you might lean heavily on direction. Indeed, you might lean too heavily on direction, where guidance often works better for areas where you understand the direction but not the path. If you’re a product engineering organization, you might have to narrow the scope of your direction to the engineers within that organization to deal with the realities of complex cross-organization dynamics. Finally, if you have a clear approach you want to take that doesn’t fit cleanly into any of these categories, then don’t let this framework dissuade you. Give it a try, and adapt if it doesn’t initially work out. Maintaining strategy altitude The chapter on when to write engineering strategy introduced the concept of strategy altitude, which is being deliberate about where certain kinds of policies are created within your organization. Without repeating that section in its entirety, it’s particularly relevant when you set policy to consider how your new policies eliminate flexibility within your organization. Consider these two somewhat opposing strategies: Stripe’s Sorbet strategy only worked in an organization that enforced the use of a single programming language across (essentially) all teams Uber’s service migration strategy worked well in an organization that was unwilling to enforce consistent programming language adoption across teams Stripe’s organization-altitude policy took away the freedom of individual teams to select their preferred technology stack. In return, they unlocked the ability to centralize investment in a powerful way. Uber went the opposite way, unlocking the ability of teams to pick their preferred technology stack, while significantly reducing their centralized teams’ leverage. Both altitudes make sense. Both have consequences. Criteria for effective policies In The Engineering Executive’s Primer’s chapter on engineering strategy, I introduced three criteria for evaluating policies. They ought to be applicable, enforced, and create leverage. Defining those a bit: Applicable: it can be used to navigate complex, real scenarios, particularly when making tradeoffs. Enforced: teams will be held accountable for following the guiding policy. Create Leverage: create compounding or multiplicative impact. The last of these three, create leverage, made sense in the context of a book about engineering executives, but probably doesn’t make as much sense here. Some policies certainly should create leverage (e.g. empower developer experience team by restricting new services), but others might not (e.g. moving to an N-1 backfill policy). Outside the executive context, what’s important isn’t necessarily creating leverage, but that a policy solves for part of the diagnosis. That leaves the other two–being applicable and enforced–both of which are necessary for a policy to actually address the diagnosis. Any policy which you can’t determine how to apply, or aren’t willing to enforce, simply won’t be useful. Let’s apply these criteria to a handful of potential policies. First let’s think about policies we might write to improve the talent density of our engineering team: “We only hire world-class engineers.” This isn’t applicable, because it’s unclear what a world-class engineer means. Because there’s no mutually agreeable definition in this policy, it’s also not consistently enforceable. “We only hire engineers that get at least one ‘strong yes’ in scorecards.” This is applicable, because there’s a clear definition. This is enforceable, depending on the willingness of the organization to reject seemingly good candidates who don’t happen to get a strong yes. Next, let’s think about a policy regarding code reuse within a codebase: “We follow a strict Don’t Repeat Yourself policy in our codebase.” There’s room for debate within a team about whether two pieces of code are truly duplicative, but this is generally applicable. Because there’s room for debate, it’s a very context specific determination to decide how to enforce a decision. “Code authors are responsible for determining if their contributions violate Don’t Repeat Yourself, and rewriting them if they do.” This is much more applicable, because now there’s only a single person’s judgment to assess the potential repetition. In some ways, this policy is also more enforceable, because there’s no longer any ambiguity around who is deciding whether a piece of code is a repetition. The challenge is that enforceability now depends on one individual, and making this policy effective will require holding individuals accountable for the quality of their judgement. An organization that’s unwilling to distinguish between good and bad judgment won’t get any value out of the policy. This is a good example of how a good policy in one organization might become a poor policy in another. If you ever find yourself wanting to include a policy that for some reason either can’t be applied or can’t be enforced, stop to ask yourself what you’re trying to accomplish and ponder if there’s a different policy that might be better suited to that goal. Developing novel policies My experience is that there are vanishingly few truly novel policies to write. There’s almost always someone else has already done something similar to your intended approach. Calm’s engineering strategy is such a case: the details are particular to the company, but the general approach is common across the industry. The most likely place to find truly novel policies is during the adoption phase of a new widespread technology, such as the rise of ubiquitous mobile phones, cloud computing, or large language models. Even then, as explored in the strategy for adopting large-language models, the new technology can be engaged with as a generic technology: Develop an LLM-backed process for reactivating departed and suspended drivers in mature markets. Through modeling our driver lifecycle, we determined that improving onboarding time will have little impact on the total number of active drivers. Instead, we are focusing on mechanisms to reactivate departed and suspended drivers, which is the only opportunity to meaningfully impact active drivers. You could simply replace “LLM” with “data-driven” and it would be equally readable. In this way, policy can generally sidestep areas of uncertainty by being a bit abstract. This avoids being overly specific about topics you simply don’t know much about. However, even if your policy isn’t novel to the industry, it might still be novel to you or your organization. The steps that I’ve found useful to debug novel policies are the same steps as running a condensed version of the strategy process, with a focus on exploration and refinement: Collect a number of similar policies, with a focus on how those policies differ from the policy you are creating Create a systems model to articulate how this policy will work, and also how it will differ from the similar policies you’re considering Run a strategy testing cycle for your proto-policy to discover any unknown-unknowns about how it works in practice Whether you run into this scenario is largely a function of the extent of your, and your organization’s, experience. Early in my career, I found myself doing novel (for me) strategy work very frequently, and these days I rarely find myself doing novel work, instead focusing on adaptation of well-known policies to new circumstances. Are competing policy proposals an anti-pattern? When creating policy, you’ll often have to engage with the question of whether you should develop one preferred policy or a series of potential strategies to pick from. Developing these is a useful stage of setting policy, but rather than helping you refine your policy, I’d encourage you to think of this as exposing gaps in your diagnosis. For example, when Stripe developed the Sorbet ruby-typing tooling, there was debate between two policies: Should we build a ruby-typing tool to allow a centralized team to gradually migrate the company to a typed codebase? Should we migrate the codebase to a preexisting strongly typed language like Golang or Java? These were, initially, equally valid hypotheses. It was only by clarifying our diagnosis around resourcing that it became clear that incurring the bulk of costs in a centralized team was clearly preferable to spreading the costs across many teams. Specifically, recognizing that we wanted to prioritize short-term product engineering velocity, even if it led to a longer migration overall. If you do develop multiple policy options, I encourage you to move the alternatives into an appendix rather than including them in the core of your strategy document. This will make it easier for readers of your final version to understand how to follow your policies, and they are the most important long-term user of your written strategy. Recognizing constraints A similar problem to competing solutions is developing a policy that you cannot possibly fund. It’s easy to get enamored with policies that you can’t meaningfully enforce, but that’s bad policy, even if it would work in an alternate universe where it was possible to enforce or resource it. To consider a few examples: The strategy for controlling access to user data might have proposed requiring manual approval by a second party of every access to customer data. However, that would have gone nowhere. Our approach to Uber’s service migration might have required more staffing for the infrastructure engineering team, but we knew that wasn’t going to happen, so it was a meaningless policy proposal to make. The strategy for navigating private equity ownership might have argued that new ownership should not hold engineering accountable to a new standard on spending. But they would have just invalidated that strategy in the next financial planning period. If you find a policy that contemplates an impractical approach, it doesn’t only indicate that the policy is a poor one, it also suggests your policy is missing an important pillar. Rather than debating the policy options, the fastest path to resolution is to align on the diagnosis that would invalidate potential paths forward. In cases where aligning on the diagnosis isn’t possible, for example because you simply don’t understand the possibilities of a new technology as encountered in the strategy for adopting LLMs, then you’ve typically found a valuable opportunity to use strategy refinement to build alignment. Dealing with missing strategies At a recent company offsite, we were debating which policies we might adopt to deal with annual plans that kept getting derailed after less than a month. Someone remarked that this would be much easier if we could get the executive team to commit to a clearer, written strategy about which business units we were prioritizing. They were, of course, right. It would be much easier. Unfortunately, it goes back to the problem we discussed in the diagnosis chapter about reframing blockers into diagnosis. If a strategy from the company or a peer function is missing, the empowering thing to do is to include the absence in your diagnosis and move forward. Sometimes, even when you do this, it’s easy to fall back into the belief that you cannot set a policy because a peer function might set a conflicting policy in the future. Whether you’re an executive or an engineer, you’ll never have the details you want to make the ideal policy. Meaningful leadership requires taking meaningful risks, which is never something that gets comfortable. Summary After working through this chapter, you know how to develop policy, how to assemble policies to solve your diagnosis, and how to avoid a number of the frequent challenges that policy writers encounter. At this point, there’s only one phase of strategy left to dig into, operating the policies you’ve created.

17 hours ago 3 votes
Fast and random sampling in SQLite

I was building a small feature for the Flickr Commons Explorer today: show a random selection of photos from the entire collection. I wanted a fast and varied set of photos. This meant getting a random sample of rows from a SQLite table (because the Explorer stores all its data in SQLite). I’m happy with the code I settled on, but it took several attempts to get right. Approach #1: ORDER BY RANDOM() My first attempt was pretty naïve – I used an ORDER BY RANDOM() clause to sort the table, then limit the results: SELECT * FROM photos ORDER BY random() LIMIT 10 This query works, but it was slow – about half a second to sample a table with 2 million photos (which is very small by SQLite standards). This query would run on every request for the homepage, so that latency is unacceptable. It’s slow because it forces SQLite to generate a value for every row, then sort all the rows, and only then does it apply the limit. SQLite is fast, but there’s only so fast you can sort millions of values. I found a suggestion from Stack Overflow user Ali to do a random sort on the id column first, pick my IDs from that, and only fetch the whole row for the photos I’m selecting: SELECT * FROM photos WHERE id IN ( SELECT id FROM photos ORDER BY RANDOM() LIMIT 10 ) This means SQLite only has to load the rows it’s returning, not every row in the database. This query was over three times faster – about 0.15s – but that’s still slower than I wanted. Approach #2: WHERE rowid > (…) Scrolling down the Stack Overflow page, I found an answer by Max Shenfield with a different approach: SELECT * FROM photos WHERE rowid > ( ABS(RANDOM()) % (SELECT max(rowid) FROM photos) ) LIMIT 10 The rowid is a unique identifier that’s used as a primary key in most SQLite tables, and it can be looked up very quickly. SQLite automatically assigns a unique rowid unless you explicitly tell it not to, or create your own integer primary key. This query works by picking a point between the biggest and smallest rowid values used in the table, then getting the rows with rowids which are higher than that point. If you want to know more, Max’s answer has a more detailed explanation. This query is much faster – around 0.0008s – but I didn’t go this route. The result is more like a random slice than a random sample. In my testing, it always returned contiguous rows – 101, 102, 103, … – which isn’t what I want. The photos in the Commons Explorer database were inserted in upload order, so photos with adjacent row IDs were uploaded at around the same time and are probably quite similar. I’d get one photo of an old plane, then nine more photos of other planes. I want more variety! (This behaviour isn’t guaranteed – if you don’t add an ORDER BY clause to a SELECT query, then the order of results is undefined. SQLite is returning rows in rowid order in my table, and a quick Google suggests that’s pretty common, but that may not be true in all cases. It doesn’t affect whether I want to use this approach, but I mention it here because I was confused about the ordering when I read this code.) Approach #3: Select random rowid values outside SQLite Max’s answer was the first time I’d heard of rowid, and it gave me an idea – what if I chose random rowid values outside SQLite? This is a less “pure” approach because I’m not doing everything in the database, but I’m happy with that if it gets the result I want. Here’s the procedure I came up with: Create an empty list to store our sample. Find the highest rowid that’s currently in use: sqlite> SELECT MAX(rowid) FROM photos; 1913389 Use a random number generator to pick a rowid between 1 and the highest rowid: >>> import random >>> random.randint(1, max_rowid) 196476 If we’ve already got this rowid, discard it and generate a new one. (The rowid is a signed, 64-bit integer, so the minimum possible value is always 1.) Look for a row with that rowid: SELECT * FROM photos WHERE rowid = 196476 If such a row exists, add it to our sample. If we have enough items in our sample, we’re done. Otherwise, return to step 3 and generate another rowid. If such a row doesn’t exist, return to step 3 and generate another rowid. This requires a bit more code, but it returns a diverse sample of photos, which is what I really care about. It’s a bit slower, but still plenty fast enough (about 0.001s). This approach is best for tables where the rowid values are mostly contiguous – it would be slower if there are lots of rowids between 1 and the max that don’t exist. If there are large gaps in rowid values, you might try multiple missing entries before finding a valid row, slowing down the query. You might want to try something different, like tracking valid rowid values separately. This is a good fit for my use case, because photos don’t get removed from Flickr Commons very often. Once a row is written, it sticks around, and over 97% of the possible rowid values do exist. Summary Here are the four approaches I tried: Approach Performance (for 2M rows) Notes ORDER BY RANDOM() ~0.5s Slowest, easiest to read WHERE id IN (SELECT id …) ~0.15s Faster, still fairly easy to understand WHERE rowid > ... ~0.0008s Returns clustered results Random rowid in Python ~0.001s Fast and returns varied results, requires code outside SQL I’m using the random rowid in Python in the Commons Explorer, trading code complexity for speed. I’m using this random sample to render a web page, so it’s important that it returns quickly – when I was testing ORDER BY RANDOM(), I could feel myself waiting for the page to load. But I’ve used ORDER BY RANDOM() in the past, especially for asynchronous data pipelines where I don’t care about absolute performance. It’s simpler to read and easier to see what’s going on. Now it’s your turn – visit the Commons Explorer and see what random gems you can find. Let me know if you spot anything cool! [If the formatting of this post looks odd in your feed reader, visit the original article]

8 hours ago 1 votes
Choosing Languages
yesterday 2 votes
05 · Syncing Keyhive

How we sync Keyhive and Automerge

yesterday 1 votes