Full Width [alt+shift+f] Shortcuts [alt+shift+k]
Sign Up [alt+shift+s] Log In [alt+shift+l]
19
While I understand and value the concept of feature detection over browser detection, sometimes the need for knowing whether or not we're dealing with a mobile device arises. For in-depth device checking, you can rely on a complex library such as The MobileESP Project. But for simpler applications, the following snippet can be useful. var isMobile = { Android: function() { return navigator.userAgent.match(/Android/i); }, BlackBerry: function() { return navigator.userAgent.match(/BlackBerry/i); }, iOS: function() { return navigator.userAgent.match(/iPhone|iPad|iPod/i); }, Opera: function() { return navigator.userAgent.match(/Opera Mini/i); }, Windows: function() { return navigator.userAgent.match(/IEMobile/i); }, any: function() { return ( isMobile.Android() || isMobile.BlackBerry() || isMobile.iOS() || isMobile.Opera() || isMobile.Windows() ); } }; While this doesn't account for all mobile platforms, it...
over a year ago

Improve your reading experience

Logged in users get linked directly to articles resulting in a better reading experience. Please login for free, it takes less than 1 minute.

More from A Beautiful Site

Revisiting FOUCE

It's been awhile since I wrote about FOUCE and I've since come up with an improved solution that I think is worth a post. This approach is similar to hiding the page content and then fading it in, but I've noticed it's far less distracting without the fade. It also adds a two second timeout to prevent network issues or latency from rendering an "empty" page. First, we'll add a class called reduce-fouce to the <html> element. <html class="reduce-fouce"> ... </html> Then we'll add this rule to the CSS. <style> html.reduce-fouce { opacity: 0; } </style> Finally, we'll wait until all the custom elements have loaded or two seconds have elapsed, whichever comes first, and we'll remove the class causing the content to show immediately. <script type="module"> await Promise.race([ // Load all custom elements Promise.allSettled([ customElements.whenDefined('my-button'), customElements.whenDefined('my-card'), customElements.whenDefined('my-rating') // ... ]), // Resolve after two seconds new Promise(resolve => setTimeout(resolve, 2000)) ]); // Remove the class, showing the page content document.documentElement.classList.remove('reduce-fouce'); </script> This approach seems to work especially well and won't end up "stranding" the user if network issues occur.

2 months ago 60 votes
If Edgar Allan Poe was into Design Systems

Once upon a midnight dreary, while I pondered, weak and weary, While I nodded, nearly napping, suddenly there came a tapping, "'Tis a design system," I muttered, "bringing order to the core— Ah, distinctly I remember, every button, every splendor, Each component, standardized, like a raven's watchful eyes, Unified in system's might, like patterns we restore— And each separate style injection, linked with careful introspection, 'Tis a design system, nothing more.

3 months ago 65 votes
Web Components Are Not the Future — They’re the Present

It’s disappointing that some of the most outspoken individuals against Web Components are framework maintainers. These individuals are, after all, in some of the best positions to provide valuable feedback. They have a lot of great ideas! Alas, there’s little incentive for them because standards evolve independently and don’t necessarily align with framework opinions. How could they? Opinions are one of the things that make frameworks unique. And therein lies the problem. If you’re convinced that your way is the best and only way, it’s natural to feel disenchanted when a decision is made that you don’t fully agree with. This is my open response to Ryan Carniato’s post from yesterday called “Web Components Are Not the Future.” WTF is a component anyway? # The word component is a loaded term, but I like to think of it in relation to interoperability. If I write a component in Framework A, I would like to be able to use it in Framework B, C, and D without having to rewrite it or include its entire framework. I don’t think many will disagree with that objective. We’re not there yet, but the road has been paved and instead of learning to drive on it, frameworks are building…different roads. Ryan states: If the sheer number of JavaScript frameworks is any indicator we are nowhere near reaching a consensus on how one should author components on the web. And even if we were a bit closer today we were nowhere near there a decade ago. The thing is, we don’t need to agree on how to write components, we just need to agree on the underlying implementation, then you can use classes, hooks, or whatever flavor you want to create them. Turns out, we have a very well-known, ubiquitous technology that we’ve chosen to do this with: HTML. But it also can have a negative effect. If too many assumptions are made it becomes harder to explore alternative space because everything gravitates around the establishment. What is more established than a web standard that can never change? If the concern is premature standardization, well, it’s a bit late for that. So let’s figure out how to get from where we are now to where we want to be. The solution isn’t to start over at the specification level, it’s to rethink how front end frameworks engage with current and emerging standards and work to improve them. Respectfully, it’s time to stop complaining, move on, and fix the things folks perceive as suboptimal. The definition of component # That said, we also need to realize that Web Components aren’t a 1:1 replacement for framework components. They’re tangentially related things, and I think a lot of confusion stems from this. We should really fix the definition of component. So the fundamental problem with Web Components is that they are built on Custom Elements. Elements !== Components. More specifically, Elements are a subset of Components. One could argue that every Element could be a Component but not all Components are Elements. To be fair, I’ve never really liked the term “Web Components” because it competes with the concept of framework components, but that’s what caught on and that's what most people are familiar with these days. Alas, there is a very important distinction here. Sure, a button and a text field can be components, but there are other types. For example, many frameworks support a concept of renderless components that exist in your code, but not in the final HTML. You can’t do that with Web Components, because every custom element results in an actual DOM element. (FWIW I don’t think this is a bad thing — but I digress…) As to why Web components don’t do all the things framework components do, that’s because they’re a lower level implementation of an interoperable element. They’re not trying to do everything framework components do. That’s what frameworks are for. It’s ok to be shiny # In fact, this is where frameworks excel. They let you go above and beyond what the platform can do on its own. I fully support this trial-and-error way of doing things. After all, it’s fun to explore new ideas and live on the bleeding edge. We got a lot of cool stuff from doing that. We got document.querySelector() from jQuery. CSS Custom Properties were inspired by Sass. Tagged template literals were inspired by JSX. Soon we’re getting signals from Preact. And from all the component-based frameworks that came before them, we got Web Components: custom HTML elements that can be authored in many different ways (because we know people like choices) and are fully interoperable (if frameworks and metaframeworks would continue to move towards the standard instead of protecting their own). Frameworks are a testbed for new ideas that may or may not work out. We all need to be OK with that. Even framework authors. Especially framework authors. More importantly, we all need to stop being salty when our way isn’t what makes it into the browser. There will always be a better way to do something, but none of us have the foresight to know what a perfect solution looks like right now. Hindsight is 20/20. As humans, we’re constantly striving to make things better. We’re really good at it, by the way. But we must have the discipline to reach various checkpoints to pause, reflect, and gather feedback before continuing. Even the cheapest cars on the road today will outperform the Model T in every way. I’m sure Ford could have made the original Model T way better if they had spent another decade working on it, but do you know made the next version even better than 10 more years? The feedback they got from actual users who bought them, sat in them, and drove them around on actual roads. Web Standards offer a promise of stability and we need to move forward to improve them together. Using one’s influence to rally users against the very platform you’ve built your success on is damaging to both the platform and the community. We need these incredible minds to be less divisive and more collaborative. The right direction # Imagine if we applied the same arguments against HTML early on. What if we never standardized it at all? Would the Web be a better place if every site required a specific browser? (Narrator: it wasn't.) Would it be better if every site was Flash or a Java applet? (Remember Silverlight? lol) Sure, there are often better alternatives for every use case, but we have to pick something that works for the majority, then we can iterate on it. Web Components are a huge step in the direction of standardization and we should all be excited about that. But the Web Component implementation isn’t compatible with existing frameworks, and therein lies an existential problem. Web Components are a threat to the peaceful, proprietary way of life for frameworks that have amassed millions of users — the majority of web developers. Because opinions vary so wildly, when a new standard emerges frameworks can’t often adapt to them without breaking changes. And breaking changes can be detrimental to a user base. Have you spotted the issue? You can’t possibly champion Web Standards when you’ve built a non-standard thing that will break if you align with the emerging standard. It’s easier to oppose the threat than to adapt to it. And of course Web Components don’t do everything a framework does. How can the platform possibly add all the features every framework added last week? That would be absolutely reckless. And no, the platform doesn’t move as fast as your framework and that’s sometimes painful. But it’s by design. This process is what gives us APIs that continue to work for decades. As users, we need to get over this hurdle and start thinking about how frameworks can adapt to current standards and how to evolve them as new ones emerge. Let’s identify shortcomings in the spec and work together to improve the ecosystem instead of arguing about who’s shit smells worse. Reinventing the wheel isn’t the answer. Lock-in isn’t the answer. This is why I believe that next generation of frameworks will converge on custom elements as an interoperable component model, enhance that model by sprinkling in awesome features of their own, and focus more on flavors (class-based, functional, signals, etc.) and higher level functionality. As for today's frameworks? How they adapt will determine how relevant they remain. Living dangerously # Ryan concludes: So in a sense there are nothing wrong with Web Components as they are only able to be what they are. It's the promise that they are something that they aren't which is so dangerous. The way their existence warps everything around them that puts the whole web at risk. It's a price everyone has to pay. So Web Components aren’t the specific vision you had for components. That's fine. But that's how it is. They're not Solid components. They’re not React components. They’re not Svelte components. They’re not Vue components. They’re standards-based Web Components that work in all of the above. And they could work even better in all of the above if all of the above were interested in advancing the platform instead of locking users in. I’m not a conspiracy theorist, but I find interesting the number of people who are and have been sponsored and/or hired by for-profit companies whose platforms rely heavily on said frameworks. Do you think it’s in their best interest to follow Web Standards if that means making their service less relevant and less lucrative? Of course not. If you’ve built an empire on top of something, there’s absolutely zero incentive to tear it down for the betterment of humanity. That’s not how capitalism works. It’s far more profitable to lock users in and keep them paying. But you know what…? Web Standards don't give a fuck about monetization. Longevity supersedes ingenuity # The last thing I’d like to talk about is this line here. Web Components possibly pose the biggest risk to the future of the web that I can see. Of course, this is from the perspective of a framework author, not from the people actually shipping and maintaining software built using these frameworks. And the people actually shipping software are the majority, but that’s not prestigious so they rarely get the high follower counts. The people actually shipping software are tired of framework churn. They're tired of shit they wrote last month being outdated already. They want stability. They want to know that the stuff they build today will work tomorrow. As history has proven, no framework can promise that. You know what framework I want to use? I want a framework that aligns with the platform, not one that replaces it. I want a framework that values incremental innovation over user lock-in. I want a framework that says it's OK to break things if it means making the Web a better place for everyone. Yes, that comes at a cost, but almost every good investment does, and I would argue that cost will be less expensive than learning a new framework and rebuilding buttons for the umpteenth time. The Web platform may not be perfect, but it continuously gets better. I don’t think frameworks are bad but, as a community, we need to recognize that a fundamental piece of the platform has changed and it's time to embrace the interoperable component model that Web Component APIs have given us…even if that means breaking things to get there. The component war is over.

5 months ago 62 votes
Component Machines

Components are like little machines. You build them once. Use them whenever you need them. Every now and then you open them up to oil them or replace a part, then you send them back to work. And work, they do. Little component machines just chugging along so you never have to write them from scratch ever again. Adapted from this tweet.

6 months ago 57 votes
Styling Custom Elements Without Reflecting Attributes

I've been struggling with the idea of reflecting attributes in custom elements and when it's appropriate. I think I've identified a gap in the platform, but I'm not sure exactly how we should fill it. I'll explain with an example. Let's say I want to make a simple badge component with primary, secondary, and tertiary variants. <my-badge variant="primary">foo</my-badge> <my-badge variant="secondary">bar</my-badge> <my-badge variant="tertiary">baz</my-badge> This is a simple component, but one that demonstrates the problem well. I want to style the badge based on the variant property, but sprouting attributes (which occurs as a result of reflecting a property back to an attribute) is largely considered a bad practice. A lot of web component libraries do it out of necessary to facilitate styling — including Shoelace — but is there a better way? The problem # I need to style the badge without relying on reflected attributes. This means I can't use :host([variant="..."]) because the attribute may or may not be set by the user. For example, if the component is rendered in a framework that sets properties instead of attributes, or if the property is set or changed programmatically, the attribute will be out of sync and my styles will be broken. So how can I style the badge based its variants without reflection? Let's assume we have the following internals, which is all we really need for the badge. <my-badge> #shadowRoot <slot></slot> </my-badge> What can we do about it? # I can't add classes to the slot, because :host(:has(.slot-class)) won't match. I can't set a data attribute on the host element, because that's the same as reflection and might cause issues with SSR and DOM morphing libraries. I could add a wrapper element around the slot and apply classes to it, but I'd prefer not to bloat the internals with additional elements. With a wrapper, users would have to use ::part(wrapper) to target it. Without the wrapper, they can set background, border, and other CSS properties directly on the host element which is more desirable. I could add custom states for each variant, but this gets messy for non-Boolean values and feels like an abuse of the API. Filling the gap # I'm not sure what the best solution is or could be, but one thing that comes to mind is a way to provide some kind of cross-root version of :has that works with :host. Something akin to: :host(:has-in-shadow-root(.some-selector)) { /* maybe one day… */ } If you have any thoughts on this one, hit me up on Twitter.

9 months ago 60 votes

More in programming

ChatGPT Would be a Decent Policy Advisor

Revealed: How the UK tech secretary uses ChatGPT for policy advice by Chris Stokel-Walker for the New Scientist

16 hours ago 3 votes
Setting policy for strategy.

This book’s introduction started by defining strategy as “making decisions.” Then we dug into exploration, diagnosis, and refinement: three chapters where you could argue that we didn’t decide anything at all. Clarifying the problem to be solved is the prerequisite of effective decision making, but eventually decisions do have to be made. Here in this chapter on policy, and the following chapter on operations, we finally start to actually make some decisions. In this chapter, we’ll dig into: How we define policy, and how setting policy differs from operating policy as discussed in the next chapter The structured steps for setting policy How many policies should you set? Is it preferable to have one policy, many policies, or does it not matter much either way? Recurring kinds of policies that appear frequently in strategies Why it’s valuable to be intentional about your strategy’s altitude, and how engineers and executives generally maintain different altitudes in their strategies Criteria to use for evaluating whether your policies are likely to be impactful How to develop novel policies, and why it’s rare Why having multiple bundles of alternative policies is generally a phase in strategy development that indicates a gap in your diagnosis How policies that ignore constraints sound inspirational, but accomplish little Dealing with ambiguity and uncertainty created by missing strategies from cross-functional stakeholders By the end, you’ll be ready to evaluate why an existing strategy’s policies are struggling to make an impact, and to start iterating on policies for strategy of your own. This is an exploratory, draft chapter for a book on engineering strategy that I’m brainstorming in #eng-strategy-book. As such, some of the links go to other draft chapters, both published drafts and very early, unpublished drafts. What is policy? Policy is interpreting your diagnosis into a concrete plan. That plan will be a collection of decisions, tradeoffs, and approaches. They’ll range from coding practices, to hiring mandates, to architectural decisions, to guidance about how choices are made within your organization. An effective policy solves the entirety of the strategy’s diagnosis, although the diagnosis itself is encouraged to specify which aspects can be ignored. For example, the strategy for working with private equity ownership acknowledges in its diagnosis that they don’t have clear guidance on what kind of reduction to expect: Based on general practice, it seems likely that our new Private Equity ownership will expect us to reduce R&D headcount costs through a reduction. However, we don’t have any concrete details to make a structured decision on this, and our approach would vary significantly depending on the size of the reduction. Faced with that uncertainty, the policy simply acknowledges the ambiguity and commits to reconsider when more information becomes available: We believe our new ownership will provide a specific target for Research and Development (R&D) operating expenses during the upcoming financial year planning. We will revise these policies again once we have explicit targets, and will delay planning around reductions until we have those numbers to avoid running two overlapping processes. There are two frequent points of confusion when creating policies that are worth addressing directly: Policy is a subset of strategy, rather than the entirety of strategy, because policy is only meaningful in the context of the strategy’s diagnosis. For example, the “N-1 backfill policy” makes sense in the context of new, private equity ownership. The policy wouldn’t work well in a rapidly expanding organization. Any strategy without a policy is useless, but you’ll also find policies without context aren’t worth much either. This is particularly unfortunate, because so often strategies are communicated without those critical sections. Policy describes how tradeoffs should be made, but it doesn’t verify how the tradeoffs are actually being made in practice. The next chapter on operations covers how to inspect an organization’s behavior to ensure policies are followed. When reworking a strategy to be more readable, it often makes sense to merge policy and operation sections together. However, when drafting strategy it’s valuable to keep them separate. Yes, you might use a weekly meeting to review whether the policy is being followed, but whether it’s an effective policy is independent of having such a meeting, and what operational mechanisms you use will vary depending on the number of policies you intend to implement. With this definition in mind, now we can move onto the more interesting discussion of how to set policy. How to set policy Every part of writing a strategy feels hard when you’re doing it, but I personally find that writing policy either feels uncomfortably easy or painfully challenging. It’s never a happy medium. Fortunately, the exploration and diagnosis usually come together to make writing your policy simple: although sometimes that simple conclusion may be a difficult one to swallow. The steps I follow to write a strategy’s policy are: Review diagnosis to ensure it captures the most important themes. It doesn’t need to be perfect, but it shouldn’t have omissions so obvious that you can immediately identify them. Select policies that address the diagnosis. Explicitly match each policy to one or more diagnoses that it addresses. Continue adding policies until every diagnosis is covered. This is a broad instruction, but it’s simpler than it sounds because you’ll typically select from policies identified during your exploration phase. However, there certainly is space to tweak those policies, and to reapply familiar policies to new circumstances. If you do find yourself developing a novel policy, there’s a later section in this chapter, Developing novel policies, that addresses that topic in more detail. Consolidate policies in cases where they overlap or adjoin. For example, two policies about specific teams might be generalized into a policy about all teams in the engineering organization. Backtest policy against recent decisions you’ve made. This is particularly effective if you maintain a decision log in your organization. Mine for conflict once again, much as you did in developing your diagnosis. Emphasize feedback from teams and individuals with a different perspective than your own, but don’t wholly eliminate those that you agree with. Just as it’s easy to crowd out opposing views in diagnosis if you don’t solicit their input, it’s possible to accidentally crowd out your own perspective if you anchor too much on others’ perspectives. Consider refinement if you finish writing, and you just aren’t sure your approach works – that’s fine! Return to the refinement phase by deploying one of the refinement techniques to increase your conviction. Remember that we talk about strategy like it’s done in one pass, but almost all real strategy takes many refinement passes. The steps of writing policy are relatively pedestrian, largely because you’ve done so much of the work already in the exploration, diagnosis, and refinement steps. If you skip those phases, you’d likely follow the above steps for writing policy, but the expected quality of the policy itself would be far lower. How many policies? Addressing the entirety of the diagnosis is often complex, which is why most strategies feature a set of policies rather than just one. The strategy for decomposing a monolithic application is not one policy deciding not to decompose, but a series of four policies: Business units should always operate in their own code repository and monolith. New integrations across business unit monoliths should be done using gRPC. Except for new business unit monoliths, we don’t allow new services. Merge existing services into business-unit monoliths where you can. Four isn’t universally the right number either. It’s simply the number that was required to solve that strategy’s diagnosis. With an excellent diagnosis, your policies will often feel inevitable, and perhaps even boring. That’s great: what makes a policy good is that it’s effective, not that it’s novel or inspiring. Kinds of policies While there are so many policies you can write, I’ve found they generally fall into one of four major categories: approvals, allocations, direction, and guidance. This section introduces those categories. Approvals define the process for making a recurring decision. This might require invoking an architecture advice process, or it might require involving an authority figure like an executive. In the Index post-acquisition integration strategy, there were a number of complex decisions to be made, and the approval mechanism was: Escalations come to paired leads: given our limited shared context across teams, all escalations must come to both Stripe’s Head of Traffic Engineering and Index’s Head of Engineering. This allowed the acquired and acquiring teams to start building trust between each other by ensuring both were consulted before any decision was finalized. On the other hand, the user data access strategy’s approval strategy was more focused on managing corporate risk: Exceptions must be granted in writing by CISO. While our overarching Engineering Strategy states that we follow an advisory architecture process as described in Facilitating Software Architecture, the customer data access policy is an exception and must be explicitly approved, with documentation, by the CISO. Start that process in the #ciso channel. These two different approval processes had different goals, so they made tradeoffs differently. There are so many ways to tweak approval, allowing for many different tradeoffs between safety, productivity, and trust. Allocations describe how resources are split across multiple potential investments. Allocations are the most concrete statement of organizational priority, and also articulate the organization’s belief about how productivity happens in teams. Some companies believe you go fast by swarming more people onto critical problems. Other companies believe you go fast by forcing teams to solve problems without additional headcount. Both can work, and teach you something important about the company’s beliefs. The strategy on Uber’s service migration has two concrete examples of allocation policies. The first describes the Infrastructure engineering team’s allocation between manual provision tasks and investing into creating a self-service provisioning platform: Constrain manual provisioning allocation to maximize investment in self-service provisioning. The service provisioning team will maintain a fixed allocation of one full time engineer on manual service provisioning tasks. We will move the remaining engineers to work on automation to speed up future service provisioning. This will degrade manual provisioning in the short term, but the alternative is permanently degrading provisioning by the influx of new service requests from newly hired product engineers. The second allocation policy is implicitly noted in this strategy’s diagnosis, where it describes the allocation policy in the Engineering organization’s higher altitude strategy: Within infrastructure engineering, there is a team of four engineers responsible for service provisioning today. While our organization is growing at a similar rate as product engineering, none of that additional headcount is being allocated directly to the team working on service provisioning. We do not anticipate this changing. Allocation policies often create a surprising amount of clarity for the team, and I include them in almost every policy I write either explicitly, or implicitly in a higher altitude strategy. Direction provides explicit instruction on how a decision must be made. This is the right tool when you know where you want to go, and exactly the way that you want to get there. Direction is appropriate for problems you understand clearly, and you value consistency more than empowering individual judgment. Direction works well when you need an unambiguous policy that doesn’t leave room for interpretation. For example, Calm’s policy for working in the monolith: We write all code in the monolith. It has been ambiguous if new code (especially new application code) should be written in our JavaScript monolith, or if all new code must be written in a new service outside of the monolith. This is no longer ambiguous: all new code must be written in the monolith. In the rare case that there is a functional requirement that makes writing in the monolith implausible, then you should seek an exception as described below. In that case, the team couldn’t agree on what should go into the monolith. Individuals would often make incompatible decisions, so creating consistency required removing personal judgment from the equation. Sometimes judgment is the issue, and sometimes consistency is difficult due to misaligned incentives. A good example of this comes in strategy on working with new Private Equity ownership: We will move to an “N-1” backfill policy, where departures are backfilled with a less senior level. We will also institute a strict maximum of one Principal Engineer per business unit. It’s likely that hiring managers would simply ignore this backfill policy if it was stated more softly, although sometimes less forceful policies are useful. Guidance provides a recommendation about how a decision should be made. Guidance is useful when there’s enough nuance, ambiguity, or complexity that you can explain the desired destination, but you can’t mandate the path to reaching it. One example of guidance comes from the Index acquisition integration strategy: Minimize changes to tokenization environment: because point-of-sale devices directly work with customer payment details, the API that directly supports the point-of-sale device must live within our secured environment where payment details are stored. However, any other functionality must not be added to our tokenization environment. This might read like direction, but it’s clarifying the desired outcome of avoiding unnecessary complexity in the tokenization environment. However, it’s not able to articulate what complexity is necessary, so ultimately it’s guidance because it requires significant judgment to interpret. A second example of guidance comes in the strategy on decomposing a monolithic codebase: Merge existing services into business-unit monoliths where you can. We believe that each choice to move existing services back into a monolith should be made “in the details” rather than from a top-down strategy perspective. Consequently, we generally encourage teams to wind down their existing services outside of their business unit’s monolith, but defer to teams to make the right decision for their local context. This is another case of knowing the desired outcome, but encountering too much uncertainty to direct the team on how to get there. If you ask five engineers about whether it’s possible to merge a given service back into a monolithic codebase, they’ll probably disagree. That’s fine, and highlights the value of guidance: it makes it possible to make incremental progress in areas where more concrete direction would cause confusion. When you’re working on a strategy’s policy section, it’s important to consider all of these categories. Which feel most natural to use will vary depending on your team and role, but they’re all usable: If you’re a developer productivity team, you might have to lean heavily on guidance in your policies and increased support for that guidance within the details of your platform. If you’re an executive, you might lean heavily on direction. Indeed, you might lean too heavily on direction, where guidance often works better for areas where you understand the direction but not the path. If you’re a product engineering organization, you might have to narrow the scope of your direction to the engineers within that organization to deal with the realities of complex cross-organization dynamics. Finally, if you have a clear approach you want to take that doesn’t fit cleanly into any of these categories, then don’t let this framework dissuade you. Give it a try, and adapt if it doesn’t initially work out. Maintaining strategy altitude The chapter on when to write engineering strategy introduced the concept of strategy altitude, which is being deliberate about where certain kinds of policies are created within your organization. Without repeating that section in its entirety, it’s particularly relevant when you set policy to consider how your new policies eliminate flexibility within your organization. Consider these two somewhat opposing strategies: Stripe’s Sorbet strategy only worked in an organization that enforced the use of a single programming language across (essentially) all teams Uber’s service migration strategy worked well in an organization that was unwilling to enforce consistent programming language adoption across teams Stripe’s organization-altitude policy took away the freedom of individual teams to select their preferred technology stack. In return, they unlocked the ability to centralize investment in a powerful way. Uber went the opposite way, unlocking the ability of teams to pick their preferred technology stack, while significantly reducing their centralized teams’ leverage. Both altitudes make sense. Both have consequences. Criteria for effective policies In The Engineering Executive’s Primer’s chapter on engineering strategy, I introduced three criteria for evaluating policies. They ought to be applicable, enforced, and create leverage. Defining those a bit: Applicable: it can be used to navigate complex, real scenarios, particularly when making tradeoffs. Enforced: teams will be held accountable for following the guiding policy. Create Leverage: create compounding or multiplicative impact. The last of these three, create leverage, made sense in the context of a book about engineering executives, but probably doesn’t make as much sense here. Some policies certainly should create leverage (e.g. empower developer experience team by restricting new services), but others might not (e.g. moving to an N-1 backfill policy). Outside the executive context, what’s important isn’t necessarily creating leverage, but that a policy solves for part of the diagnosis. That leaves the other two–being applicable and enforced–both of which are necessary for a policy to actually address the diagnosis. Any policy which you can’t determine how to apply, or aren’t willing to enforce, simply won’t be useful. Let’s apply these criteria to a handful of potential policies. First let’s think about policies we might write to improve the talent density of our engineering team: “We only hire world-class engineers.” This isn’t applicable, because it’s unclear what a world-class engineer means. Because there’s no mutually agreeable definition in this policy, it’s also not consistently enforceable. “We only hire engineers that get at least one ‘strong yes’ in scorecards.” This is applicable, because there’s a clear definition. This is enforceable, depending on the willingness of the organization to reject seemingly good candidates who don’t happen to get a strong yes. Next, let’s think about a policy regarding code reuse within a codebase: “We follow a strict Don’t Repeat Yourself policy in our codebase.” There’s room for debate within a team about whether two pieces of code are truly duplicative, but this is generally applicable. Because there’s room for debate, it’s a very context specific determination to decide how to enforce a decision. “Code authors are responsible for determining if their contributions violate Don’t Repeat Yourself, and rewriting them if they do.” This is much more applicable, because now there’s only a single person’s judgment to assess the potential repetition. In some ways, this policy is also more enforceable, because there’s no longer any ambiguity around who is deciding whether a piece of code is a repetition. The challenge is that enforceability now depends on one individual, and making this policy effective will require holding individuals accountable for the quality of their judgement. An organization that’s unwilling to distinguish between good and bad judgment won’t get any value out of the policy. This is a good example of how a good policy in one organization might become a poor policy in another. If you ever find yourself wanting to include a policy that for some reason either can’t be applied or can’t be enforced, stop to ask yourself what you’re trying to accomplish and ponder if there’s a different policy that might be better suited to that goal. Developing novel policies My experience is that there are vanishingly few truly novel policies to write. There’s almost always someone else has already done something similar to your intended approach. Calm’s engineering strategy is such a case: the details are particular to the company, but the general approach is common across the industry. The most likely place to find truly novel policies is during the adoption phase of a new widespread technology, such as the rise of ubiquitous mobile phones, cloud computing, or large language models. Even then, as explored in the strategy for adopting large-language models, the new technology can be engaged with as a generic technology: Develop an LLM-backed process for reactivating departed and suspended drivers in mature markets. Through modeling our driver lifecycle, we determined that improving onboarding time will have little impact on the total number of active drivers. Instead, we are focusing on mechanisms to reactivate departed and suspended drivers, which is the only opportunity to meaningfully impact active drivers. You could simply replace “LLM” with “data-driven” and it would be equally readable. In this way, policy can generally sidestep areas of uncertainty by being a bit abstract. This avoids being overly specific about topics you simply don’t know much about. However, even if your policy isn’t novel to the industry, it might still be novel to you or your organization. The steps that I’ve found useful to debug novel policies are the same steps as running a condensed version of the strategy process, with a focus on exploration and refinement: Collect a number of similar policies, with a focus on how those policies differ from the policy you are creating Create a systems model to articulate how this policy will work, and also how it will differ from the similar policies you’re considering Run a strategy testing cycle for your proto-policy to discover any unknown-unknowns about how it works in practice Whether you run into this scenario is largely a function of the extent of your, and your organization’s, experience. Early in my career, I found myself doing novel (for me) strategy work very frequently, and these days I rarely find myself doing novel work, instead focusing on adaptation of well-known policies to new circumstances. Are competing policy proposals an anti-pattern? When creating policy, you’ll often have to engage with the question of whether you should develop one preferred policy or a series of potential strategies to pick from. Developing these is a useful stage of setting policy, but rather than helping you refine your policy, I’d encourage you to think of this as exposing gaps in your diagnosis. For example, when Stripe developed the Sorbet ruby-typing tooling, there was debate between two policies: Should we build a ruby-typing tool to allow a centralized team to gradually migrate the company to a typed codebase? Should we migrate the codebase to a preexisting strongly typed language like Golang or Java? These were, initially, equally valid hypotheses. It was only by clarifying our diagnosis around resourcing that it became clear that incurring the bulk of costs in a centralized team was clearly preferable to spreading the costs across many teams. Specifically, recognizing that we wanted to prioritize short-term product engineering velocity, even if it led to a longer migration overall. If you do develop multiple policy options, I encourage you to move the alternatives into an appendix rather than including them in the core of your strategy document. This will make it easier for readers of your final version to understand how to follow your policies, and they are the most important long-term user of your written strategy. Recognizing constraints A similar problem to competing solutions is developing a policy that you cannot possibly fund. It’s easy to get enamored with policies that you can’t meaningfully enforce, but that’s bad policy, even if it would work in an alternate universe where it was possible to enforce or resource it. To consider a few examples: The strategy for controlling access to user data might have proposed requiring manual approval by a second party of every access to customer data. However, that would have gone nowhere. Our approach to Uber’s service migration might have required more staffing for the infrastructure engineering team, but we knew that wasn’t going to happen, so it was a meaningless policy proposal to make. The strategy for navigating private equity ownership might have argued that new ownership should not hold engineering accountable to a new standard on spending. But they would have just invalidated that strategy in the next financial planning period. If you find a policy that contemplates an impractical approach, it doesn’t only indicate that the policy is a poor one, it also suggests your policy is missing an important pillar. Rather than debating the policy options, the fastest path to resolution is to align on the diagnosis that would invalidate potential paths forward. In cases where aligning on the diagnosis isn’t possible, for example because you simply don’t understand the possibilities of a new technology as encountered in the strategy for adopting LLMs, then you’ve typically found a valuable opportunity to use strategy refinement to build alignment. Dealing with missing strategies At a recent company offsite, we were debating which policies we might adopt to deal with annual plans that kept getting derailed after less than a month. Someone remarked that this would be much easier if we could get the executive team to commit to a clearer, written strategy about which business units we were prioritizing. They were, of course, right. It would be much easier. Unfortunately, it goes back to the problem we discussed in the diagnosis chapter about reframing blockers into diagnosis. If a strategy from the company or a peer function is missing, the empowering thing to do is to include the absence in your diagnosis and move forward. Sometimes, even when you do this, it’s easy to fall back into the belief that you cannot set a policy because a peer function might set a conflicting policy in the future. Whether you’re an executive or an engineer, you’ll never have the details you want to make the ideal policy. Meaningful leadership requires taking meaningful risks, which is never something that gets comfortable. Summary After working through this chapter, you know how to develop policy, how to assemble policies to solve your diagnosis, and how to avoid a number of the frequent challenges that policy writers encounter. At this point, there’s only one phase of strategy left to dig into, operating the policies you’ve created.

22 hours ago 3 votes
Fast and random sampling in SQLite

I was building a small feature for the Flickr Commons Explorer today: show a random selection of photos from the entire collection. I wanted a fast and varied set of photos. This meant getting a random sample of rows from a SQLite table (because the Explorer stores all its data in SQLite). I’m happy with the code I settled on, but it took several attempts to get right. Approach #1: ORDER BY RANDOM() My first attempt was pretty naïve – I used an ORDER BY RANDOM() clause to sort the table, then limit the results: SELECT * FROM photos ORDER BY random() LIMIT 10 This query works, but it was slow – about half a second to sample a table with 2 million photos (which is very small by SQLite standards). This query would run on every request for the homepage, so that latency is unacceptable. It’s slow because it forces SQLite to generate a value for every row, then sort all the rows, and only then does it apply the limit. SQLite is fast, but there’s only so fast you can sort millions of values. I found a suggestion from Stack Overflow user Ali to do a random sort on the id column first, pick my IDs from that, and only fetch the whole row for the photos I’m selecting: SELECT * FROM photos WHERE id IN ( SELECT id FROM photos ORDER BY RANDOM() LIMIT 10 ) This means SQLite only has to load the rows it’s returning, not every row in the database. This query was over three times faster – about 0.15s – but that’s still slower than I wanted. Approach #2: WHERE rowid > (…) Scrolling down the Stack Overflow page, I found an answer by Max Shenfield with a different approach: SELECT * FROM photos WHERE rowid > ( ABS(RANDOM()) % (SELECT max(rowid) FROM photos) ) LIMIT 10 The rowid is a unique identifier that’s used as a primary key in most SQLite tables, and it can be looked up very quickly. SQLite automatically assigns a unique rowid unless you explicitly tell it not to, or create your own integer primary key. This query works by picking a point between the biggest and smallest rowid values used in the table, then getting the rows with rowids which are higher than that point. If you want to know more, Max’s answer has a more detailed explanation. This query is much faster – around 0.0008s – but I didn’t go this route. The result is more like a random slice than a random sample. In my testing, it always returned contiguous rows – 101, 102, 103, … – which isn’t what I want. The photos in the Commons Explorer database were inserted in upload order, so photos with adjacent row IDs were uploaded at around the same time and are probably quite similar. I’d get one photo of an old plane, then nine more photos of other planes. I want more variety! (This behaviour isn’t guaranteed – if you don’t add an ORDER BY clause to a SELECT query, then the order of results is undefined. SQLite is returning rows in rowid order in my table, and a quick Google suggests that’s pretty common, but that may not be true in all cases. It doesn’t affect whether I want to use this approach, but I mention it here because I was confused about the ordering when I read this code.) Approach #3: Select random rowid values outside SQLite Max’s answer was the first time I’d heard of rowid, and it gave me an idea – what if I chose random rowid values outside SQLite? This is a less “pure” approach because I’m not doing everything in the database, but I’m happy with that if it gets the result I want. Here’s the procedure I came up with: Create an empty list to store our sample. Find the highest rowid that’s currently in use: sqlite> SELECT MAX(rowid) FROM photos; 1913389 Use a random number generator to pick a rowid between 1 and the highest rowid: >>> import random >>> random.randint(1, max_rowid) 196476 If we’ve already got this rowid, discard it and generate a new one. (The rowid is a signed, 64-bit integer, so the minimum possible value is always 1.) Look for a row with that rowid: SELECT * FROM photos WHERE rowid = 196476 If such a row exists, add it to our sample. If we have enough items in our sample, we’re done. Otherwise, return to step 3 and generate another rowid. If such a row doesn’t exist, return to step 3 and generate another rowid. This requires a bit more code, but it returns a diverse sample of photos, which is what I really care about. It’s a bit slower, but still plenty fast enough (about 0.001s). This approach is best for tables where the rowid values are mostly contiguous – it would be slower if there are lots of rowids between 1 and the max that don’t exist. If there are large gaps in rowid values, you might try multiple missing entries before finding a valid row, slowing down the query. You might want to try something different, like tracking valid rowid values separately. This is a good fit for my use case, because photos don’t get removed from Flickr Commons very often. Once a row is written, it sticks around, and over 97% of the possible rowid values do exist. Summary Here are the four approaches I tried: Approach Performance (for 2M rows) Notes ORDER BY RANDOM() ~0.5s Slowest, easiest to read WHERE id IN (SELECT id …) ~0.15s Faster, still fairly easy to understand WHERE rowid > ... ~0.0008s Returns clustered results Random rowid in Python ~0.001s Fast and returns varied results, requires code outside SQL I’m using the random rowid in Python in the Commons Explorer, trading code complexity for speed. I’m using this random sample to render a web page, so it’s important that it returns quickly – when I was testing ORDER BY RANDOM(), I could feel myself waiting for the page to load. But I’ve used ORDER BY RANDOM() in the past, especially for asynchronous data pipelines where I don’t care about absolute performance. It’s simpler to read and easier to see what’s going on. Now it’s your turn – visit the Commons Explorer and see what random gems you can find. Let me know if you spot anything cool! [If the formatting of this post looks odd in your feed reader, visit the original article]

12 hours ago 1 votes
Choosing Languages
yesterday 3 votes
05 · Syncing Keyhive

How we sync Keyhive and Automerge

yesterday 1 votes