More from Eric Bailey
The city next to the town I grew up in had a shopping center that was built atop a sanitary landfill. For those unfamiliar, creating a sanitary landfill involves taking landfill waste, spreading it into thin layers, compacting it, then paving over it. It enjoyed popularity as a practice in the 60s and 70s. I’m all but sure this shopping center was built in the mid-to-late 70s. It was the 90s when I was a kid and started going to it. Of note was a Kmart that I’d frequent with my parents when running errands. At some point, the certain parts of the tile floor of the Kmart started to buckle and bubble—their edges tinged with an ominous yellowish brown color. The reason for this is that is that the landfill was sealed improperly—accumulating methane and carbon dioxide gasses was pushing decomposing waste upwards and outwards. The employees put up some warning signage around the worst of it, but that was pretty much it. They did not have the expertise, nor the incentives to do anything past indicating the problem. One day we tried to go to the Kmart, and it was closed for good. A few years later the entire shopping center had to be demolished and the landfill had to be dug up and disposed of.
I made an app! What Can’t I Press? is an app that allows you to: Read, search, and filter through screen reader keyboard shortcuts. Scan open apps and add their keyboard shortcuts to the collection. Copy a keyboard shortcut to your clipboard, with modifier keys to format the keyboard shortcut in different ways. Export the list of keyboard shortcuts in its current state as JSON. It is intended to be both a reference and a discovery tool. Here’s a quick demo of it in action: Two open apps placed side-by-side on macOS: TextEdit and What Can't I Press? TextEdit is open to a blank document. What Can't I Press? shows a search input, an expand/collapse all toggle button, a list of 4 disclosures in a collapsed state, a notification message, a shortcut filter input, a download icon, and two buttons labeled "Scan all open apps" and "Scan last focused app". The disclosures are labeled, "JAWS", "Narrator", "NVDA", and "VoiceOver" and each has a badge tallying how many shortcuts are present. The notification message reads, "What Can't I Press cannot detect all possible keyboard shortcuts. Be sure to also check manually." The mouse cursor clicks the "Scan last focused app" button and a brief scan happens. The list of disclosures updates to show global keyboard shortcuts and shortcuts for TextEdit. All disclosures are set to an expanded state, and links to jump to the next section are also revealed. The mouse cursor then clicks on the search input, where the word "save" is entered. It then briefly scrolls through the list to show what apps have functionality related to saving. The search input is cleared by clicking a clear button, then focus is placed on the filter input, where Commandkbd> + s is entered. The list of keyboard shortcuts filters to show all shortcuts that incorporate these keystrokes. The cursor then clicks on the Commandkbd> + S save row button, and places focus on the TextEdit document. The content "⌘S" is inserted into the document via pasting. The app is also available as a web experience, and can be installed as a Progressive Web App. This form of the experience only allows viewing and filtering screen reader keyboard shortcuts, as scanning open apps is not possible using current web technologies. Why did I make it? App and webapp keyboard shortcuts are an often overlooked area of user experience, especially when it comes to working nicely with assistive technology. As much as possible we should not put the burden on the person using assistive technology to use workarounds. In other words: what is added as a convenience for someone who doesn’t require assistive technology should not impair someone who does require assistive technology functionality to get what they want or need. I made this app to help speed up and streamline the process of adding keyboard shortcuts in a safe and responsible way. For more background on this act, reference my post, How an accessibility designer adds keyboard shortcuts to a web app. Limitations What Can’t I Press? cannot detect all keyboard shortcuts used by an app. Consider this as a tool that helps as a first pass, but always be sure to also manually check. There are specific ways of declaring keyboard shortcuts in code that can be picked up by a scan, but not everyone uses those methods. Microsoft is an especially poor actor in this regard. The app is also currently unsigned. This means you’ll have to bypass operating system warnings about installing unsigned apps. All app source code is on GitHub and set to public, so you can review exactly what you’re signing up for. Getting the app signed is on the roadmap, so consider this more of a soft-launch announcement. Is it free? Yes. This kinda reads like you think all keyboard shortcuts should be rebindable Also yes. Looks like you used a LLM to make this A third yes. I honestly couldn’t have made this without it, and I have complicated feelings about that. Believe me, there was a ton of human intervention required. What’s with the icon? It’s a stovetop gas burner. You shouldn’t press it with your hand. I’m really funny, I know. And what about this anecdote that I couldn’t resist sharing? A friend mentioned one of their coworkers uses an app whose main mode of operation is text entry. For whatever reason, pressing Command + b toggles open a navigation drawer, and not, you know, apply bold formatting. This isn’t even an accessibility issue, it’s straight-up bad design. Is this app helpful? Let me know!
A bold first sentence that draws you in. A steering second sentence to set you further down the path. A third sentence that tantalizes and alludes to content to follow. Following is an initial explanatory paragraph. It serves to help back up the previous paragraphs, and start to ground it in more applicable information. Expectations are set, and potential skepticism is addressed. A link to prior art is supplied, to provide additional context. There is then a paragraph that serves as a segue. It connects the high-level concepts and begins to draw them down the realm of the practical. A subheading to help segment the content This paragraph begins to answer the questions in the reader’s mind that the segue paragraph introduced. It begins with level-setting. Certain key concepts are bolded to accommodate people who skim. Following that is supplying more context, albeit still at a relatively high level. A short sentence isolated as its own paragraph to drive attention and impact. The following paragraph begins to dive into particulars. It introduces a new concept related to the subsection’s topic at-hand, careful to stay focused to help the reader understand the larger goal you’re driving towards. Bulleted lists help the reader digest these particulars, Break up the flow of content, and Step through a process. A follow-up paragraph at the end of a subsection may allude to the author‘s opinions or larger thoughts about the topic. It also sets up the next subsection. Another subheading The next concept is addressed, getting more technical as the reader becomes more acclimated. A link to a peer resource is threaded in as an appeal to authority, to help reinforce confidence in the author. An ordered list is used to: Communicate a series of instructions the reader should take, In which order they should be followed, and Do so in a way that both makes it easy to follow and also breaks up reading flow to be more noticeable. There might then be another follow-up paragraph. This one might contain a sentence with an em dash—indicative of a trailing thought that is still topically related. A deeper subheading This section works deeper into the technical topic. It dispenses with reasoning and analogies and discusses practical specifics. A piece of code that translates the practical specifics into language a computer can be instructed with. Comments are supplied to help facilitate understanding. A bulleted list is used, It breaks down and explains code concepts that may not be self-evident, and These points may be more holistic, meaning they aren’t a good fit for inline code comments. A new concept is introduced We break out of the depth of the deeper subheading’s content and return to a level higher. This is a new concept, yet still living under the umbrella of the overall topic at-hand. The format of this new concept mirrors the structure of the previous section. This predictability helps with reading flow and answering the reader’s unspoken expectations. A subheading that begins to tie all the previous sections’ content together This subsection takes each of the previous points raised and reinforces their need to be sequentially discussed. It then explicitly confirms the case the rest of the content has been implicitly building, that this topic was worth breaking down to better appreciate as a holistic whole. Some established trust is cashed in. The author is allowed some space to wax philosophical about larger implications, or discuss their feelings on the matter. A conclusion The bold first sentence is revisited now that the reader has completed learning about the concept being discussed. A subsequent sentence explicitly ties the nuance the rest of the content discusses to the overall point. The reader is thanked, and the content ends.
Hey there, fellow designer! Chances are good you’ve been linked to this after doing some annotation work on a design you've been creating. First off, I want to thank you for taking the time to address accessibility considerations in your work. No, seriously. I mean it. Accessibility-oriented annotations are still a relatively rare thing in our industry. I applaud your efforts to buck that trend, and hope that this post does not dissuade you from future attempts. On focus order There’s a subtle thing at play here, and it mostly revolves around the notion that adding something demonstrates effort and value. For focus order, you’ll actually want to suppress that urge and just let things be. Yes, many accessibility annotation kits do come with the ability to specify focus order. And yes, being able to place focus on the interactive parts of your website or web app is an important thing. However, interactive elements allow you to place focus on them without needing to do anything else. This includes buttons, links, form fields, checkboxes and radio buttons, expand/collapse toggles, tab panels, etc. This is possible if the engineer—or LLM—who builds the user interface (UI) uses semantic elements. This means things like the button element for buttons, the anchor element for links, etc. Also know that it is a whole other set of problems if semantic elements aren’t used. The order itself When it comes to web accessibility, the order interactive elements receive should be a logical order. For the vast majority of situations, this means matching reading order. For example, English means focus order is left-to-right, top-to-bottom. Arabic means right-to-left, top-to-bottom, etc. The easiest way to guarantee focus order matches reading order is to ensure the underlying HTML that powers your web experience is written in top-to-bottom order, where the width of the viewport controls where content flows to the next line. Fortunately, top-to-bottom HTML order is how the majority of the web is still built. If you don’t believe me you can spend a little time poking around with browser developer tooling. I like using the inspect element feature for this because you can highlight UI in the browser tab and see its corresponding code in context of the rest of the page or view. Why is focus important? A consistent, predictable, and logical focus order helps a large number of people use your website or web app. This includes working with assistive technologies such as: Screen readers, Magnification, Voice control, Switches, Etc. It’s also something I’m willing to bet you’ve benefitted from when filling out forms, using Tab to quickly jump from input to input. So, why do annotation kits include focus order options? Honestly, good question. Focus order annotations serve two general use cases: When the HTML order does not match reading order. This sometimes happens as a result of third party code, some relatively obscure CSS layout techniques, or other corner-case scenarios. When you need to do deep custom UI work. There are certain kinds of complicated components and flows that benefit from having the order of focus explicitly illustrated for development handoff. For example, I used them when helping to make GitHub’s interactive list component. What should I do instead? If you want to use your time efficiently: There is a tremendous amount of value in manually evaluating focus order on the developed UI. And it’s easy to perform manual evaluation! All you have to do is press the Tab key and ensure each time you do that: There is a visible focus treatment applied to each interactive piece of UI, and Focus moves in the reading order of the language used for the design, and does not skip anything over. You’ll want to do this on the live, public version of the website or webapp, since that is what the people who use your service will be using. Then file bug reports in the appropriate places if you run into things not working as expected. As a pro-tip, you can also prime your focus so you don’t have to Tab over primary navigation and other templated interactive elements repeated across different pages or views. Given that assumptions are just that, you could also look into explicitly documenting: HTML order should follow reading order, and also Semantic HTML should be used for interactive elements. Be strategic in how you go about this. It might ruffle some feathers, especially in larger and more siloed organizations. Other misconceptions While I have you, here are a few other missteps related to focus order that I’ve encountered. These all break expectations and may confuse and disorient the people who use your service: Manually orchestrating a designed experience for where focus should go via use of tabindex. Making inert, static text focusable to make it more “important”. People already have ways to work with this kind of content. Removing the ability for interactive elements to receive focus in an attempt to direct someone to features the organization prioritizes. Assuming engineers know what component you’re using in your design, and where its documentation lives. Annotating the repeated, templated parts of a page or flow that have already been developed. Again, thank you Meryl K. Evans has a quote about accessibility that I try to live by, and that is “Progress over perfection.” To that point, thank you for being open to reading this post—I hope you don’t feel anger, shame, frustration, or other negative emotions in response to trying to do the right thing. I also hope that you don’t drop your accessibility efforts all together. Instead, know that making the web a little bit more accessible—and therefore a little bit better—adds up over time. This also includes how you learn about the details and nuance of designing for disability. One of the web’s better capabilities is its ability to be revisited and updated. It means that past efforts can be improved on as easily as making something new. In a way, it can also be a form of kindness you can extend to your past self—it’s something I’m trying to do. In closing, I’ve included a list of more resources at the end of this post. I hope it can serve as a springboard to learn more about focus.
More in programming
How can something that “just works” be so annoying? situation We live in Cambridge off a little road down a drive in shared ownership between us and our neighbouring houses. All the utilities are buried under this drive, including the phone line. anticipation Over the last few years we have been canvassed repeatedly by CityFibre saying that they can deliver fibre all way to our house. I saw them digging trenches and leaving tails of purple fibre cladding along nearby roads, ready to hook up all the houses. I thought they would need to do something similar to deliver fibre to us. So when they turned up and knocked on our door, I talked to their salesbods and walked them up and down the drive and pointed out where the existing BT line goes. Then they gave up trying to sell to us. This happened about three times. disaffection We were not eager enough for an upgrade to deal with these impediments. notification A few months ago we were told that CityFibre would soon come and do the upgrade, since there’s a nationwide deadline for turning off the copper phone network at the end of the year. We expected that this would force them to actually plan some digging works, so we talked to our neighbours about it. We were all ready for some huge faff to follow the next visit by the CityFibre bods. installation CityFibre turned up on the promised morning bright and early. To our enormous surprise, a brown fibre housing was already poking out of the ground next to our copper phone line. It had been fed through 50 metres of 5cm duct without us being aware they were even working on the street. Within a couple of hours, the technicians had drilled through our wall, installed the ONT, blown fibre through the unexpected pipe, plugged in the CPE (superficially identical to the old one), and left telling us to anticipate that it might not work properly until tomorrow. activation Around lunch time, the copper phone line stopped working completely. Some faff ensued, switching all our devices over to the new WiFi network. For a while we thought this was the death of our land line, but in the course of debugging other issues, I realised that the router has a built-in VoIP adapter (I don’t think we were told it has a built-in VoIP adapter) so I plugged the phone in and it Just Worked: they had ported our phone number across and everything. Flawless. I was seriously impressed. rumination It has been a few weeks since the switchover, and apart from a couple of horrible Clown-afflicted IoT devices, it has been fairly smooth. What prompted me to write this up was realising that we delayed this upgrade for years because the sales people were not given enough technical information about how the installation process works: the fact that houses typically have a 5cm duct containing the copper lines (probably standard for the last 40 years) and the fact that fibre can be shoved through a few tens of metres without difficulty. And worse, the sales people didn’t have an esclation path for difficult cases: they just gave up instead. From a technical point of view, the installation was impeccable. (I guess the loose 24 hour window for the cutover time was because OpenReach and CityFibre don’t have tight requirements on ISP reconfiguration schedules.) From the sales point of view, it was crap. Maybe it would have gone faster if we offered to switch early without asking if the drive would be a problem? But I guess the difference between “yes!” and “yes, but will this be a problem?” is too much to expect from a minimum-wage door-to-door salesbod whose employer didn’t give them enough information or any escalation path.
I listen to a lot of podcasts, and I like how they fit around other tasks. I press play, lock my phone, and put it down. I’m free to wash the dishes, fold the laundry, or shop for groceries. Unfortunately, more and more information is only published as a video. Technical talks, conference sessions, video essays – they don’t work in an audio-only podcast app. I could convert these videos to MP3 files, but that breaks down the moment a video isn’t pure spoken word. If a speaker says, “Look at this slide” or holds up a diagram, an audio-only file leaves me stranded. I don’t want to give up the podcast player I like, nor stare at a screen for an hour – but I do want the information in these videos. To solve this, I’m abusing my podcast player’s chapter support. This gives me the best of both worlds: I can listen to a video as audio-first, and glance at my lock screen if I need a moment of visual context. The idea: Chapters every few seconds MP3 files can have ID3 metadata, and ID3 metadata can include chapters. A chapter covers a particular time range, and it can have an associated title, description, and cover art. My podcast app of choice is Overcast, which can’t play videos, but it does have robust chapter support. I can jump between chapters, navigate a table of contents, and see per-chapter cover art. To get videos into Overcast, I’m creating MP3 files with a new chapter every few seconds, and the per-chapter cover art is a corresponding frame from the video. As I play the file, I get a slow, stop-motion-like rendition of the original video. If my phone is locked, I can glance at my lock screen and see the current frame in the Now Playing screen. Overcast is developed by Marco Arment, and I got this idea from Forecast, his app for adding chapters to podcasts. In particular, I was struck by its ability to create chapters that don’t display in the chapter list – ideal if I don’t want a table of contents with hundreds of entries. As I was developing my script, I compared my output to the output from Forecast to ensure I was creating the chapters correctly. The code: FFmpeg and Mutagen There are three steps in this process: Convert a video file to an MP3 Extract images from the video at a fixed interval Insert the images as hidden chapters in the MP3 file Let’s go through each in turn. 1. Convert a video file to an MP3 Converting a video file to an MP3 is a single FFmpeg command: ffmpeg -i video.mp4 audio.mp3 This is consistently the slowest step of the process, and I do wonder if I could use different settings or an alternative encoder to make it go faster – but it’s not slow enough to be worth further investigation. 2. Extract images from the video at a fixed interval Extracting images from a video needs a more complicated FFmpeg command: ffmpeg -i video.mp4 \ -vf 'fps=1/5,scale=iw*sar:ih,scale=min(iw\,945):min(ih\,945):force_original_aspect_ratio=decrease' \ thumbnail_%04d.jpg This extracts an image every 5 seconds, downscales any image larger than 945 pixels square (while preserving the original aspect ratio), and saves the results as sequentially numbered JPEG images (thumbnail_0001.png, thumbnail_0002.png, and so on). The key is the -vf flag, which defines two FFmpeg filters: The fps filter selects one frame every 5 seconds (fps=1/5). The first scale filter scales the width based on the sample aspect ratio (scale=iw*sar:ih). Without this filter, frames can be stretched and distorted. The second scale filter scales the input video, preserving the original aspect ratio (force_original_aspect_ratio=decrease), and ensuring the output images fit within 945×945px or the size of the input video, whichever is smaller. My limit is 945 pixels because that’s the largest size that cover art is shown on my iPhone. This filter still isn’t completely correct – it sometimes creates images from portrait videos that are smaller than I’m expecting – but it’s good enough. These are only thumbnails for glancing at, and if I want to change it later, I can always do the image resizing outside FFmpeg. 3. Insert the images as hidden chapters in the MP3 file Inserting the chapters into the MP3 file is more complicated. Although FFmpeg has basic support for ID3 metadata, as far as I know, it can’t insert chapters with per-chapter artwork. Instead, I’m going to reach for Python and the Mutagen library. Here’s the code to add a chapter to an MP3 file: from mutagen.id3 import APIC, CHAP, ID3, PictureType audio = ID3("audio.mp3") with open("thumbnail_0001.jpg", "rb") as f: img_data = f.read() image_frame = APIC(mime="image/jpeg", type=PictureType.OTHER, data=img_data) chapter_frame = CHAP( element_id="chp1", start_time=0, end_time=5 * 1000, sub_frames=[image_frame] ) audio.add(chapter_frame) audio.save() This creates a single chapter that lasts the first 5 seconds (0 to 5000 milliseconds), and the per-chapter cover art is thumbnail_0001.jpg. If we ran this in a loop, we could add images for every 5 second slice of the original video. This code is inserting two frames into the ID3 metadata: The CHAP (chapter) frame contains the timing information, and it can have subframes for metadata like title, chapter art, or associated URL. The APIC (attached picture) subframe contains information about a picture, which can either be a blob of image data or a URL to an image on the web. Normally, you’d also insert a CTOC frame which defines a table of contents, but I don’t want a TOC with hundreds of 5-second chapters, so I’m deliberately not doing this here. This is allowed by the ID3 spec – you’re not required to insert a CTOC frame if you’re using chapters, and you can have chapters that aren’t listed in your table of contents. To work out which frames I needed, I used Forecast to create some chapters by hand, and I inspected their frames. In particular, loading an MP3 and calling Mutagen’s pprint() method shows a human-readable list of frames, and then I could drill into the individual fields: from mutagen.id3 import ID3 audio = ID3("audio.mp3") print(audio.pprint()) I wrapped all this code in a project called glancecast, which allows you to convert a video file with a single command, with optional flags to set the frame length and chapter art size: $ python3 glancecast.py interesting_talk.mp4 interesting_talk.mp3 The process takes a minute or so to complete, most of which is spent transcoding the video file to MP3. The resulting MP3s are usually 40 to 50 MB in size, which is very reasonable. The outcome: How it looks in practice Here’s what one of these “glanceable” podcasts looks like in Overcast and on my lock screen: Maggie Appleton presented this talk over two years ago and it’s been on my “talks to watch” list ever since. Once I put it in Overcast? I listened to it in less than a day. It’s not a lot of extra information, but enough that I can quickly glance down and get the gist of what a speaker is saying. Both views update with a new frame every few seconds, or I can put my phone in my pocket and ignore the screen. I’ve used this approach for half a dozen videos so far, and I’m happy with the results. I expect to keep using it, because I have a long queue of videos I’ve been meaning to watch. If you’d like to try this, check out glancecast for the full code and instructions. [If the formatting of this post looks odd in your feed reader, visit the original article]
Andrew Baker, the current Group CIO at Capitec Bank wrote an interesting piece on AI and open source, and how these tools that generate code according to one’s specification may replace the general reliance on open source implementations done by contributors around the world. I’d really recommend reading it. I have great admiration and respectContinue reading "AI Isn’t Replacing Open Source"
I've mostly given up keeping up with agent trends. Every few months, I ignore all of it and ask what I'm actually getting use out of. Three things…
A framework for thinking about when AI involvement is additive or a violation