Night is falling on Cerro Pachón. Stray clouds reflect the last few rays of golden light as the sun dips below the horizon. I focus my camera across the summit to the westernmost peak of the mountain. Silhouetted within a dying blaze of red and orange light looms the sphinxlike shape of the Vera C. Rubin Observatory. “Not bad,” says William O’Mullane, the observatory’s deputy project manager, amateur photographer, and master of understatement. We watch as the sky fades through reds and purples to a deep, velvety black. It’s my first night in Chile. For O’Mullane, and hundreds of other astronomers and engineers, it’s the culmination of years of work, as the Rubin Observatory is finally ready to go “on sky.” Rubin is unlike any telescope ever built. Its exceptionally wide field of view, extreme speed, and massive digital camera will soon begin the 10-year Legacy Survey of Space and Time (LSST) across the entire southern sky. The result will be a high-resolution movie of how our solar system, galaxy, and universe change over time, along with hundreds of petabytes of data representing billions of celestial objects that have never been seen before. Stars begin to appear overhead, and O’Mullane and I pack up our cameras. It’s astronomical twilight, and after nearly 30 years, it’s time for Rubin to get to work. On 23 June, the Vera C. Rubin Observatory released the first batch of images to the public. One of them, shown here, features a small section of the Virgo cluster of galaxies. Visible are two prominent spiral galaxies (lower right), three merging galaxies (upper right), several groups of distant galaxies, and many stars in the Milky Way galaxy. Created from over 10 hours of observing data, this image represents less than 2 percent of the field of view of a single Rubin image. NSF-DOE Rubin Observatory A second image reveals clouds of gas and dust in the Trifid and Lagoon nebulae, located several thousand light-years from Earth. It combines 678 images taken by the Rubin Observatory over just seven hours, revealing faint details—like nebular gas and dust—that would otherwise be invisible. NSF-DOE Rubin Observatory Engineering the Simonyi Survey Telescope The top of Cerro Pachón is not a big place. Spanning about 1.5 kilometers at 2,647 meters of elevation, its three peaks are home to the Southern Astrophysical Research Telescope (SOAR), the Gemini South Telescope, and for the last decade, the Vera Rubin Observatory construction site. An hour’s flight north of the Chilean capital of Santiago, these foothills of the Andes offer uniquely stable weather. The Humboldt Current flows just offshore, cooling the surface temperature of the Pacific Ocean enough to minimize atmospheric moisture, resulting in some of the best “seeing,” as astronomers put it, in the world. It’s a complicated but exciting time to be visiting. It’s mid-April of 2025, and I’ve arrived just a few days before “first photon,” when light from the night sky will travel through the completed telescope and into its camera for the first time. In the control room on the second floor, engineers and astronomers make plans for the evening’s tests. O’Mullane and I head up into a high bay that contains the silvering chamber for the telescope’s mirrors and a clean room for the camera and its filters. Increasingly exhausting flights of stairs lead to the massive pier on which the telescope sits, and then up again into the dome. I suddenly feel very, very small. The Simonyi Survey Telescope towers above us—350 tonnes of steel and glass, nestled within the 30-meter-wide, 650-tonne dome. One final flight of stairs and we’re standing on the telescope platform. In its parked position, the telescope is pointed at horizon, meaning that it’s looking straight at me as I step in front of it and peer inside. The telescope’s enormous 8.4-meter primary mirror is so flawlessly reflective that it’s essentially invisible. Made of a single piece of low-expansion borosilicate glass covered in a 120-nanometer-thick layer of pure silver, the huge mirror acts as two different mirrors, with a more pronounced curvature toward the center. Standing this close means that different reflections of the mirrors, the camera, and the structure of the telescope all clash with one another in a way that shifts every time I move. I feel like if I can somehow look at it in just the right way, it will all make sense. But I can’t, and it doesn’t. I’m rescued from madness by O’Mullane snapping photos next to me. “Why?” I ask him. “You see this every day, right?” “This has never been seen before,” he tells me. “It’s the first time, ever, that the lens cover has been off the camera since it’s been on the telescope.” Indeed, deep inside the nested reflections I can see a blue circle, the r-band filter within the camera itself. As of today, it’s ready to capture the universe. Rubin’s Wide View Unveils the Universe Back down in the control room, I find director of construction Željko Ivezić. He’s just come up from the summit hotel, which has several dozen rooms for lucky visitors like myself, plus a few even luckier staff members. The rest of the staff commutes daily from the coastal town of La Serena, a 4-hour round trip. To me, the summit hotel seems luxurious for lodgings at the top of a remote mountain. But Ivezić has a slightly different perspective. “The European-funded telescopes,” he grumbles, “have swimming pools at their hotels. And they serve wine with lunch! Up here, there’s no alcohol. It’s an American thing.” He’s referring to the fact that Rubin is primarily funded by the U.S. National Science Foundation and the U.S. Department of Energy’s Office of Science, which have strict safety requirements. Originally, Rubin was intended to be a dark-matter survey telescope, to search for the 85 percent of the mass of the universe that we know exists but can’t identify. In the 1970s, astronomer Vera C. Rubin pioneered a spectroscopic method to measure the speed at which stars orbit around the centers of their galaxies, revealing motion that could be explained only by the presence of a halo of invisible mass at least five times the apparent mass of the galaxies themselves. Dark matter can warp the space around it enough that galaxies act as lenses, bending light from even more distant galaxies as it passes around them. It’s this gravitational lensing that the Rubin observatory was designed to detect on a massive scale. But once astronomers considered what else might be possible with a survey telescope that combined enormous light-collecting ability with a wide field of view, Rubin’s science mission rapidly expanded beyond dark matter. Trading the ability to focus on individual objects for a wide field of view that can see tens of thousands of objects at once provides a critical perspective for understanding our universe, says Ivezić. Rubin will complement other observatories like the Hubble Space Telescope and the James Webb Space Telescope. Hubble’s Wide Field Camera 3 and Webb’s Near Infrared Camera have fields of view of less than 0.05 square degrees each, equivalent to just a few percent of the size of a full moon. The upcoming Nancy Grace Roman Space Telescope will see a bit more, with a field of view of about one full moon. Rubin, by contrast, can image 9.6 square degrees at a time—about 45 full moons’ worth of sky. RELATED: A Trillion Rogue Planets and Not One Sun to Shine on Them That ultrawide view offers essential context, Ivezić explains. “My wife is American, but I’m from Croatia,” he says. “Whenever we go to Croatia, she meets many people. I asked her, ‘Did you learn more about Croatia by meeting many people very superficially, or because you know me very well?’ And she said, ‘You need both. I learn a lot from you, but you could be a weirdo, so I need a control sample.’ ” Rubin is providing that control sample, so that astronomers know just how weird whatever they’re looking at in more detail might be. Every night, the telescope will take a thousand images, one every 34 seconds. After three or four nights, it’ll have the entire southern sky covered, and then it’ll start all over again. After a decade, Rubin will have taken more than 2 million images, generated 500 petabytes of data, and visited every object it can see at least 825 times. In addition to identifying an estimated 6 million bodies in our solar system, 17 billion stars in our galaxy, and 20 billion galaxies in our universe, Rubin’s rapid cadence means that it will be able to delve into the time domain, tracking how the entire southern sky changes on an almost daily basis. Cutting-Edge Technology Behind Rubin’s Speed Achieving these science goals meant pushing the technical envelope on nearly every aspect of the observatory. But what drove most of the design decisions is the speed at which Rubin needs to move (3.5 degrees per second)—the phrase most commonly used by the Rubin staff is “crazy fast.” Crazy fast movement is why the telescope looks the way it does. The squat arrangement of the mirrors and camera centralizes as much mass as possible. Rubin’s oversize supporting pier is mostly steel rather than mostly concrete so that the movement of the telescope doesn’t twist the entire pier. And then there’s the megawatt of power required to drive this whole thing, which comes from huge banks of capacitors slung under the telescope to prevent a brownout on the summit every 30 seconds all night long. Rubin is also unique in that it utilizes the largest digital camera ever built. The size of a small car and weighing 2,800 kilograms, the LSST camera captures 3.2-gigapixel images through six swappable color filters ranging from near infrared to near ultraviolet. The camera’s focal plane consists of 189 4K-by-4K charge-coupled devices grouped into 21 “rafts.” Every CCD is backed by 16 amplifiers that each read 1 million pixels, bringing the readout time for the entire sensor down to 2 seconds flat. Astronomy in the Time Domain As humans with tiny eyeballs and short lifespans who are more or less stranded on Earth, we have only the faintest idea of how dynamic our universe is. To us, the night sky seems mostly static and also mostly empty. This is emphatically not the case. In 1995, the Hubble Space Telescope pointed at a small and deliberately unremarkable part of the sky for a cumulative six days. The resulting image, called the Hubble Deep Field, revealed about 3,000 distant galaxies in an area that represented just one twenty-four-millionth of the sky. To observatories like Hubble, and now Rubin, the sky is crammed full of so many objects that it becomes a problem. As O’Mullane puts it, “There’s almost nothing not touching something.” One of Rubin’s biggest challenges will be deblending—­identifying and then separating things like stars and galaxies that appear to overlap. This has to be done carefully by using images taken through different filters to estimate how much of the brightness of a given pixel comes from each object. At first, Rubin won’t have this problem. At each location, the camera will capture one 30-second exposure before moving on. As Rubin returns to each location every three or four days, subsequent exposures will be combined in a process called coadding. In a coadded image, each pixel represents all of the data collected from that location in every previous image, which results in a much longer effective exposure time. The camera may record only a few photons from a distant galaxy in each individual image, but a few photons per image added together over 825 images yields much richer data. By the end of Rubin’s 10-year survey, the coadding process will generate images with as much detail as a typical Hubble image, but over the entire southern sky. A few lucky areas called “deep drilling fields” will receive even more attention, with each one getting a staggering 23,000 images or more. Rubin will add every object that it detects to its catalog, and over time, the catalog will provide a baseline of the night sky, which the observatory can then use to identify changes. Some of these changes will be movement—Rubin may see an object in one place, and then spot it in a different place some time later, which is how objects like near-Earth asteroids will be detected. But the vast majority of the changes will be in brightness rather than movement. RELATED: Three Steps to Stopping Killer Asteroids Every image that Rubin collects will be compared with a baseline image, and any change will automatically generate a software alert within 60 seconds of when the image was taken. Rubin’s wide field of view means that there will be a lot of these alerts—on the order of 10,000 per image, or 10 million alerts per night. Other automated systems will manage the alerts. Called alert brokers, they ingest the alert streams and filter them for the scientific community. If you’re an astronomer interested in Type Ia supernovae, for example, you can subscribe to an alert broker and set up a filter so that you’ll get notified when Rubin spots one. Many of these alerts will be triggered by variable stars, which cyclically change in brightness. Rubin is also expected to identify somewhere between 3 million and 4 million supernovae—that works out to over a thousand new supernovae for every night of observing. And the rest of the alerts? Nobody knows for sure, and that’s why the alerts have to go out so quickly, so that other telescopes can react to make deeper observations of what Rubin finds. Managing Rubin’s Vast Data Output After the data leaves Rubin’s camera, most of the processing will take place at the SLAC National Accelerator Laboratory in Menlo Park, Calif., over 9,000 kilometers from Cerro Pachón. It takes less than 10 seconds for an image to travel from the focal plane of the camera to SLAC, thanks to a 600-gigabit fiber connection from the summit to La Serena, and from there, a dedicated 100-gigabit line and a backup 40-gigabit line that connect to the Department of Energy’s science network in the United States. The 20 terabytes of data that Rubin will produce nightly makes this bandwidth necessary. “There’s a new image every 34 seconds,” O’Mullane tells me. “If I can’t deal with it fast enough, I start to get behind. So everything has to happen on the cadence of half a minute if I want to keep up with the data flow.” At SLAC, each image will be calibrated and cleaned up, including the removal of satellite trails. Rubin will see a lot of satellites, but since the satellites are unlikely to appear in the same place in every image, the impact on the data is expected to be minimal when the images are coadded. The processed image is compared with a baseline image and any alerts are sent out, by which time processing of the next image has already begun. As Rubin’s catalog of objects grows, astronomers will be able to query it in all kinds of useful ways. Want every image of a particular patch of sky? No problem. All the galaxies of a certain shape? A little trickier, but sure. Looking for 10,000 objects that are similar in some dimension to 10,000 other objects? That might take a while, but it’s still possible. Astronomers can even run their own code on the raw data. “Pretty much everyone in the astronomy community wants something from Rubin,” O’Mullane explains, “and so they want to make sure that we’re treating the data the right way. All of our code is public. It’s on GitHub. You can see what we’re doing, and if you’ve got a better solution, we’ll take it.” One better solution may involve AI. “I think as a community we’re struggling with how we do this,” says O’Mullane. “But it’s probably something we ought to do—curating the data in such a way that it’s consumable by machine learning, providing foundation models, that sort of thing.” The data management system is arguably as much of a critical component of the Rubin observatory as the telescope itself. While most telescopes make targeted observations that get distributed to only a few astronomers at a time, Rubin will make its data available to everyone within just a few days, which is a completely different way of doing astronomy. “We’ve essentially promised that we will take every image of everything that everyone has ever wanted to see,” explains Kevin Reil, Rubin observatory scientist. “If there’s data to be collected, we will try to collect it. And if you’re an astronomer somewhere, and you want an image of something, within three or four days we’ll give you one. It’s a colossal challenge to deliver something on this scale.” The more time I spend on the summit, the more I start to think that the science that we know Rubin will accomplish may be the least interesting part of its mission. And despite their best efforts, I get the sense that everyone I talk to is wildly understating the impact it will have on astronomy. The sheer volume of objects, the time domain, the 10 years of coadded data—what new science will all of that reveal? Astronomers have no idea, because we’ve never looked at the universe in this way before. To me, that’s the most fascinating part of what’s about to happen. Reil agrees. “You’ve been here,” he says. “You’ve seen what we’re doing. It’s a paradigm shift, a whole new way of doing things. It’s still a telescope and a camera, but we’re changing the world of astronomy. I don’t know how to capture—I mean, it’s the people, the intensity, the awesomeness of it. I want the world to understand the beauty of it all.” The Intersection of Science and Engineering Because nobody has built an observatory like Rubin before, there are a lot of things that aren’t working exactly as they should, and a few things that aren’t working at all. The most obvious of these is the dome. The capacitors that drive it blew a fuse the day before I arrived, and the electricians are off the summit for the weekend. The dome shutter can’t open either. Everyone I talk to takes this sort of thing in stride—they have to, because they’ve been troubleshooting issues like these for years. I sit down with Yousuke Utsumi, a camera operations scientist who exudes the mixture of excitement and exhaustion that I’m getting used to seeing in the younger staff. “Today is amazingly quiet,” he tells me. “I’m happy about that. But I’m also really tired. I just want to sleep.” Just yesterday, Utsumi says, they managed to finally solve a problem that the camera team had been struggling with for weeks—an intermittent fault in the camera cooling system that only seemed to happen when the telescope was moving. This was potentially a very serious problem, and Utsumi’s phone would alert him every time the fault occurred, over and over again in the middle of the night. The fault was finally traced to a cable within the telescope’s structure that used pins that were slightly too small, leading to a loose connection. Utsumi’s contract started in 2017 and was supposed to last three years, but he’s still here. “I wanted to see first photon,” he says. “I’m an astronomer. I’ve been working on this camera so that it can observe the universe. And I want to see that light, from those photons from distant galaxies.” This is something I’ve also been thinking about—those lonely photons traveling through space for billions of years, and within the coming days, a lucky few of them will land on the sensors Utsumi has been tending, and we’ll get to see them. He nods, smiling. “I don’t want to lose one, you know?” Rubin’s commissioning scientists have a unique role, working at the intersection of science and engineering to turn a bunch of custom parts into a functioning science instrument. Commissioning scientist Marina Pavlovic is a postdoc from Serbia with a background in the formation of supermassive black holes created by merging galaxies. “I came here last year as a volunteer,” she tells me. “My plan was to stay for three months, and 11 months later I’m a commissioning scientist. It’s crazy!” Pavlovic’s job is to help diagnose and troubleshoot whatever isn’t working quite right. And since most things aren’t working quite right, she’s been very busy. “I love when things need to be fixed because I am learning about the system more and more every time there’s a problem—every day is a new experience here.” I ask her what she’ll do next, once Rubin is up and running. “If you love commissioning instruments, that is something that you can do for the rest of your life, because there are always going to be new instruments,” she says. Before that happens, though, Pavlovic has to survive the next few weeks of going on sky. “It’s going to be so emotional. It’s going to be the beginning of a new era in astronomy, and knowing that you did it, that you made it happen, at least a tiny percent of it, that will be a priceless moment.” “I had to learn how to calm down to do this job,” she admits, “because sometimes I get too excited about things and I cannot sleep after that. But it’s okay. I started doing yoga, and it’s working.” From First Photon to First Light My stay on the summit comes to an end on 14 April, just a day before first photon, so as soon as I get home I check in with some of the engineers and astronomers that I met to see how things went. Guillem Megias Homar manages the adaptive optics system—232 actuators that flex the surfaces of the telescope’s three mirrors a few micrometers at a time to bring the image into perfect focus. Currently working on his Ph.D., he was born in 1997, one year after the Rubin project started. First photon, for him, went like this: “I was in the control room, sitting next to the camera team. We have a microphone on the camera, so that we can hear when the shutter is moving. And we hear the first click. And then all of a sudden, the image shows up on the screens in the control room, and it was just an explosion of emotions. All that we have been fighting for is finally a reality. We are on sky!” There were toasts (with sparkling apple juice, of course), and enough speeches that Megias Homar started to get impatient: “I was like, when can we start working? But it was only an hour, and then everything became much more quiet.” Another newly released image showing a small section of the Rubin Observatory’s total view of the Virgo cluster of galaxies. Visible are bright stars in the Milky Way galaxy shining in the foreground, and many distant galaxies in the background. NSF-DOE Rubin Observatory “It was satisfying to see that everything that we’d been building was finally working,” Victor Krabbendam, project manager for Rubin construction, tells me a few weeks later. “But some of us have been at this for so long that first photon became just one of many firsts.” Krabbendam has been with the observatory full-time for the last 21 years. “And the very moment you succeed with one thing, it’s time to be doing the next thing.” Since first photon, Rubin has been undergoing calibrations, collecting data for the first images that it’s now sharing with the world, and preparing to scale up to begin its survey. Operations will soon become routine, the commissioning scientists will move on, and eventually, Rubin will largely run itself, with just a few people at the observatory most nights. But for astronomers, the next 10 years will be anything but routine. “It’s going to be wildly different,” says Krabbendam. “Rubin will feed generations of scientists with trillions of data points of billions of objects. Explore the data. Harvest it. Develop your idea, see if it’s there. It’s going to be phenomenal.” Listen to a Conversation About the Rubin Observatory As part of an experiment with AI storytelling tools, author Evan Ackerman—who visited the Vera C. Rubin Observatory in Chile for four days this past April—fed over 14 hours of raw audio from his interviews and other reporting notes into NotebookLM, an AI-powered research assistant developed by Google. The result is a podcast-style audio experience that you can listen to here. While the script and voices are AI-generated, the conversation is grounded in Ackerman’s original reporting, and includes many details that did not appear in the article above. Ackerman reviewed and edited the audio to ensure accuracy, and there are minor corrections in the transcript. Let us know what you think of this experiment in AI narration. Your browser does not support the audio tag. See transcript 0:01: Today we’re taking a deep dive into the engineering marvel that is the Vera C. Rubin Observatory. 0:06: And and it really is a marvel. 0:08: This project pushes the limits, you know, not just for the science itself, like mapping the Milky Way or exploring dark energy, which is amazing, obviously. 0:16: But it’s also pushing the limits in just building the tools, the technical ingenuity, the, the sheer human collaboration needed to make something this complex actually work. 0:28: That’s what’s really fascinating to me. 0:29: Exactly. 0:30: And our mission for this deep dive is to go beyond the headlines, isn’t it? 0:33: We want to uncover those specific Kind of hidden technical details, the stuff from the audio interviews, the internal docs that really define this observatory. 0:41: The clever engineering solutions. 0:43: Yeah, the nuts and bolts, the answers to challenges nobody’s faced before, stuff that anyone who appreciates, you know, complex systems engineering would find really interesting. 0:53: Definitely. 0:54: So let’s start right at the heart of it. 0:57: The Simonyi survey telescope itself. 1:00: It’s this 350 ton machine inside a 600 ton dome, 30 m wide, huge. [The dome is closer to 650 tons.] 1:07: But the really astonishing part is its speed, speed and precision. 1:11: How do you even engineer something that massive to move that quickly while keeping everything stable down to the submicron level? [Micron level is more accurate.] 1:18: Well, that’s, that’s the core challenge, right? 1:20: This telescope, it can hit a top speed of 3.5 degrees per second. 1:24: Wow. 1:24: Yeah, and it can, you know, move to basically any point in the sky. 1:28: In under 20 seconds, 20 seconds, which makes it by far the fastest moving large telescope ever built, and the dome has to keep up. 1:36: So it’s also the fastest moving dome. 1:38: So the whole building is essentially racing along with the telescope. 1:41: Exactly. 1:41: And achieving that meant pretty much every component had to be custom designed like the pier holding the telescope up. 1:47: It’s mostly steel, not concrete. 1:49: Oh, interesting. 1:50: Why steel? 1:51: Specifically to stop it from twisting or vibrating when the telescope makes those incredibly fast moves. 1:56: Concrete just wouldn’t handle the torque the same way. [The pier is more steel than concrete, but it's still substantially concrete.] 1:59: OK, that makes sense. 1:59: And the power needed to accelerate and decelerate, you know, 300 tons, that must be absolutely massive. 2:06: Oh. 2:06: The instantaneous draw would be enormous. 2:09: How did they manage that without like dimming the lights on the whole. 2:12: Mountaintop every 30 seconds. 2:14: Yeah, that was a real concern, constant brownouts. 2:17: The solution was actually pretty elegant, involving these onboard capacitor banks. 2:22: Yep, slung right underneath the telescope structure. 2:24: They can slowly sip power from the grid, store it up over time, and then bam, discharge it really quickly for those big acceleration surges. 2:32: like a giant camera flash, but for moving a telescope, of yeah. 2:36: It smooths out the demand, preventing those grid disruptions. 2:40: Very clever engineering. 2:41: And beyond the movement, the mirrors themselves, equally critical, equally impressive, I imagine. 2:47: How did they tackle designing and making optics that large and precise? 2:51: Right, so the main mirror, the primary mirror, M1M3. 2:55: It’s a single piece of glass, 8.4 m across, low expansion borosilicate glass. 3:01: And that 8.4 m size, was that just like the biggest they could manage? 3:05: Well, it was a really crucial early decision. 3:07: The science absolutely required something at least 7 or 8 m wide. 3:13: But going much bigger, say 10 or 12 m, the logistics became almost impossible. 3:19: The big one was transport. 3:21: There’s a tunnel on the mountain road up to the summit, and a mirror, much larger than 8.4 m, physically wouldn’t fit through it. 3:28: No way. 3:29: So the tunnel actually set an upper limit on the mirror size. 3:31: Pretty much, yeah. 3:32: Building new road or some other complex transport method. 3:36: It would have added enormous cost and complexity. 3:38: So 8.4 m was that sweet spot between scientific need. 3:42: And, well, physical reality. 3:43: Wow, a real world constraint driving fundamental design. 3:47: And the mirror itself, you said M1 M3, it’s not just one simple mirror surface. 3:52: Correct. 3:52: It’s technically two mirror surfaces ground into that single piece of glass. 3:57: The central part has a more pronounced curvature. 3:59: It’s M1 and M3 combined. 4:00: OK, so fabricating that must have been tricky, especially with what, 10 tons of glass just in the center. 4:07: Oh, absolutely novel and complicated. 4:09: And these mirrors, they don’t support their own weight rigidly. 4:12: So just handling them during manufacturing, polishing, even getting them out of the casting mold, was a huge engineering challenge. 4:18: You can’t just lift it like a dinner plate. 4:20: Not quite, and then there’s maintaining it, re-silvering. 4:24: They hope to do it every 5 years. 4:26: Well, traditionally, big mirrors like this often need it more, like every 1.5 to 2 years, and it’s a risky weeks-long job. 4:34: You have to unbolt this priceless, unique piece of equipment, move it. 4:39: It’s nerve-wracking. 4:40: I bet. 4:40: And the silver coating itself is tiny, right? 4:42: Incredibly thin, just a few nanometers of pure silver. 4:46: It takes about 24 g for the whole giant surface, bonded with the adhesive layers that are measured in Angstroms. [It's closer to 26 grams of silver.] 4:52: It’s amazing precision. 4:54: So tying this together, you have this fast moving telescope, massive mirrors. 4:59: How do they keep everything perfectly focused, especially with multiple optical elements moving relative to each other? 5:04: that’s where these things called hexapods come in. 5:08: Really crucial bits of kit. 5:09: Hexapods, like six feet? 5:12: Sort of. 5:13: They’re mechanical systems with 6 adjustable arms or struts. 5:17: A simpler telescope might just have one maybe on the camera for basic focusing, but Ruben needs more because it’s got the 3 mirrors plus the camera. 5:25: Exactly. 5:26: So there’s a hexapod mounted on the secondary mirror, M2. 5:29: Its job is to keep M2 perfectly positioned relative to M1 and M3, compensating for tiny shifts or flexures. 5:36: And then there’s another hexapod on the camera itself. 5:39: That one adjusts the position and tilt of the entire camera’s sensor plane, the focal plane. 5:43: To get that perfect focus across the whole field of view. 5:46: And these hexapods move in 6 ways. 5:48: Yep, 6 degrees of freedom. 5:50: They can adjust position along the X, Y, and Z axis, and they can adjust rotation or tilt around those 3 axes as well. 5:57: It allows for incredibly fine adjustments, microp precision stuff. 6:00: So they’re constantly making these tiny tweaks as the telescope moves. 6:04: Constantly. 6:05: The active optics system uses them. 6:07: It calculates the needed corrections based on reference stars in the images, figures out how the mirror might be slightly bending. 6:13: And then tells the hexapods how to compensate. 6:15: It’s controlling like 26 g of silver coating on the mirror surface down to micron precision, using the mirror’s own natural bending modes. 6:24: It’s pretty wild. 6:24: Incredible. 6:25: OK, let’s pivot to the camera itself. 6:28: The LSST camera. 6:29: Big digital camera ever built, right? 6:31: Size of a small car, 2800 kg, captures 3.2 gigapixel images, just staggering numbers. 6:38: They really are, and the engineering inside is just as staggering. 6:41: That Socal plane where the light actually hits. 6:43: It’s made up of 189 individual CCD sensors. 6:47: Yep, 4K by 4K CCDs grouped into 21 rafts. 6:50: They give them like tiles, and each CCD has 16 amplifiers reading it out. 6:54: Why so many amplifiers? 6:56: Speed. 6:56: Each amplifier reads out about a million pixels. 6:59: By dividing the job up like that, they can read out the entire 3.2 gigapixel sensor in just 2 seconds. 7:04: 2 seconds for that much data. 7:05: Wow. 7:06: It’s essential for the survey’s rapid cadence. 7:09: Getting all those 189 CCDs perfectly flat must have been, I mean, are they delicate? 7:15: Unbelievably delicate. 7:16: They’re silicon wafers only 100 microns thick. 7:18: How thick is that really? 7:19: about the thickness of a human hair. 7:22: You could literally break one by breathing on it wrong, apparently, seriously, yeah. 7:26: And the challenge was aligning all 189 of them across this 650 millimeter wide focal plane, so the entire surface is flat. 7:34: To within just 24 microns, peak to valley. 7:37: 24 microns. 7:39: That sounds impossibly flat. 7:40: It’s like, imagine the entire United States. 7:43: Now imagine the difference between the lowest point and the highest point across the whole country was only 100 ft. 7:49: That’s the kind of relative flatness they achieved on the camera sensor. 7:52: OK, that puts it in perspective. 7:53: And why is that level of flatness so critical? 7:56: Because the telescope focuses light. 7:58: terribly. 7:58: It’s an F1.2 system, which means it has a very shallow depth of field. 8:02: If the sensors aren’t perfectly in that focal plane, even by a few microns, parts of the image go out of focus. 8:08: Gotcha. 8:08: And the pixels themselves, the little light buckets on the CCDs, are they special? 8:14: They’re custom made, definitely. 8:16: They settled on 10 micron pixels. 8:18: They figured anything smaller wouldn’t actually give them more useful scientific information. 8:23: Because you start hitting the limits of what the atmosphere and the telescope optics themselves can resolve. 8:28: So 10 microns was the optimal size, right? 8:31: balancing sensor tech with physical limits. 8:33: Now, keeping something that sensitive cool, that sounds like a nightmare, especially with all those electronics. 8:39: Oh, it’s a huge thermal engineering challenge. 8:42: The camera actually has 3 different cooling zones, 3 distinct temperature levels inside. 8:46: 3. 8:47: OK. 8:47: First, the CCDs themselves. 8:49: They need to be incredibly cold to minimize noise. 8:51: They operate at -125 °C. 8:54: -125C, how do they manage that? 8:57: With a special evaporator plate connected to the CCD rafts by flexible copper braids, which pulls heat away very effectively. 9:04: Then you’ve got the cameras, electronics, the readout boards and stuff. 9:07: They run cooler than room temp, but not that cold, around -50 °C. 9:12: OK. 9:12: That requires a separate liquid cooling loop delivered through these special vacuum insulated tubes to prevent heat leaks. 9:18: And the third zone. 9:19: That’s for the electronics in the utility trunk at the back of the camera. 9:23: They generate a fair bit of heat, about 3000 watts, like a few hair dryers running constantly. 9:27: Exactly. 9:28: So there’s a third liquid cooling system just for them, keeping them just slightly below the ambient room temperature in the dome. 9:35: And all this cooling, it’s not just to keep the parts from overheating, right? 9:39: It affects the images, absolutely critical for image quality. 9:44: If the outer surface of the camera body itself is even slightly warmer or cooler than the air inside the dome, it creates tiny air currents, turbulence right near the light path. 9:57: And that shows up as little wavy distortions in the images, messing up the precision. 10:02: So even the outside temperature of the camera matters. 10:04: Yep, it’s not just a camera. 10:06: They even have to monitor the heat generated by the motors that move the massive dome, because that heat could potentially cause enough air turbulence inside the dome to affect the image quality too. 10:16: That’s incredible attention to detail, and the camera interior is a vacuum you mentioned. 10:21: Yes, a very strong vacuum. 10:23: They pump it down about once a year, first using turbopumps spinning at like 80,000 RPM to get it down to about 102 tor. 10:32: Then they use other methods to get it down much further. 10:34: The 107 tor, that’s an ultra high vacuum. 10:37: Why the vacuum? 10:37: Keep frost off the cold part. 10:39: Exactly. 10:40: Prevents condensation and frost on those negatives when it 25 degree CCDs and generally ensures everything works optimally. 10:47: For normal operation, day to day, they use something called an ion pump. 10:51: How does that work? 10:52: It basically uses a strong electric field to ionize any stray gas molecules, mostly hydrogen, and trap them, effectively removing them from the vacuum space, very efficient for maintaining that ultra-high vacuum. 11:04: OK, so we have this incredible camera taking these massive images every few seconds. 11:08: Once those photons hit the CCDs and become digital signals, What happens next? 11:12: How does Ruben handle this absolute flood of data? 11:15: Yeah, this is where Ruben becomes, you know, almost as much a data processing machine as a telescope. 11:20: It’s designed for the data output. 11:22: So photons hit the CCDs, get converted to electrical signals. 11:27: Then, interestingly, they get converted back into light signals, photonic signals back to light. 11:32: Why? 11:33: To send them over fiber optics. 11:34: They’re about 6 kilometers of fiber optic cable running through the observatory building. 11:39: These signals go to FPGA boards, field programmable gate arrays in the data acquisition system. 11:46: OK. 11:46: And those FPGAs are basically assembling the complete image data packages from all the different CCDs and amplifiers. 11:53: That sounds like a fire hose of data leaving the camera. 11:56: How does it get off the mountain and where does it need to go? 11:58: And what about all the like operational data, temperatures, positions? 12:02: Good question. 12:03: There are really two main data streams all that telemetry you mentioned, sensor readings, temperatures, actuator positions, command set, everything about the state of the observatory that all gets collected into something called the Engineering facility database or EFD. 12:16: They use Kafka for transmitting that data. 12:18: It’s good for high volume streams, and store it in an influx database, which is great for time series data like sensor readings. 12:26: And astronomers can access that. 12:28: Well, there’s actually a duplicate copy of the EFD down at SLAC, the research center in California. 12:34: So scientists and engineers can query that copy without bogging down the live system running on the mountain. 12:40: Smart. 12:41: How much data are we talking about there? 12:43: For the engineering data, it’s about 20 gigabytes per night, and they plan to keep about a year’s worth online. 12:49: OK. 12:49: And the image data, the actual science pixels. 12:52: That takes a different path. [All of the data from Rubin to SLAC travels over the same network.] 12:53: It travels over dedicated high-speed network links, part of ESET, the research network, all the way from Chile, usually via Boca Raton, Florida, then Atlanta, before finally landing at SLAC. 13:05: And how fast does that need to be? 13:07: The goal is super fast. 13:09: They aim to get every image from the telescope in Chile to the data center at SLAC within 7 seconds of the shutter closing. 13:15: 7 seconds for gigabytes of data. 13:18: Yeah. 13:18: Sometimes network traffic bumps it up to maybe 30 seconds or so, but the target is 7. 13:23: It’s crucial for the next step, which is making sense of it all. 13:27: How do astronomers actually use this, this torrent of images and data? 13:30: Right. 13:31: This really changes how astronomy might be done. 13:33: Because Ruben is designed to generate alerts, real-time notifications about changes in the sky. 13:39: Alerts like, hey, something just exploded over here. 13:42: Pretty much. 13:42: It takes an image compared to the previous images of the same patch of sky and identifies anything that’s changed, appeared, disappeared, moved, gotten brighter, or fainter. 13:53: It expects to generate about 10,000 such alerts per image. 13:57: 10,000 per image, and they take an image every every 20 seconds or so on average, including readouts. [Images are taken every 34 seconds: a 30 second exposure, and then about 4 seconds for the telescope to move and settle.] 14:03: So you’re talking around 10 million alerts every single night. 14:06: 10 million a night. 14:07: Yep. 14:08: And the goal is to get those alerts out to the world within 60 seconds of the image being taken. 14:13: That’s insane. 14:14: What’s in an alert? 14:15: It contains the object’s position, brightness, how it’s changed, and little cut out images, postage stamps in the last 12 months of observations, so astronomers can quickly see the history. 14:24: But surely not all 10 million are real astronomical events satellites, cosmic rays. 14:30: Exactly. 14:31: The observatory itself does a first pass filter, masking out known issues like satellite trails, cosmic ray hits, atmospheric effects, with what they call real bogus stuff. 14:41: OK. 14:42: Then, this filtered stream of potentially real alerts goes out to external alert brokers. 14:49: These are systems run by different scientific groups around the world. 14:52: Yeah, and what did the brokers do? 14:53: They ingest the huge stream from Ruben and apply their own filters, based on what their particular community is interested in. 15:00: So an astronomer studying supernovae can subscribe to a broker that filters just for likely supernova candidates. 15:06: Another might filter for near Earth asteroids or specific types of variable stars. 15:12: so it makes the fire hose manageable. 15:13: You subscribe to the trickle you care about. 15:15: Precisely. 15:16: It’s a way to distribute the discovery potential across the whole community. 15:19: So it’s not just raw images astronomers get, but these alerts and presumably processed data too. 15:25: Oh yes. 15:26: Rubin provides the raw images, but also fully processed images, corrected for instrument effects, calibrated called processed visit images. 15:34: And also template images, deep combinations of previous images used for comparison. 15:38: And managing all that data, 15 petabytes you mentioned, how do you query that effectively? 15:44: They use a system called Keyserve. [The system is "QServ."] 15:46: It’s a distributed relational database, custom built basically, designed to handle these enormous astronomical catalogs. 15:53: The goal is to let astronomers run complex searches across maybe 15 petabytes of catalog data and get answers back in minutes, not days or weeks. 16:02: And how do individual astronomers actually interact with it? 16:04: Do they download petabytes? 16:06: No, definitely not. 16:07: For general access, there’s a science platform, the front end of which runs on Google Cloud. 16:11: Users interact mainly through Jupiter notebooks. 16:13: Python notebooks, familiar territory for many scientists. 16:17: Exactly. 16:18: They can write arbitrary Python code, access the catalogs directly, do analysis for really heavy duty stuff like large scale batch processing. 16:27: They can submit jobs to the big compute cluster at SLEC, which sits right next to the data storage. 16:33: That’s much more efficient. 16:34: Have they tested this? 16:35: Can it handle thousands of astronomers hitting it at once? 16:38: They’ve done extensive testing, yeah, scaled it up with hundreds of users already, and they seem confident they can handle up to maybe 3000 simultaneous users without issues. 16:49: And a key point. 16:51: After an initial proprietary period for the main survey team, all the data and importantly, all the software algorithms used to process it become public. 17:00: Open source algorithms too. 17:01: Yes, the idea is, if the community can improve on their processing pipelines, they’re encouraged to contribute those solutions back. 17:08: It’s meant to be a community resource. 17:10: That open approach is fantastic, and even the way the images are presented visually has some deep thought behind it, doesn’t it? 17:15: You mentioned Robert Leptina’s perspective. 17:17: Yes, this is fascinating. 17:19: It’s about how you assign color to astronomical images, which usually combine data from different filters, like red, green, blue. 17:28: It’s not just about making pretty pictures, though they can be beautiful. 17:31: Right, it should be scientifically meaningful. 17:34: Exactly. 17:35: Lepton’s approach tries to preserve the inherent color information in the data. 17:40: Many methods saturate bright objects, making their centers just white blobs. 17:44: Yeah, you see that a lot. 17:46: His algorithm uses a different mathematical scaling, more like a logarithmic scale, that avoids this saturation. 17:52: It actually propagates the true color information back into the centers of bright stars and galaxies. 17:57: So, a galaxy that’s genuinely redder, because it’s red shifted, will actually look redder in the image, even in its bright core. 18:04: Precisely, in a scientifically meaningful way. 18:07: Even if our eyes wouldn’t perceive it quite that way directly through a telescope, the image renders the data faithfully. 18:13: It helps astronomers visually interpret the physics. 18:15: It’s a subtle but powerful detail in making the data useful. 18:19: It really is. 18:20: Beyond just taking pictures, I heard Ruben’s wide view is useful for something else entirely gravitational waves. 18:26: That’s right. 18:26: It’s a really cool synergy. 18:28: Gravitational wave detectors like Lego and Virgo, they detect ripples in space-time, often from emerging black holes or neutron stars, but they usually only narrow down the location to a relatively large patch of sky, maybe 10 square degrees or sometimes much more. 18:41: Ruben’s camera has a field of view of about 9.6 square degrees. 18:45: That’s huge for a telescope. 18:47: It almost perfectly matches the typical LIGO alert area. 18:51: so when LIGO sends an alert, Ruben can quickly scan that whole error box, maybe taking just a few pointings, looking for any new point of light. 19:00: The optical counterpart, the Killanova explosion, or whatever light accompany the gravitational wave event. 19:05: It’s a fantastic follow-up machine. 19:08: Now, stepping back a bit, this whole thing sounds like a colossal integration challenge. 19:13: A huge system of systems, many parts custom built, pushed to their limits. 19:18: What were some of those big integration hurdles, bringing it all together? 19:22: Yeah, classic system of systems is a good description. 19:25: And because nobody’s built an observatory quite like this before, a lot of the commissioning phase, getting everything working together involves figuring out the procedures as they go. 19:34: Learning by doing on a massive scale. 19:36: Pretty much. 19:37: They’re essentially, you know, teaching the system how to walk. 19:40: And there’s this constant tension, this balancing act. 19:43: Do you push forward, maybe build up some technical debt, things you know you’ll have to fix later, or do you stop and make sure every little issue is 100% perfect before moving on, especially with a huge distributed team? 19:54: I can imagine. 19:55: And you mentioned the dome motors earlier. 19:57: That discovery about heat affecting images sounds like a perfect example of unforeseen integration issues. 20:03: Exactly. 20:03: Marina Pavvich described that. 20:05: They ran the dome motors at full speed, something maybe nobody had done for extended periods in that exact configuration before, and realized, huh. 20:13: The heat these generate might actually cause enough air turbulence to mess with our image quality. 20:19: That’s the kind of thing you only find when you push the integrated system. 20:23: Lots of unexpected learning then. 20:25: What about interacting with the outside world? 20:27: Other telescopes, the atmosphere itself? 20:30: How does Ruben handle atmospheric distortion, for instance? 20:33: that’s another interesting point. 20:35: Many modern telescopes use lasers. 20:37: They shoot a laser up into the sky to create an artificial guide star, right, to measure. 20:42: Atmospheric turbulence. 20:43: Exactly. 20:44: Then they use deformable mirrors to correct for that turbulence in real time. 20:48: But Ruben cannot use a laser like that. 20:50: Why? 20:51: Because its field of view is enormous. 20:53: It sees such a wide patch of sky at once. 20:55: A single laser beam, even a pinpoint from another nearby observatory, would contaminate a huge fraction of Ruben’s image. 21:03: It would look like a giant streak across, you know, a quarter of the sky for Ruben. 21:06: Oh, wow. 21:07: OK. 21:08: Too much interference. 21:09: So how does it correct for the atmosphere? 21:11: Software. 21:12: It uses a really clever approach called forward modeling. 21:16: It looks at the shapes of hundreds of stars across its wide field of view in each image. 21:21: It knows what those stars should look like, theoretically. 21:25: Then it builds a complex mathematical model of the atmosphere’s distorting effect across the entire field of view that would explain the observed star shapes. 21:33: It iterates this model hundreds of times per image until it finds the best fit. [The model is created by iterating on the image data, but iteration is not necessary for every image.] 21:38: Then it uses that model to correct the image, removing the atmospheric blurring. 21:43: So it calculates the distortion instead of measuring it directly with a laser. 21:46: Essentially, yes. 21:48: Now, interestingly, there is an auxiliary telescope built alongside Ruben, specifically designed to measure atmospheric properties independently. 21:55: Oh, so they could use that data. 21:57: They could, but currently, they’re finding their software modeling approach using the science images themselves, works so well that they aren’t actively incorporating the data from the auxiliary telescope for that correction right now. 22:08: The software solution is proving powerful enough on its own. 22:11: Fascinating. 22:12: And they still have to coordinate with other telescopes about their lasers, right? 22:15: Oh yeah. 22:15: They have agreements about when nearby observatories can point their lasers, and sometimes Ruben might have to switch to a specific filter like the Iband, which is less sensitive to the laser. 22:25: Light if one is active nearby while they’re trying to focus. 22:28: So many interacting systems. 22:30: What an incredible journey through the engineering of Ruben. 22:33: Just the sheer ingenuity from the custom steel pier and the capacitor banks, the hexapods, that incredibly flat camera, the data systems. 22:43: It’s truly a machine built to push boundaries. 22:45: It really is. 22:46: And it’s important to remember, this isn’t just, you know, a bigger version of existing telescopes. 22:51: It’s a fundamentally different kind of machine. 22:53: How so? 22:54: By creating this massive all-purpose data set, imaging the entire southern sky over 800 times, cataloging maybe 40 billion objects, it shifts the paradigm. 23:07: Astronomy becomes less about individual scientists applying for time to point a telescope at one specific thing and more about statistical analysis, about mining this unprecedented ocean of data that Rubin provides to everyone. 23:21: So what does this all mean for us, for science? 23:24: Well, it’s a generational investment in fundamental discovery. 23:27: They’ve optimized this whole system, the telescope, the camera, the data pipeline. 23:31: For finding, quote, exactly the stuff we don’t know we’ll find. 23:34: Optimized for the unknown, I like that. 23:36: Yeah, we’re basically generating this incredible resource that will feed generations of astronomers and astrophysicists. 23:42: They’ll explore it, they’ll harvest discoveries from it, they’ll find patterns and objects and phenomena within billions and billions of data points that we can’t even conceive of yet. 23:50: And that really is the ultimate excitement, isn’t it? 23:53: Knowing that this monumental feat of engineering isn’t just answering old questions, but it’s poised to open up entirely new questions about the universe, questions we literally don’t know how to ask today. 24:04: Exactly. 24:05: So, for you, the listener, just think about that. 24:08: Consider the immense, the completely unknown discoveries that are waiting out there just waiting to be found when an entire universe of data becomes accessible like this. 24:16: What might we find? Back to top

A version of this post originally appeared on Tedium, Ernie Smith’s newsletter, which hunts for the end of the long tail. For roughly three decades, the JPEG has been the World Wide Web’s primary image format. But it wasn’t the one the Web started with. In fact, the first mainstream graphical browser, NCSA Mosaic, didn’t initially support inline JPEG files—just inline GIFs, along with a couple of other formats forgotten to history. However, the JPEG had many advantages over the format it quickly usurped. aspect_ratio Despite not appearing together right away—it first appeared in Netscape in 1995, three years after the image standard was officially published—the JPEG and web browser fit together naturally. JPEG files degraded more gracefully than GIFs, retaining more of the picture’s initial form—and that allowed the format to scale to greater levels of success. While it wasn’t capable of animation, it progressively expanded from something a modem could pokily render to a format that was good enough for high-end professional photography. For the internet’s purposes, the degradation was the important part. But it wasn’t the only thing that made the JPEG immensely valuable to the digital world. An essential part was that it was a documented standard built by numerous stakeholders. The GIF was a de facto standard. The JPEG was an actual one How important is it that JPEG was a standard? Let me tell you a story. During a 2013 New York Times interview conducted just before he received an award honoring his creation, GIF creator Steve Wilhite stepped into a debate he unwittingly created. Simply put, nobody knew how to pronounce the acronym for the image format he had fostered, the Graphics Interchange Format. He used the moment to attempt to set the record straight—it was pronounced like the peanut butter brand: “It is a soft ‘G,’ pronounced ‘jif.’ End of story,” he said. I posted a quote from Wilhite on my popular Tumblr around that time, a period when the social media site was the center of the GIF universe. And soon afterward, my post got thousands of reblogs—nearly all of them disagreeing with Wilhite. Soon, Wilhite’s quote became a meme. The situation paints how Wilhite, who died in 2022, did not develop his format by committee. He could say it sounded like “JIF” because he built it himself. He was handed the project as a CompuServe employee in 1987; he produced the object, and that was that. The initial document describing how it works? Dead simple. 38 years later, we’re still using the GIF—but it never rose to the same prevalence of JPEG. The JPEG, which formally emerged about five years later, was very much not that situation. Far from it, in fact—it’s the difference between a de facto standard and an actual one. And that proved essential to its eventual ubiquity. We’re going to degrade the quality of this image throughout this article. At its full image size, it’s 13.7 megabytes.Irina Iriser How the JPEG format came to life Built with input from dozens of stakeholders, the Joint Photographic Experts Group ultimately aimed to create a format that fit everyone’s needs. (Reflecting its committee-led roots, there would be no confusion about the format’s name—an acronym of the organization that designed it.) And when the format was finally unleashed on the world, it was the subject of a more than 600-page book. JPEG: Still Image Data Compression Standard, written by IBM employees and JPEG organization stakeholders William B. Pennebaker and Joan L. Mitchell, describes a landscape of multimedia imagery, held back without a way to balance the need for photorealistic images and immediacy. Standardization, they believed, could fix this. “The problem was not so much the lack of algorithms for image compression (as there is a long history of technical work in this area),” the authors wrote, “but, rather, the lack of a standard algorithm—one which would allow an interchange of images between diverse applications.” And they were absolutely right. For more than 30 years, JPEG has made high-quality, high-resolution photography accessible in operating systems far and wide. Although we no longer need to compress JPEGs to within an inch of their life, having that capability helped enable the modern internet. As the book notes, Mitchell and Pennebaker were given IBM’s support to follow through this research and work with the JPEG committee, and that support led them to develop many of the JPEG format’s foundational patents. Described in patents filed by Mitchell and Pennebaker in 1988, IBM and other members of the JPEG standards committee, such as AT&T and Canon, were developing ways to use compression to make high-quality images easier to deliver in confined settings. Each member brought their own needs to the process. Canon, obviously, was more focused on printers and photography, while AT&T’s interests were tied to data transmission. Together, the companies left behind a standard that has stood the test of time. All this means, funnily enough, that the first place that a program capable of using JPEG compression appeared was not MacOS or Windows, but OS/2—a fascinating-but-failed graphical operating system created by Pennebaker and Mitchell’s employer, IBM. As early as 1990, OS/2 supported the format through the OS/2 Image Support application. At 50 percent of its initial quality, the image is down to about 2.6 MB. By dropping half of the image’s quality, we brought it down to one-fifth of the original file size. Original image: Irina Iriser What a JPEG does when you heavily compress it The thing that differentiates a JPEG file from a PNG or a GIF is how the data degrades as you compress it. The goal for a JPEG image is to still look like a photo when all is said and done, even if some compression is necessary to make it all work at a reasonable size. That way, you can display something that looks close to the original image in fewer bytes. Or, as Pennebaker and Mitchell put it, “the most effective compression is achieved by approximating the original image (rather than reproducing it exactly).” Central to this is a compression process called discrete cosine transform (DCT), a lossy form of compression encoding heavily used in all sorts of compressed formats, most notably in digital audio and signal processing. Essentially, it delivers a lower-quality product by removing details, while still keeping the heart of the original product through approximation. The stronger the cosine transformation, the more compressed the final result. The algorithm, developed by researchers in the 1970s, essentially takes a grid of data and treats it as if you’re controlling its frequency with a knob. The data rate is controlled like water from a faucet: The more data you want, the higher the setting. DCT allows a trickle of data to still come out in highly compressed situations, even if it means a slightly compromised result. In other words, you may not keep all the data when you compress it, but DCT allows you to keep the heart of it. (See this video for a more technical but still somewhat easy-to-follow description of DCT.) DCT is everywhere. If you have ever seen a streaming video or an online radio stream that degraded in quality because your bandwidth suddenly declined, you’ve witnessed DCT being utilized in real time. A JPEG file doesn’t have to leverage the DCT with just one method, as JPEG: Still Image Data Compression Standard explains: The JPEG standard describes a family of large image compression techniques, rather than a single compression technique. It provides a “tool kit” of compression techniques from which applications can select elements that satisfy their particular requirements. The toolkit has four modes: Sequential DCT, which displays the compressed image in order, like a window shade slowly being rolled down Progressive DCT, which displays the full image in the lowest-resolution format, then adds detail as more information rolls in Sequential lossless, which uses the window shade format but doesn’t compress the image Hierarchical mode, which combines the prior three modes—so maybe it starts with a progressive mode, then loads DCT compression slowly, but then reaches a lossless final result At the time the JPEG was being created, modems were extremely common. That meant images loaded slowly, making Progressive DCT the most fitting format for the early internet. Over time, the progressive DCT mode has become less common, as many computers can simply load the sequential DCT in one fell swoop. That same forest, saved at 5 percent quality. Down to about 419 kilobytes.Original image: Irina Iriser When an image is compressed with DCT, the change tends to be less noticeable in busier, more textured areas of the picture, like hair or foliage. Those areas are harder to compress, which means they keep their integrity longer. It tends to be more noticeable, however, with solid colors or in areas where the image sharply changes from one color to another—like text on a page. Ever screenshot a social media post, only for it to look noisy? Congratulations, you just made a JPEG file. Other formats, like PNG, do better with text, because their compression format is intended to be non-lossy. (Side note: PNG’s compression format, DEFLATE, was designed by Phil Katz, who also created the ZIP format. The PNG format uses it in part because it was a license-free compression format. So it turns out the brilliant coder with the sad life story improved the internet in multiple ways before his untimely passing.) In many ways, the JPEG is one tool in our image-making toolkit. Despite its age and maturity, it remains one of our best options for sharing photos on the internet. But it is not a tool for every setting—despite the fact that, like a wrench sometimes used as a hammer, we often leverage it that way. Forgent Networks claimed to own the JPEG’s defining algorithm The JPEG format gained popularity in the ’90s for reasons beyond the quality of the format. Patents also played a role: Starting in 1994, the tech company Unisys attempted to bill individual users who relied on GIF files, which used a patent the company owned. This made the free-to-use JPEG more popular. (This situation also led to the creation of the patent-free PNG format.) While the JPEG was standards-based, it could still have faced the same fate as the GIF, thanks to the quirks of the patent system. A few years before the file format came to life, a pair of Compression Labs employees filed a patent application that dealt with the compression of motion graphics. By the time anyone noticed its similarity to JPEG compression, the format was ubiquitous. Our forest, saved at 1 percent quality. This image is only about 239 KB in size, yet it’s still easily recognizable as the same photo. That’s the power of the JPEG.Original image: Irina Iriser Then in 1997, a company named Forgent Networks acquired Compression Labs. The company eventually spotted the patent and began filing lawsuits over it, a series of events it saw as a stroke of good luck. “The patent, in some respects, is a lottery ticket,” Forgent Chief Financial Officer Jay Peterson told CNET in 2005. “If you told me five years ago that ‘You have the patent for JPEG,’ I wouldn’t have believed it.” While Forgent’s claim of ownership of the JPEG compression algorithm was tenuous, it ultimately saw more success with its legal battles than Unisys did. The company earned more than $100 million from digital camera makers before the patent finally ran out of steam around 2007. The company also attempted to extract licensing fees from the PC industry. Eventually, Forgent agreed to a modest $8 million settlement. As the company took an increasingly aggressive approach to its acquired patent, it began to lose battles both in the court of public opinion and in actual courtrooms. Critics pounced on examples of prior art, while courts limited the patent’s use to motion-based uses like video. By 2007, Forgent’s compression patent expired—and its litigation-heavy approach to business went away. That year, the company became Asure Software, which now specializes in payroll and HR solutions. Talk about a reboot. Why the JPEG won’t die The JPEG file format has served us well. It’s been difficult to remove the format from its perch. The JPEG 2000 format, for example, was intended to supplant it by offering more lossless options and better performance. The format is widely used by the Library of Congress and specialized sites like the Internet Archive, however, it is less popular as an end-user format. See the forest JPEG degrade from its full resolution to 1 percent quality in this GIF. Original image: Irina Iriser Other image technologies have had somewhat more luck getting past the JPEG format. The Google-supported WebP is popular with website developers (and controversial with end users). Meanwhile, the formats AVIF and HEIC, each developed by standards bodies, have largely outpaced both JPEG and JPEG 2000. Still, the JPEG will be difficult to kill at this juncture. These days, the format is similar to MP3 or ZIP files—two legacy formats too popular and widely used to kill. Other formats that compress the files better and do the same things more efficiently are out there, but it’s difficult to topple a format with a 30-year head start. Shaking off the JPEG is easier said than done. I think most people will be fine to keep it around. Ernie Smith is the editor of Tedium, a long-running newsletter that hunts for the end of the long tail.

This article is excerpted from Every American an Innovator: How Innovation Became a Way of Life, by Matthew Wisnioski (The MIT Press, 2025). Imagine a point-to-point transportation service in which two parties communicate at a distance. A passenger in need of a ride contacts the service via phone. A complex algorithm based on time, distance, and volume informs both passenger and driver of the journey’s cost before it begins. This novel business plan promises efficient service and lower costs. It has the potential to disrupt an overregulated taxi monopoly in cities across the country. Its enhanced transparency may even reduce racial discrimination by preestablishing pickups regardless of race. aspect_ratio Every American an Innovator: How Innovation Became a Way of Life, by Matthew Wisnioski (The MIT Press, 2025).The MIT Press Carnegie Mellon University. The dial-a-ride service was designed to resurrect a defunct cab company that had once served Pittsburgh’s African American neighborhoods. National Science Foundation, the CED was envisioned as an innovation “hatchery,” intended to challenge the norms of research science and higher education, foster risk-taking, birth campus startups focused on market-based technological solutions to social problems, and remake American science to serve national needs. Are innovators born or made? During the Cold War, the model for training scientists and engineers in the United States was one of manpower in service to a linear model of innovation: Scientists pursued “basic” discovery in universities and federal laboratories; engineer–scientists conducted “applied” research elsewhere on campus; engineers developed those ideas in giant teams for companies such as Lockheed and Boeing; and research managers oversaw the whole process. This model dictated national science policy, elevated the scientist as a national hero in pursuit of truth beyond politics, and pumped hundreds of millions of dollars into higher education. In practice, the lines between basic and applied research were blurred, but the perceived hierarchy was integral to the NSF and the university research culture that it helped to foster. RELATED: Innovation Magazine and the Birth of a Buzzword The question was, how? And would the universities be willing to remake themselves to support innovation? The NSF experiments with innovation At the Utah Innovation Center, engineering students John DeJong and Douglas Kihm worked on a programmable electronics breadboard.Special Collections, J. Willard Marriott Library, The University of Utah In 1972, NSF director H. Guyford Stever established the Office of Experimental R&D Incentives to “incentivize” innovation for national needs by supporting research on “how the government [could] most effectively accelerate the transfer of new technology into productive enterprise.” Stever stressed the experimental nature of the program because many in the NSF and the scientific community resisted the idea of goal-directed research. Innovation, with its connotations of profit and social change, was even more suspect. To lead the initiative, Stever appointed C.B. Smith, a research manager at United Aircraft Corp., who in turn brought in engineers with industrial experience, including Robert Colton, an automotive engineer. Colton led the university Innovation Center experiment that gave rise to Carnegie Mellon’s CED. The NSF chose four universities that captured a range of approaches to innovation incubation. MIT targeted undergrads through formal coursework and an innovation “co-op” that assisted in turning ideas into products. The University of Oregon evaluated the ideas of garage inventors from across the country. The University of Utah emphasized an ecosystem of biotech and computer graphics startups coming out of its research labs. And Carnegie Mellon established a nonprofit corporation to support graduate student ventures, including the dial-a-ride service. Grad student Fritz Faulhaber holds one of the radio-coupled taxi meters that Carnegie Mellon students installed in Pittsburgh cabs in the 1970s.Ralph Guggenheim;Jerome McCavitt/Carnegie-Mellon Alumni News Carnegie Mellon got one of the first university incubators Carnegie Mellon had all the components that experts believed were necessary for innovation: strong engineering, a world-class business school, novel approaches to urban planning with a focus on community needs, and a tradition of industrial design and the practical arts. CMU leaders claimed that the school was smaller, younger, more interdisciplinary, and more agile than MIT. Dwight Baumann. Baumann exemplified a new kind of educator-entrepreneur. The son of North Dakota farmers, he had graduated from North Dakota State University, then headed to MIT for a Ph.D. in mechanical engineering, where he discovered a love of teaching. He also garnered a reputation as an unusually creative engineer with an interest in solving problems that addressed human needs. In the 1950s and 1960s, first as a student and then as an MIT professor, Baumann helped develop one of the first computer-aided-design programs, as well as computer interfaces for the blind and the nation’s first dial-a-ride paratransit system. Dwight Baumann, director of Carnegie Mellon’s Center for Entrepreneurial Development, believed that a modern university should provide entrepreneurial education. Carnegie Mellon University Archives The CED’s mission was to support entrepreneurs in the earliest stages of the innovation process when they needed space and seed funding. It created an environment for students to make a “sequence of nonfatal mistakes,” so they could fail and develop self-confidence for navigating the risks and uncertainties of entrepreneurial life. It targeted graduate students who already had advanced scientific and engineering training and a viable idea for a business. Carnegie Mellon’s dial-a-ride service replicated the Peoples Cab Co., which had provided taxi service to Black communities in Pittsburgh. Charles “Teenie” Harris/Carnegie Museum of Art/Getty Images A few CED students did create successful startups. The breakout hit was Compuguard, founded by electrical engineering Ph.D. students Romesh Wadhwani and Krishnahadi Pribad, who hailed from India and Indonesia, respectively. The pair spent 18 months developing a security bracelet that used wireless signals to protect vulnerable people in dangerous work environments. But after failing to convert their prototype into a working design, they pivoted to a security- and energy-monitoring system for schools, prisons, and warehouses. Wadhwani Foundation supports innovation and entrepreneurship education worldwide, particularly in emerging economies. Wharton School and elsewhere. In 1983, Baumann’s onetime partner Jack Thorne took the lead of the new Enterprise Corp., which aimed to help Pittsburgh’s entrepreneurs raise venture capital. Baumann was kicked out of his garage to make room for the initiative. Was the NSF’s experiment in innovation a success? As the university Innovation Center experiment wrapped up in the late 1970s, the NSF patted itself on the back in a series of reports, conferences, and articles. “The ultimate effect of the Innovation Centers,” it stated, would be “the regrowth of invention, innovation, and entrepreneurship in the American economic system.” The NSF claimed that the experiment produced dozens of new ventures with US $20 million in gross revenue, employed nearly 800 people, and yielded $4 million in tax revenue. Yet, by 1979, license returns from intellectual property had generated only $100,000. “Today, the legacies of the NSF experiment are visible on nearly every college campus.” Critics included Senator William Proxmire of Wisconsin, who pointed to the banana peelers, video games, and sports equipment pursued in the centers to lambast them as “wasteful federal spending” of “questionable benefit to the American taxpayer.” And so the impacts of the NSF’s Innovation Center experiment weren’t immediately obvious. Many faculty and administrators of that era were still apt to view such programs as frivolous, nonacademic, or not worth the investment.

Eight years is a long time in the world of patents. When we last published what we then called the Patent Power Scorecard, in 2017, it was a different technological and social landscape—Google had just filed a patent application on the transformer architecture, a momentous advance that spawned the generative AI revolution. China was just beginning to produce quality, affordable electric vehicles at scale. And the COVID pandemic wasn’t on anyone’s dance card. Eight years is also a long time in the world of magazines, where we regularly play around with formats for articles and infographics. We now have more readers online than we do in print, so our art team is leveraging advances in interactive design software to make complex datasets grokkable at a glance, whether you’re on your phone or flipping through the pages of the magazine. The scorecard’s return in this issue follows the return last month of The Data, which ran as our back page for several years; it’s curated by a different editor every month and edited by Editorial Director for Content Development Glenn Zorpette. As we set out to recast the scorecard for this decade, we sought to strike the right balance between comprehensiveness and clarity, especially on a mobile-phone screen. As our Digital Product Designer Erik Vrielink, Assistant Editor Gwendolyn Rak, and Community Manager Kohava Mendelsohn explained to me, they wanted something that would be eye-catching while avoiding information overload. The solution they arrived at—a dynamic sunburst visualization—lets readers grasp the essential takeaways at glance in print, while the digital version, allows readers to dive as deep as they want into the data. Working with sci-tech-focused data-mining company 1790 Analytics, which we partnered with on the original Patent Power Scorecard, the team prioritized three key metrics or characteristics: patent Pipeline Power (which goes beyond mere quantity to assess quality and impact), number of patents, and the country where companies are based. This last characteristic has become increasingly significant as geopolitical tensions reshape the global technology landscape. As 1790 Analytics cofounders Anthony Breitzman and Patrick Thomas note, the next few years could be particularly interesting as organizations adjust their patenting strategies in response to changing market access. Some trends leap out immediately. In consumer electronics, Apple dominates Pipeline Power despite having a patent portfolio one-third the size of Samsung’s—a testament to the Cupertino company’s focus on high-impact innovations. The aerospace sector has seen dramatic consolidation, with RTX (formerly Raytheon Technologies) now encompassing multiple subsidiaries that appear separately on our scorecard. And in the university rankings, Harvard has seized the top spot from traditional tech powerhouses like MIT and Stanford, driven by patents that are more often cited as prior art in other recent patents. And then there are the subtle shifts that become apparent only when you dig deeper into the data. The rise of SEL (Semiconductor Energy Laboratory) over TSMC (Taiwan Semiconductor Manufacturing Co.) in semiconductor design, despite having far fewer patents, suggests again that true innovation isn’t just about filing patents—it’s about creating technologies that others build upon. Looking ahead, the real test will be how these patent portfolios translate into actual products and services. Patents are promises of innovation; the scorecard helps us see what companies are making those promises and the R&D investments to realize them. As we enter an era when technological leadership increasingly determines economic and strategic power, understanding these patterns is more crucial than ever.

Eight years is a long time in the world of patents. When we last published what we then called the Patent Power Scorecard, in 2017, it was a different technological and social landscape—Google had just filed a patent application on the transformer architecture, a momentous advance that spawned the generative AI revolution. China was just beginning to produce quality, affordable electric vehicles at scale. And the COVID pandemic wasn’t on anyone’s dance card. Eight years is also a long time in the world of magazines, where we regularly play around with formats for articles and infographics. We now have more readers online than we do in print, so our art team is leveraging advances in interactive design software to make complex datasets grokkable at a glance, whether you’re on your phone or flipping through the pages of the magazine. The scorecard’s return in this issue follows the return last month of The Data, which ran as our back page for several years; it’s curated by a different editor every month and edited by Editorial Director for Content Development Glenn Zorpette. As we set out to recast the scorecard for this decade, we sought to strike the right balance between comprehensiveness and clarity, especially on a mobile-phone screen. As our Digital Product Designer Erik Vrielink, Assistant Editor Gwendolyn Rak, and Community Manager Kohava Mendelsohn explained to me, they wanted something that would be eye-catching while avoiding information overload. The solution they arrived at—a dynamic sunburst visualization—lets readers grasp the essential takeaways at glance in print, while the digital version, allows readers to dive as deep as they want into the data. Working with sci-tech-focused data-mining company 1790 Analytics, which we partnered with on the original Patent Power Scorecard, the team prioritized three key metrics or characteristics: patent Pipeline Power (which goes beyond mere quantity to assess quality and impact), number of patents, and the country where companies are based. This last characteristic has become increasingly significant as geopolitical tensions reshape the global technology landscape. As 1790 Analytics cofounders Anthony Breitzman and Patrick Thomas note, the next few years could be particularly interesting as organizations adjust their patenting strategies in response to changing market access. Some trends leap out immediately. In consumer electronics, Apple dominates Pipeline Power despite having a patent portfolio one-third the size of Samsung’s—a testament to the Cupertino company’s focus on high-impact innovations. The aerospace sector has seen dramatic consolidation, with RTX (formerly Raytheon Technologies) now encompassing multiple subsidiaries that appear separately on our scorecard. And in the university rankings, Harvard has seized the top spot from traditional tech powerhouses like MIT and Stanford, driven by patents that are more often cited as prior art in other recent patents. And then there are the subtle shifts that become apparent only when you dig deeper into the data. The rise of SEL (Semiconductor Energy Laboratory) over TSMC (Taiwan Semiconductor Manufacturing Co.) in semiconductor design, despite having far fewer patents, suggests again that true innovation isn’t just about filing patents—it’s about creating technologies that others build upon. Looking ahead, the real test will be how these patent portfolios translate into actual products and services. Patents are promises of innovation; the scorecard helps us see what companies are making those promises and the R&D investments to realize them. As we enter an era when technological leadership increasingly determines economic and strategic power, understanding these patterns is more crucial than ever.