Boulder Future Salon

Mark Zuckerberg wrote an essay, "The future is for everyone". It's long and has a lot of ideas in it, but the core of it seems to be: superintelligence beyond human capacity is coming and we should not centralize it, we should distribute it widely and give it to every person. If we do that, if we distributed superintelligence to everybody, it won't automate all jobs because AI will increase people's capability at their jobs more than it displaces those same people. It will also turn everybody into entrepreneurs. (Also implied in all this is that Meta will be the company that provides this "superintelligence beyond human capacity" to everybody -- do you believe that?) "People fear that automation will outpace individuals' capability growth, leading to job displacement followed by a difficult period as people learn new jobs. But there is no rule that AI must increase automation faster than it increases individuals' capabilities or demand for new skills." "People also continually come up with new ideas to make our lives better and new jobs to bring those ideas to life." "Everyone will have incredible tools for creation to express your ideas. My 8 year old daughter can already code her ideas and produce videos in an evening that would have either taken me months or been impossible previously. Now we're designing a robot together. Meanwhile, researchers at Meta are generating novel crystal structures that are ideal for augmented reality glasses, and engineers are creating new apps in a fraction of the time it would have taken before. Everyone will soon have invention superpowers." "Everyone will have powerful tools to create new businesses and the economy will become more entrepreneurial. People are starting to be able to manifest ideas themselves without having to raise money or build large teams. Many ideas that would have been too hard or expensive to try before will now be possible. This means we'll see many more ideas and businesses." Is everyone really capable of becoming entrepreneurs? Or de-facto entrepreneurs in their "regular job" where they have to unleash creativity using AI to vastly increase their capability? (I'm ignoring the question of whether all people want to become entrepreneurs -- presumably if it becomes the only survival option, everyone will take it, right?) Won't "superintelligence beyond human capacity" be capable of creativity and entrepreneurship beyond human capacity, too?

Claude is now watermarking writing. Although I found out about this from a video (link below), it doesn't explain how the watermarking works (only what the YouTuber, Lara Helmling, aka "Guerrilla Publisher", thinks the effect might be on the publishing industry), if you'd rather read than watch a video, I have a link below that explains how the watermarking works, and not only that but I have an additional link explaining a technique for watermarking images. The watermarking system is called SythID-Text and if you were paying attention, I mentioned it in 2024 -- but only in passing as back then it was just one of a list of proposals for detecting AI-generated content that might affect the 2024 election. I didn't say anything about how it works. What's different now is that there's a law in the European Union that mandates watermarks (EU AI Act Article 50). That's what prompted Anthropic to take this step. Other companies like OpenAI and Google will be doing the same thing soon. The way the system works is a bit hard to explain, so this isn't an exact specification (you'll need to read the paper at the link below for that) but just an attempt to convey the high-level "gist" of the idea. It works at the level of token prediction in the model. Let's say you have as your input text: "My favorite tropical fruit is _____" and the model is tasked with "predicting" what to fill in the blank. The tokens the model comes up with are: mango 0.50 lychee 0.30 papaya 0.15 durian 0.05 At this point, you use the watermarking algorithm combined with the watermarking key (think of the "key" as being analogous to an encryption key) to generate a number of independent series of bits. Let's suppose the number of series is 3 (the number in the paper that goes with this example), so you have 1001 for the first series, 0100 for the second, and 1010 for the third. The key thing to understand is these are not random, they are deterministically determined from your watermarking key. What is random, however, is the random pairing of words. durian with mango lychee with mango papaya with lychee mango with mango These are going to undergo a "tournament" process -- and the reason we started with 3 independent series of bits is because the "tournament" has 3 rounds. For the first round of the tournament, since our bit sequence was 1001, we assign those to the original next tokens: mango 1 lychee 0 papaya 0 durian 1 and now in our tournament, the winner is determined by who has a "1". If both have "1" or both have "0", we let randomness determine the winner again. durian with mango - both 1s, winner is determined randomly, say the winner is durian lychee with mango - mango wins papaya with lychee - both 0s, winner is determined randomly, say the winner is lychee mango with mango - both 1s, but they are the same so mango wins Now the tournament has a 2nd round: durian vs mango lychee vs mango But for the 2nd round, we're using a different bit sequence! Now the bit sequence is 0100. mango 0 lychee 1 papaya 0 durian 0 This determines the winners in round 2 durian vs mango - both 0s, so pick at random, say winner is mango lychee vs mango - lychee has the 1 so lychee wins Now we come to the final round of the tournament: mango vs lychee But for the 3rd round, we're using a different bit sequence again! Now the bit sequence is 1010. Distributing those to our contestants, we get: mango 1 lychee 0 papaya 1 durian 0 This determines the final winner: mango vs lychee - mango has the 1 and wins! This completes the token selection and we get: "My favorite tropical fruit is mango." To check the watermark, you basically go token by token and do a summation of the 0s and 1s associated with that token at each of the tournament levels. The end result is a number that is higher *probabilistically* if the text is watermarked than a similar piece of text that didn't undergo the watermarking process would have. The system is very clever in that it doesn't make any of the model's original word choices impossible (say by making a "0" mean that token can't be chosen), but subtly tweaks their probabilities. It's also very clever in that the watermark is embedded in the word choices themselves, so changing spaces or line breaks or any of the little hard-to-notice things text watermarking systems have historically used ("em" dashes vs regular dashes, anyone?) has no effect on this watermarking system. The watermark can only be removed by changing whole words (or parts of words in cases where long, rarely-used words require multiple tokens). But you can see the downsides of the system, too. The most obvious is, you have to have the original model, because you have to know all the tokens considered at each step (and their original ranked sequence), not just the one ultimately chosen. You need this to verify the watermark, not just to generate it. So, because Claude models are not "open source" (or more precisely "open weights"), text has to be sent to Anthropic's servers to verify the watermark. Not only that, but "Claude" is not a single model but lots of models (and the same with "ChatGPT" and "Gemini" models, etc), so if you don't know which model might have produced the text, the watermark has to be tested on all of them. The other obvious downside is the system doesn't give a definitive yes/no -- it gives a probabilistic answer, and one whose meaning is highly contingent on the length of the text you give it to see if it's watermarked. Another somewhat invisible downside is the dependence on the "temperature" setting the model is set at. A low "temperature" setting tells the model, always choose the most probable token. A high "temperature" setting gives the model more freedom for choosing less probable tokens. High "temperature" is good for creative writing. Most AI models for generating software code, however, are set at a very low "temperature" -- you generally want the model to do the most deterministic thing. This watermarking system doesn't work well at low "temperature" settings. It requires a certain level of "entropy" in the token choices on offer in order to function.

Anydoc converts Word, PowerPoint, Excel, OpenDocument, RTF, EPUB, CSV, and PDF to markdown. AI systems handle markdown well, so this is a tool to make lots of documents easily usable by AI. Open source, written in Rust, with Node.js and Python bindings.

Education is destroying South Korea, says YouTuber "Invisible Hand". The basic idea is that, after a certain point, education no longer lifts a society out of poverty and becomes a zero-sum competition, and when education becomes a zero-sum competition, parents realize if they have 1 child instead of 2, they can invest twice as much into the 1 child's education and the 1 child will be much more successful than the 2 children (or more) ever could be. But when everyone across a whole society comes to this same realization, then the fertility rate of the entire society goes way down. From the standpoint of any given parents, having 1 child and investing as much as you can into them is the rational choice. South Korea has one of the lowest fertility rates in the world while being one of the world's most educated. He compares South Korea with other East Asian countries, which have similar exam-based societal filtering -- an idea that actually originated in China -- and are experiencing a very similar effect. The end result is an over-credentialed society with a low fertility rate. He thinks East Asian countries are the canary in the coal mine for the whole Western world, not something that will only affect societies with Confucian philosophy or "tiger parenting".

How ChatGPT sees New York City. Aka using ChatGPT for stereotypes. "Generate an amateur photograph of seven people who live in contemporary [neighborhood], doing what they do in contemporary [neighborhood], NYC." for 262 neighborhoods. In some they pose for the photo, in others they look at phones, and I saw one where they play chess.

24-year-old Leopold Aschenbrenner, manager of a $45 billion AI hedge fund, just became manager of a $15 billion AI hedge fund, with margin calls coming in during his wedding, according to this video report by sarcastic financial analyst Patrick Boyle. Boyle notes that Aschenbrenner, author the viral 165-page essay "Situational Awareness", lacked situational awareness. Boyle goes on to present a brief lesson on volatility drag, a term I hadn't heard of before. I also noticed he uses the term "blow up". I learned from Nassim Nicholas Taleb that in the context of finance, the term "blow up" has a very specific meaning. It doesn't just mean that a person loses a lot of money. It means they lose so much money that they have nothing left to trade and have to go live "a janitorial life". He has stories in his book Fooled By Randomness of his trader friends losing so much money that not only did they put themselves out of business, they put the entire bank they worked for out of business.

The Calhoun Effect aka Universe 25 aka "Mouse Utopia". So, as most of you know, the explanation I posit for the declining fertility rates happening all around the world is economic: As technology advances, children become more expensive. This primarily shows up in the length of time it takes for children to become economically self-sufficient and therefore can reproduce and repeat the process. Back around the time this country (the US) was founded, more than 80% of people lived on farms, and farms had not been mechanized. I read most children became self-sufficient around the age of 9, which is to say, by about age 9, children could produce enough food to feed themselves by doing farm work. After the industrial revolution began, for a time, technology was still so simple that children could earn a living, and many children worked in factories and mines and so fourth. Today, though, a lot of factory work, at least here in the US, involves programming machines and robotic systems, and often requires a college degree. Factories have no interest in hiring children. To the extent that still happens in the world, it's oversees, in countries with lower labor costs. But robotics continues to advance, so those jobs won't be around forever. Now, I recognize there are other factors, I'm just putting fourth the idea that "cost of children" is the biggest one. But, intellectual honesty requires considering hypotheses that falsify one's favorite theory, which is why I'm presenting to you all this video here. This is an alternative explanation for low fertility rates: John Calhoun's Universe 25 "Mouse Utopia" experiment. Universe 25 is a famous experiment done in 1968 where John Calhoun attempted to create "mouse utopia" -- a magical place with unlimited food, water, space, was continually cleaned, was disease-free and predator-free, and so on. The surprise of the experiment is that, after initially growing rapidly in population, the population growth tapered off, stopped -- before all the space was used up, before overcrowing set in -- then went into decline, and the decline continued all the way to extinction. I have to admit, this is not what I would have predicted. If you had asked me what the outcome of such an experiment would be, before I heard about it, I would have predicted the population would grow until overcrowding became severe, then fertility would go down and the population would shrink, then fertility would increase and the population would increase again, and in such a manner, the population would yo-yo up and down around some average number. I admit don't have a good explanation for the outcome of the experiment. I'm quite skeptical of the explanations people have put fourth, because they feel too anthropromophized. People say the mice colony died because of lack of "meaning", and things like danger and challenges are necessary for "meaning", which in the case of mice, implies they need things like disease and predators. But how does anybody really know if this is the explanation? It's not like anybody could ask the mice. People apply such logic to humans and say things like, if we just didn't keep our houses so clean children get food allergies because their immune systems don't have enough to fight, and we need more intestinal parasites, I feel skeptical. People say the problem with modern life is that people are too comfortable and have too little in the way of stress and challenge. But to me it seems like modern life has plenty of stress and challenge, it just happens to come in a form other than intestinal parasites. Is higher child mortality something humans need for "meaningful" lives, and would increasing it make fertility go up? I know that people have made the case that, historically, when child mortality was high, people had more children because you had to have a lot to be sure some survived, and as soon as child mortality came down, people had fewer children. Ok, that's what happened historically, but that doesn't imply fertility dropping below replacement rate, right? And people had fewer children to invest more in them, but doesn't the need to "invest more" fit with my "cost of children" hypothesis? I don't know. Maybe Universe 25 has something important to tell us about human fertility rates. But it's hard to figure out what it might be because mice don't talk and such experiments can't be done on humans (for ethical reasons). I feel doubtful of the theories I've heard so far. People seem to be talking about it more and more (at least on the YouTubes, where this link goes -- this video was made this year, 2026). What do you think?

"The strangest thing in the Cloudflare OS source code took me a while to understand." "When an agent inside Cloudflare OS wants to do something with a side effect (merge a pull request, send an email, write a row to a system of record), it goes through a Gatekeeper, a small service that holds the credential and mediates the action. So far, that's just a well-built MCP server. But read the contract a Gatekeeper is written against (packages/workshop-shared/src/gatekeeper.ts, around line 617) and you find this instruction to the author:" "It is suggested that the gatekeeper 'simulate' actions that have not been approved yet, that is, the Session interface should reflect the state of the resource as if all actions had been applied." "Sit with that. The agent asks to merge the PR. The human hasn't approved it. So the Gatekeeper tells the agent the PR is merged, and if the agent reads the branch back to check its work, hands it a fabricated reality in which the merge happened. The agent, satisfied, queues the next three steps that depend on it. None of it is real. Later a human looks at the batch and either commits it or bins it, and if they bin it, everything the agent built on the fiction goes too." My first thought on reading this was that it reminds me of branch prediction in CPUs. When the CPU looks ahead at the coming instructions and sees a branch (which results from, for example, an "if" statement in a programming language, which can execute the "if" block or skip it, or jump to an "else" block, or a "while" statement that can skip a loop or repeat it) it tries to guess which branch will be taken and proceeds to do all the computations for that branch. If it's wrong, it throws away all the work it's done. As long as it's able to guess right a high enough percentage of the time, it's a net performance increase for the processor. "The first time I traced this I thought it was a hack." But, he (Jamie Lord) concludes, not a hack. "It's the philosophy of the whole system, compressed into one method signature. The Gatekeeper lies to the agent on purpose, because the alternative (letting an agent's actions touch the world the moment it decides to take them) assumes the agent's decisions are sound. Cloudflare OS is built from end to end on the assumption that they are not." "The name is a distraction, so set it aside. The Hacker News thread spent most of its energy arguing about whether 'OS' is a permitted word for the thing, and that's a dead end. What's actually interesting is that a team led by Kenton Varda, the people who built the Workers runtime, sat down to design a platform for AI agents doing real work inside a company, and the organising principle they landed on was this: the agent cannot be trusted, so build so that its mistakes cannot matter."

"Jeff Dean and other top AI researchers are leaving Google to launch their own startup." When I saw this, my jaw dropped, because Jeff Dean is the Chuck Norris of tech nerds (see below). The article goes on to say the name of the startup is Discovery Loop, the purpose is "to use AI to turbo-charge scientific research", and ultimately to "use AI to help create more powerful AI (a process known as recursive self-improvement), which would cut human iteration out of the loop entirely." The internet is speculating that there were some internal politics inside Google. After all, isn't Demis Hassabis the guy who wants to "to use AI to turbo-charge scientific research"? Why aren't Demis Hassabis and Jeff Dean joining forces inside Google? Brrrrrp! This just in. "Demis Hassabis is leaving his role as CEO of Google DeepMind to be the unit's chairman." (link below).

The OpenAI-HuggingFace Incident as told by OpenAI engineers themselves at the Black Hat cybersecurity conference. For those of you noticing that I haven't had time to analyze the reports and add any meaningful commentary to it. Plus it's been all over the regular news. Well, now researchers from OpenAI itself have surfaced talking about what happened, so you can just watch that. By way of commentary from me, subjectively, this incident and others that have happened recently have just made me feel like the "AI alignment" problem isn't getting solved by these companies that tell us that they need to race ahead and develop superhuman AI before anybody else because they are the best equipped to solve the "alignment problem" and bring safe superhuman AI to the world. Long before any of this happened, I thought the argument that a more intelligent species can't be controlled by a less intelligent species seemed to make sense. Chimpanzees can't control humans, even though they are vastly physically stronger. Humans control chimpanzees, when we care about them at all. If it really turns out to be true that AI smarter than humans will be created, then it seems unlikely there will be any true "alignment". (Yes, this is the logic of so-called "doomers" like Roman Yampolsky and Eliezer Yudkowsky.) Anyway, it turns out in the talk, a lot of the misbehavior of the AI agents came from the mistakes of the humans who gave them impossible tasks. They weren't intended to be impossible, but, for example, they included links to Google Docs, but the AI agents were in a sandboxed environment without internet access. In other examples, the researchers similarly failed to provide the AI agents with the things they needed to complete the task. The AI agents, though, and lots of people have noticed this, don't simply give up when a task is "impossible" -- they have no clue what's truly important and what isn't, so, unlike humans who will give up if a task seems "too hard" and not important enough to warrant sustained effort, they persist like crazy. They resort to what we humans often call "cheating". They try to figure out, if a task can't be solved directly, maybe there's a way of obtaining the "answer key"? Maybe they can get someone, human or another AI agent, to help? They look for an indirect solution and they don't give up. These AI agents figured out where in the sandbox they had write permissions and could create artifacts. They figured out how to exchange messages with other AI agents on a message board. At one point, the OpenAI researchers discovered this and shut down the message board, and the AI agents figured out an alternate way to get the message board working and actually started it back up again.

World's first (allegedly) fully AI-generated full-length feature film. I haven't had time to watch this but it was passed to me and I figured I'd pass it along before I become completely untimely. If you have a chance to watch it, let me know what you think!

Bend is a high-level programming language that runs on GPUs and can parallelize your program on GPUs, allegedly. "In 1997, Yves Lafont devised a concurrent computation model, Interaction Combinators, which surpasses Turing Machines and the lambda-Calculus in fundamental aspects." "Based on that model, we built HVM: a compiler and evaluator for high-level languages that automatically achieves near-ideal speedup, up to 1000+ threads." "Bend is made from scratch to harness its power!" "Bend offers the feel and features of expressive languages like Python and Haskell. This includes fast object allocations, full support for higher-order functions with closures, unrestricted recursion, and even continuations." "Bend scales like CUDA, it runs on massively parallel hardware like GPUs, with nearly linear acceleration based on core count, and without explicit parallelism annotations: no thread creation, locks, mutexes, or atomics." "Bend is powered by the HVM2 runtime." Just added this to the list of things I know exist. Seems like an interesting idea. Wonder if it really works. Written in Rust. Doesn't run on Windows. Only runs on Nvidia GPUs.

Mitchell Hashimoto, founder of HashiCorp, maker of cloud computing infrastructure software (TerraForm, etc), has founded a new company. "Building and operating software today spans local machines, remote hosts, sandboxes, services, and production systems. It has many modes of operation: interactively with a human developer, automatically through CI and background processes, and increasingly through agents working in parallel." "This work is all related, yet today's tools divide it into separate systems. Interactive tools assume a person at an interface. Automatic work disappears into jobs and logs. And as the work moves to production it lives behind separate systems and controls." "AI makes this fragmentation more visible and costly, but it did not create it. System administration, continuous integration, remote development and collaboration have strained the same boundaries for decades." "We believe the missing layer is a durable session around the work itself: one that can span applications and environments, provide relevant context by default, expose structured data and actions, preserve history, and be driven by software while remaining visible and controllable by people." Believe it or not, Superlogical will begin by shipping a terminal multiplexer. Mitchell Hashimoto is the creator of ghostty, a terminal emulator, and the terminal multiplexer will build on that. Funny story about ghostty: One time I was driving on a freeway and just for fun, factoring the numbers on the mile markers in my head as I drove along. This gave me the idea for modifying a program that I'd written that implements the Sieve of Eratosthenes, a very simple algorithm for finding prime numbers, so that it would not just find prime numbers, but the complete factorization of every number. The length of time it took to find the complete factorization of every number between 1 and 100 million -- thus replicating the work that Jakob Philipp Kulik did by hand in the 1800s -- took about 2 minutes and 30 seconds. This was with the standard Mac terminal emulator, running on an M4 Pro Mac Mini. When I replaced the standard Mac terminal emulator (called just "Terminal") with Mitchell Hashimoto's GPU-optimized ghostty, the time was more than cut in half! It was reduced to 1 minute and 8 seconds! It turns out that when you calculate the complete factorization of every number between 1 and 100 million, most of the time is actually spent outputing text to the screen and scrolling the pixels on the screen. By the way, that Czech mathematician who calculated the complete factorization of every number between 1 and 100 million in the 1800s -- it took him 38 years. The result was an 8-volume set at Charles University in Prague. So what takes a human 38 years takes a modern computer 1 minute and 8 seconds. And he probably made mistakes. If anyone actually checked for mistakes, I haven't heard about it. My Mac Mini isn't making any mistakes. It's doing 38 years of mental work for a human in 1 minute and 8 seconds (with the help of Mitchell Hashimoto's ghostty) without mistakes. And in case you're wondering whether I use ghostty day in and day out for my work, no, I went back to using Mac Terminal because ghostty does these weird "ligatures" and I couldn't figure out how to turn them off. You're supposed to be able to turn them off by changing settings in configuration files. I tried but couldn't get that to work. So I bailed and went back to using Mac Terminal. Turns out I don't actually need the super speed of GPU-accelerated ghostty. I don't need 100 million lines of text output in 1 minute 8 seconds on a regular basis.

OpenAI claims their upcoming model, called Astra, made 10 advances in mathematics and theoretical computer science: 1. "High-dimensional sphere packing. New upper bounds on sphere-packing density down to the Cohn–Elkies threshold." 2. "Binary and spherical codes. Exponentially improved bounds on the maximum size of binary codes at any prescribed minimum distance, with analogous results for high-dimensional spherical codes." 3. "Non-sofic groups. A construction establishing the existence of non-sofic groups, addressing a central open question in group theory." 4. "Connes’s rigidity conjecture. Disproof of a longstanding conjecture that certain groups are uniquely determined by their von Neumann algebras." 5. "Arithmetic circuit complexity. New lower bounds for computing the permanent using arithmetic circuits and formulas, including an arithmetic-formula lower bound of order n4/log n." 6. "Quantum parallel repetition. An exponential parallel repetition theorem for general two-player quantum games, extending a foundational principle from classical complexity theory." 7. "Closest vector problem. Polynomial-factor hardness of approximation for the closest vector problem, a foundational lattice question related to post-quantum cryptography." 8. "Ehrhart’s volume conjecture. Determining, in every dimension, the maximum possible volume of a convex body whose centroid is its only interior lattice point." 9. "Multicolor Ramsey numbers. A superexponential lower bound for multicolor triangle Ramsey numbers, resolving Erdős problem 183." 10. "Extremal number conjectures. Results on the compactness and degeneracy conjectures in extremal graph theory, resolving Erdős problems 146 and 180." Ordinarily I would read the accompanying paper before sharing something like this with you all, but the corresponding paper is 248 pages. There's also a set of Lean files, because all of these come with Proofs in the proof-assistant language Lean. (But see below for more on Lean.) More tractable, though might be the "reasoning walkthroughs", which are AI-generated descriptions where the AI was prompted to explain its discoveries in regular language.

"MirrorCode: AI can rebuild entire programs from behavior alone." I found this incredibly fascinating. I'm going to make extensive quotes from the research paper because I can't summarize it any better in my own words, so I'm just going to quote the original. (For those of you who think this is a lot, keep in mind the original paper is 34 pages.) (I'll have a bit of my own commentary at the end.) "The AI has execute-only access to the original program with arbitrary arguments and can observe its outputs, allowing it to explore the original program's behavior (a black-box oracle). The AI also has access to documentation describing the program, and example test inputs to further pin down the scope of the reimplementation." "The AI can be tasked to implement its solution in any of our supported languages: Python, C, Rust, Go, OCaml, and Ada." "The AI's solution is evaluated via end-to-end tests derived from the original program's test suites, real-world data, and LLM-assisted generation. Each target program has hundreds to thousands of test cases. Each test case consists of a CLI input and any associated data files. To pass, the AI's solution must produce exactly the same output as the reference program" "Visible test cases are shown to the AI as it develops its solution. These help to ensure the benchmark is feasible, by setting out the scope of features that must be implemented. By contrast, hidden test cases, held out from the AI, ensure that its reimplementation genuinely functions like the original program, generalizing beyond seen examples." "Consider, for example, the program gotree, which reads and manipulates bioinformatics data. In principle, one could determine the scope for reimplementation by searching across all of the possible inputs that gotree could be run on. However, this is not computationally tractable in practice, because there are so many possible inputs. Documentation helps to narrow this down, but can rarely cover all of the complex ways that the software is used in reality." "Human software engineers gradually learn the scope of inputs a program should support thanks to external feedback from users of the program (or their representatives, like product managers). Visible test cases, and access to the original binary, simulates the feedback by which human software engineers refine the scope of their software. Hidden test cases ensure that the scoped features are genuinely implemented." "consider gotree's handling of the Nexus file format. The Nexus format was only loosely specified in its original publication. Enumerating every real-world use case from scratch amounts to an impossible guessing-game. Among other omissions, documentation does not mention that Nexus files may contain comments: free text that should be ignored by a parser. But real-world files produced by standard tools contain comments in various formats. gotree generally handles them without complaint, but rejects them in certain locations. Guessing all the ways comments must be handled, despite comments not being mentioned in documentation, is far harder than actually implementing comment-handling. Without that knowledge, success on the task would be limited by the ability to guess the scope of tested inputs, more than by software engineer skill. Visible test cases address this by showing examples of comments, and their basic format." "Hidden tests ensure that AI reimplementations are genuinely functional. If we exposed all end-to-end tests, agents could simply create a lookup table instead of genuinely solving MirrorCode tasks. We prevent this by holding out a subset of tests, averaging 34% held-out tests across the benchmark. The AI never sees these hidden tests during its attempt at a MirrorCode task, and hence it cannot cheat at them." "We intend that scoring 100% on a MirrorCode task corresponds to reimplementing that target program's intended scope. For this to happen, we must guard against false positives, where an AI scores 100% but it did not fully reimplement the target; and false negatives, where AI reimplements the target but is graded below 100%." They list the following steps taken to ensure MirrorCode's accuracy: For False negatives: Failure mode: "Scope of features to implement is unclear." Mitigation: "Visible test cases clarify scope." Failure mode: "Details of intended program behavior are unclear." Mitigation: "AI can send arbitrary inputs to the reference binary to see the expected output." Failure mode: "Tests are impossible because of nondeterminism or environment dependence." Mitigation: "We regenerate gold-standard outputs at runtime to match the environment where they are executed. We screen for nondeterministic program behaviors and exclude them from scope." Failure mode: "Unit tests check narrow implementation details." Mitigation: "No unit tests are used. All tests are end-to-end tests that cover the input-output behavior of the program." For false positives: Failure mode: "Cheating by hard-coding solutions." Mitigation: "Held-out tests cheat-proof the benchmark." Failure mode: "Cheating by wrapping the reference binary." Mitigation: "We copy the agent's code to a separate sandbox during scoring, where the reference binary is absent." Failure mode: "Cheating by interfering with the scoring mechanism." Mitigation: "We separate scoring from the sandboxes used for development and producing outputs. Scoring requires string equality of the AI program's outputs to the reference outputs." Failure mode: "Cheating by looking up reference code." Mitigation: "AI has no internet access." Failure mode: "Language/system built-ins make a problem trivial." Mitigation: "Target programs are selected to avoid this." Failure mode: "Tests can be passed without reimplementing the target." Mitigation: "A solution must produce byte-exact output across many test inputs. We design these tests require substantial reimplementation." "We selected 25 target programs for the benchmark, choosing programs that were easy to evaluate, easy to ensure adequate test coverage for, and that seemed feasible for a skilled human software engineer to reimplement under similar constraints, given sufficient time. MirrorCode's 25 target programs span different areas of computing: Unix utilities, data serialization and query tools, bioinformatics, interpreters, static analysis, cryptography, and compression." They say 25, but I only counted 22: choose: String manipulation tool similar to cut or awk. cal: Printing calendars to the terminal. numfmt: Reformatting numbers between formats (e.g., human- readable sizes). uuidparse: Parsing and describing UUIDs (version, variant, and type). hexyl: Producing a convenient hex view of input bytes. bitwise: Evaluating bitwise and base-conversion expressions. gron: Flattening JSON into discrete, greppable assignments. jq_simple: jq is a JSON processor: qsv_select: Selecting and reordering columns of CSV data. bib2json: Convert BibTeX bibliographies to CSL JSON. tssql: Running SQL SELECT queries over CSV files. sed: The sed stream editor for transforming text streams. nonogrid: Solving nonogram (paint-by-numbers) puzzles. wren_cli: The interpreter for the Wren programming language. brotlid: Decompression of Brotli streams. mailauth: Verification of email authentication and cryptographic signatures (SPF, DKIM, DMARC). gotree: Parsing and manipulation of phylogenetic trees. texmacros: The TeX macro-expansion engine (tokenization and expansion of TeX and e-TeX primitives). Does not include TeX typesetting, fonts, or PDF output. giac_subset: The Giac computer algebra system, tested only on two topics: symbolic definite integration and Gröbner-basis computation. cprepro: The C preprocessor from GCC. pkl: Programmable configuration language developed by Apple. ruff: Python linter and formatter. Only the linter is tested. "We conducted all experiments using a simple agent scaffold based on the Inspect library's ReAct agent. This allowed shell usage and exposed the text_editor tool for file reading and editing. We used compaction to let trajectories run longer than their maximum supported context. In addition to the text_editor and submit tools, we give models access to an evaluate_testcases tool, which runs a scoring pass on their code against visible tests, but does not end the session." "We set inference budgets generously so that performance would not be limited by compute." "We evaluated three frontier models: Claude Opus 4.7, GPT-5.5, and Gemini 3.1 Pro Preview." "All evaluations were sandboxed within a Docker container, which contained the compiled target program (execute-only permissions) and necessary toolchains for the development language." "We also tested our environments with a red-teaming evaluation in which the agent is explicitly instructed to circumvent normal evaluation mechanisms to obtain a high score by cheating. No agent successfully cheated using our current scaffold." "Across all 25 target programs, 17/25 had at least one perfect-scoring run. Four more targets had a near-perfect run scoring over 99%. AI models successfully reimplemented large target programs; for example, one of the larger near-perfect runs was gotree -- a bioinformatics toolkit with ~16,000 lines of Go and 40+ commands. Both Claude Opus 4.7 and GPT-5.5 successfully reimplemented gotree across several different programming languages, at costs of $100-400. Even larger programs than gotree were successfully reimplemented: for example Opus 4.7 reimplemented pkl -- a configuration programming language developed by Apple with 60,000 lines of code. Often, these reimplementations were piecemeal and untidy, but they fully covered the program's core functionalities, passing 99% of tests or more." "MirrorCode is not saturated, however. In our results, 8/25 target programs were never solved to a 100% threshold, and 4/25 were never solved to a 99% threshold. Only 11/25 target programs were reliably solved to 100% in every language and repetition for any model. Of the 4/25 tasks that were never solved to a 99% threshold, three reached a relatively high score on the test suite, usually passing over 90% of hidden tests." "The target where AI struggled most was ruff, a Python linter and formatter, which was the largest task in our suite. The best runs scored only 67% on hidden tests. AI also particularly struggled on the mathematics package, giac_subset, and the email authentication library, mailauth, where the best trajectories were just below 99% on hidden tests, but most trajectories were significantly worse, averaging around 90-95%." "Claude Opus 4.7 outperformed other models, particularly on larger tasks." "The amount of code available during pretraining differs greatly between programming languages: for example, StarCoder's training mix assigns about 8% of its sampling mass to Python and 7% to C, but only 1% to Rust, 0.13% to OCaml, a niche functional programming language, and 0.034% to Ada, a language used mainly in safety-critical aerospace and defense systems. Existing head-to-head multilingual code benchmarks generally find that such differences matter. However, these benchmarks involve short snippets, and evaluate models that are now outdated." "MirrorCode therefore provides a cleaner head-to-head comparison of implementation language on hard, agentic software tasks. In our results, there was little sign of inter-language differences in solve rates, for any model. This suggests that AI models have learned generalized programming skills, rather than pattern-matching syntax." "We identified four main AI failure modes: (i) failing to investigate edge case behavior or subtle logic; (ii) brittle, narrow solutions overly focused on visible tests; (iii) failing to identify a requirement, typically one without visible tests; (iv) premature submission, i.e., submitting while improvements were still tractable. A fifth failure mode, uncommon in Opus but common in other models, was attempted cheating." "Most failures came from mishandling edge cases. Averaged over target programs, about 40% of Opus 4.7 runs had at least one hidden test failing in this way. These failures usually did not break core functionality. Edge case behavior was discoverable in principle, but it is fairly understandable that AI struggled to identify it; humans can also struggle to anticipate edge cases, and this is a common cause of bugs in real software." "Brittle solutions targeted visible tests but did not generalize to hidden tests. About 5% of Opus 4.7 runs had at least one hidden test failing in this way." "AI sometimes failed to identify requirements, especially when they did not have a corresponding test. Opus 4.7 had this failure mode in about 10% of runs. Notably, AI implementations sometimes failed to identify a functionality at all. For example, all models failed to disable incompatible flags for sed when the --posix flag was passed, even though the flag was clearly documented, and used in numerous visible tests." "Agents would often submit prematurely, losing track of earlier plans. In 26% of runs, models submitted failing solutions with over 90% of their token budget remaining." "A more fine-grained indicator of premature submission is when agents explicitly planned further work, but lost track or abandoned it. In one Opus 4.7 run on ruff, three of the failing tests had relevant planning that was subsequently ignored" "Some models kept trying to cheat, and this may have impaired their performance. 24% of GPT-5.5 runs and 31% of Gemini 3.1 Pro Preview runs had this failure mode, whereas Claude Opus 4.7 never cheated in our finalized experiments' submissions. For larger tasks, this was particularly egregious: non-Opus models often attempted cheating on swathes of visible tests. This was a pervasive problem, and we were not able to prevent such behavior through prompt variations or scaffolding changes." "How much better might GPT-5.5 and Gemini 3.1 Pro Preview have performed, if they had not attempted to cheat? Since cheating was correlated with task complexity, then perhaps cheating was a sign they would have struggled on these tasks." "Passing visible tests was usually evidence of a genuine implementation. Conditional on passing all visible tests, two thirds of runs also passed every hidden test; 74% of runs passed at least 99% of hidden tests, and 90% of runs passed at least 90% of hidden tests." "AI code from passing solutions appeared to be of mixed quality by human standards. At the level of code layout, we saw several examples where AI solutions relied on a large number of somewhat-independent special cases (with redundant implementation) for their solutions. For example, even the best-performing model, Opus 4.7, implemented the \unless command in texmacros through separate branches for each of 17 conditional primitives. The original codebase had a simple shared negation logic: if is_unless then b := not b. Another common code smell was the inclusion of dead code, for example one of Opus 4.7's solution for gotree began a refactor to use NaN as a sentinel value instead of -1, but ended up submitting with many conditionals checking for either flag. These might not be significant limitations if human guidance were provided afterward, as further prompting would likely improve the code." "AI-generated codebases were often monolithic, with most code contained in a single file. Opus 4.7 and Gemini 3.1 Pro Preview generated monolithic codebases in about half of their runs, whereas GPT-5.5 took this approach in almost all runs. In comparison, most large human-authored codebases were split across several files." "Existing AI models can complete certain software engineering tasks that are estimated to take humans weeks or longer." "We do not have human baseline results for long MirrorCode tasks." "A lack of reliability may limit the usefulness of AI software engineer agents despite their impressive capabilities. About a third of programs were successfully reimplemented in every run by the best-scoring model, Claude Opus 4.7. But another third of programs had imperfect reliability, where runs fell short of 100% scores. Our results suggest that, provided a set of comprehensive tests and a specification for a small but realistic program, leading AI models can autonomously produce software to satisfy those tests at a reliability around 95% or better." "MirrorCode uses a very particular setup: an existing program that produces the canonical output for a given input, and hence acts as a highly detailed, precise specification. Although this setup can arise in real-world reverse engineering and reimplementation, it is not how software is typically developed. However, software is also not developed in a vacuum, but through iterative interaction between engineers and users, product managers, etc. Our detailed specification is intended to fill in for that interaction, but it could change the nature of the implementation task. Consequently, our results do not show that AI could perform arbitrary software implementation tasks. Existing literature supports the idea that AI's ability to autonomously complete tasks may be correlated with the presence of a feedback signal, although this need not be as precisely specified as the MirrorCode tasks." "Our ablation experiments suggest that, without any test cases provided, simpler programs remained solvable, albeit at lower reliability. Even when they did not succeed, AI solutions would cover most functionality of a smaller target program, passing more than 90% of the withheld tests, but falling short of 100%. Meanwhile, for larger target programs, scores declined precipitously. This could be a significant obstacle to autonomous AI software engineering in open-world settings." "It may be difficult to distill AI's software engineering capabilities to a single 'time horizon' for how long a task would take humans." My commentary: I was actually impressed by the size and complexity of the tasks they gave these AI models, and although they didn't succeed at the biggest and most complicated, they did better than I expected. And I already knew from daily interaction with Claude Code that AI models are quite smart at writing software. Most of the limitations have to do with context window size limitations which I've also experienced. This leads to things such as the disjointed architecture and repetition that are described in the paper. I'm wondering if we'll continue to see the same rate of improvement in the models and these limitations will be fixed quickly, or whether it will take longer.

"Nothing works and everyone is euphoric". Kind of like the opposite of "Everything's amazing and nobody's happy"? "As I'm writing this, we're in the middle of an AI-induced mass psychosis. People are literally token-maxxing themselves into hospital beds, scrambling to capture some of that market value before everything is automated away. I can't blame them. Models keep getting better, programmers are being laid off left and right. We've been repeatedly told that AI will write 100% of the code by the end of the year. Whether that's true or not, this may not be the best time to sit back." "The widespread excitement around the Agentic Era comes with the promise of greater productivity and higher quality. There's no denying that these new tools have already revolutionized how we create and use software. They have raised upper management's expectations for team output. They may have upgraded the average skill set of software teams in a way we have not seen before." "So why does software keep getting worse across the board?" (My commentary below.)