AI regulation: four dogs that ought to be barking - guest post by Tim O'Reilly

As part of our series on AI Risk Management and Financial Regulation, Tim O'Reilly explores the risks that AI safety efforts are ignoring.

We publish this post as part of our series joining insights from AI and financial regulation (see here, here, here, here, and here). This post comes to us from Tim O’Reilly, a leading thinker and public intellectual on tech, the internet, and artificial intelligence.

As in Conan Doyle’s story “Silver Blaze,” in which Sherlock Holmes realizes that the essential clue is the dog that doesn’t bark – “the curious incident of the dog in the nighttime” – we have to ask ourselves what current AI safety efforts are ignoring.

There are four dogs that ought to be barking, but aren’t:

1. Why do we focus so exclusively on the risks inherent in model capabilities? That’s a bit like leaving auto safety solely to the NHTSA, with its crash test dummies, crumple zones, and so on, and ignoring all the other things that make autos safe, including driver training and licensing, requirements for auto insurance, rules of the road, speed limits, traffic lights, stop signs, and traffic circles, and an enforcement mechanism for those and other related laws, like those against driving under the influence. We need to think of AI safety systemically. What does that mean in practice? It means looking at AI as deployed, not just as tested in the lab. AI models are already integrated into everything, shaping recommendation algorithms and ad targeting, to the moderation decisions of LLMs themselves. So AI safety means building regulations that are informed both by the deployment infrastructure at scale, and the incentives and behavior of AI’s corporate owners, third party developers, and other participants in the AI economy.

2. Why do we assume that risks from humans putting AI towards bad aims only come from external bad actors? What are the economic incentives of the creators of AI models and services? When do they become the bad actors, either actively (e.g. by preying on their customers) or passively (by failing to enforce their policies for economic reasons.) The race for monopoly has led to a “move fast and break things” mindset, which didn’t work well for users in the social media era.

We also have to watch for abuses of the power to shape human knowledge and attention – what my colleagues Ilan Strauss, Mariana Mazzucato, and I call “algorithmic attention rents”. For example, right now, persuasiveness is considered an AI risk. But if advertising becomes a significant source of revenue for AI companies, you can easily imagine the narrative changing radically, with persuasiveness being dialed up to 11. How do we ensure that AIs aren’t tuned to be self-dealing in the interests of their owners?

3. What are the monitoring and response capabilities – an essential foundation of control – that are in place for AI? If we speed run lessons from the social media era and apply them to AI, we see that in many ways, both the Myanmar massacre and the Cambridge Analytica scandal were the result of Facebook deploying services at scale ahead of putting in place controls at the same scale. Given that it was Microsoft, not OpenAI, that noticed and reported the exfiltration of OpenAI’s prized data by Deepseek, it is clear that OpenAI’s own controls were insufficient, and that they are repeating Facebook’s mistake rather than learning from it.

Companies have significant economic incentives to minimize their investments in AI safety. Therefore, requiring disclosure of the total investment in AI safety, and a narrative about how and where safety tools are being deployed, and reports on how what is measured in daily operation (e.g. the number and nature of attempts to evade safeguards, and their success rate), would be more valuable than any disclosure of model capabilities.

4. What will we wish we had known ten years from now? In an emerging field like AI, it is too early to have strict regulations about what companies can and can’t do, but we should be requiring disclosures (or at least data collection and retention, much as airlines do with black boxes) that will help us to analyze what went wrong. What is the informational infrastructure that will make this industry “regulable” when the time comes? What should be disclosed? Far more than model capabilities! As I wrote when beginning my investigations of AI safety, there is one essential lesson from accounting standards that is worth starting with:

The systems of accounting that we take for granted today and use to hold companies accountable were originally developed by medieval merchants for their own use. They were not imposed from without, but were adopted because they allowed merchants to track and manage their own trading ventures. They are universally used by businesses today for the same reason. So, what better place to start with developing regulations for AI than with the management and control frameworks used by the companies that are developing and deploying advanced AI systems?

Once we identify best practices, the market itself can enforce them with minimal government intervention. By way of historical analogy, once securities law mandated registration and regular, standardized reporting for those companies that wished to issue securities to the public, banks came to require the same standardized reporting when issuing loans, investors when considering injecting capital, or when evaluating whether to buy a smaller company or a line of business from another. Standardized accounting principles then also shaped income tax reporting for businesses and individuals, and as Congress changed the tax code, tax reporting requirements in turn shaped GAAP.

Except in the case of tax auditing, compliance is delegated to an ecosystem of auditors and tax professionals who are given privileged access to a company’s finances in order to verify and attest that they are correct. But more importantly, those disclosures that are required to be made public are scrutinized by “the market” of investors, who use them to shape their decisions whether to invest or to short a stock. Government regulations that thoughtfully mandate disclosures that enable the scrutiny of the market will, I suspect, enable far more extensive oversight than any kind of centralized AI auditing agency. Gillian Hadfield and Jack Clark call for “regulatory markets” for AI as though they are something novel, rather than something that emerges naturally in an information-rich marketplace.

Accordingly, Ilan Strauss and I have started a project at the Social Science Research Council to explore these issues, plus a Substack called Asimov’s Addendum to report on our findings. We’re particularly interested in learning from AI professionals outside the explicit AI safety community, because that’s where we believe we will learn the most about how AI systems are actually measured and managed by the companies that build and deploy them. I hope you will help to inform our research and to follow its results.