Evaluating the Ban Artificial Superintelligence Act

MIRI recently endorsed the Ban Artificial Superintelligence Act, but others like ControlAI have called it overly broad, and a few other respected experts have outright endorsed it in its current form. I decided to take a look at it myself and see both what the bid is trying to do and whether it actually does it well. The goal of this act is to ban any AI that shows traits that indicate it could cause great harm to society, and to restrict the development of advanced AI to certified institutions who can be trusted to do so safely. This seems like a reasonable goal. The problem with this bill, as far as I can see it, isn't that its goal is wrong, but that the way it's been drafted contains lots of flaws, which are likely to have unintended negative consequences. At it's core, the bill does four things: * Ban AI that has one of six dangerous capabilities * Automate or greatly accelerate AI research and development * Access secured systems without authorization * Keep operating despite attempts to shut it down * Meaningfully help with nuclear, chemical, or biological weapons * Modify its own functions * Scheme, deceive, or avoid human oversight * Only allow certified orgs to train “Advanced AI Systems” - An advanced AI system is defined as any model that has been trained with greater than 10^25 operations. Only certified orgs are allowed to develop such models. As such, they are the only people allowed to posses an advanced AI system that has not yet been certified as safe. * Establish a government department tasked with certifying that an AI system is safe - No Advanced AI System can be distributed unless it has received certification from this department. * Impose strict penalties for people who recklessly distribute dangerous AI - Up to 20 years in prison and the forfeiture of all assets of a business. This basic framework seems broadly sensible, although there are legitimate questions about the extent to which we want to ban the automatio