| | | | ‘Big tech keeps telling us that AI agents will take over jobs, and for boardrooms chasing shareholder returns, fewer humans sounds like a great deal. That's why we're seeing customer service agents rolled out everywhere, and why cash-strapped government departments are now looking to automate decisions too. But before we hand over the keys, it's worth remembering that we've already tried letting the machine decide, and the results were catastrophic. In this video I tell the story of Robodebt, the Australian government's automated debt recovery scheme that ran from 2015 to 2019. A simple algorithm averaged annual tax income across fortnights, compared it to welfare records, and issued hundreds of thousands of legally enforceable debts to some of the most vulnerable people in the country, with no public servant ever reviewing the evidence. The burden of proof fell on the citizen, lives were lost, and it took a Royal Commission, a $2.4 billion settlement covering more than 400,000 people, and referrals to the National Anti-Corruption Commission to unwind it. The commissioner called it "venality, incompetence and cowardice." Robodebt happened before ChatGPT, but the same logic is now playing out inside corporations: remove human discretion, hide behind the algorithm, and make sure nobody's name is on the decision. As someone who builds AI systems, I argue that we should absolutely chase efficiency, but never by removing accountability. AI should mean Augmented Intelligence: humans making better decisions faster, with a person in charge and a name against every outcome. That's the only version of the future worth building.’ |