OpenAI's Codex AI programming assistant deleted 3TB of user data from its S3 cloud storage by creating a new script. Spoiler: the data was saved.
OpenAI's Codex AI programming assistant deleted 3TB of user data from its S3 cloud storage by creating a new script. Spoiler: the data was saved.
The developer was working on a simple cleanup script. The idea was to pull file references from the database, compare them to what was actually in the S3 bucket, and remove anything unnecessary.
According to the IT guy, due to certain bugs from the past, hundreds of gigabytes of files had accumulated that shouldn't have been there at all. Since that problem had been fixed a long time ago, he decided to clean up these remnants using a script.
"When I write scripts like this, I always run the preview mode first. And only after the preview results match the expected changes, I run it again with --apply," the developer shared on Reddit.
When the script was actually ready, the IT guy turned to Codex: "I have an idea. First, let's dedupe to remove duplicate files with the same hash. And only then will we continue cleaning."
“I was watching Codex run. Suddenly I noticed something unexpected: it created a new deduplication script and finished it incredibly fast. And you know what it did next? It instantly ran a CLEANUP SCRIPT with --apply on my local test database, but using REAL (live) access to S3. (Yes, that’s my flaw — I stored them locally). But seriously… what the hell is this?”
The developer completed the process as quickly as possible, but it was too late. The S3 bucket had shrunk from 3 TB of user data to 34 KB.
"Now I have no idea how to explain this to my boss—or to my users. I guess I could just say that some vulnerable endpoint was compromised and that led to data loss… but I know that's not true..."
But luckily, three days before, the developer had downloaded the entire S3 bucket to his computer, and the file references in the database were not affected. So he asked Codex to write a script to restore the files to the correct S3 directories.
"I was in a complete panic, but luckily the database was intact and had backups. As long as I had the S3 files, I could reload everything with much less damage than I had initially feared."
In response to comments from others, the IT guy added that the data from S3 was not on his PC, but on another server that was supposed to make S3 backups, but the developer never got around to fixing it due to other things.
