AWS billing goes haywire, hits customers with bills as high as $7.8 trillion

midian182

Posts: 11,751   +178
Staff member
WTF?! Amazon has a reputation for squeezing customers for all they've got, but increasing billing prices from a couple of cents to $1.5 trillion really shows how Jeff Bezos got so rich. Thankfully, though, this was just an error by Amazon Web Services, but it still caused one customer's soul to leave their body, in their own words.

Bill Radjewski, who runs CollegeFootballData.com, received the sort of email most of us would rather not receive last week: an AWS alert informing him that he had accrued $1.5 billion in usage fees. Should he manage to win several lotteries and somehow pay it, he was still expected to face a $3 billion bill on August 1.

During the six years he'd been with AWS, his account had never exceeded $0.02 – most months his bill was just $0.01.

However, those figures were rookie numbers compared to what Dan Harvey, the head of marketing at the Hampshire-based Learning Through Landscapes, was billed.

Harvey told the Guardian that after his previous month's AWS bill came to 43 cents, he was especially surprised to find a bill for $7.8 billion. "I had to have a real dig around with our tech support team, while I was in full panic mode, trying to find what was going on in our account," he said.

There was also a student in Delhi whose regular monthly AWS bill of $1.28 increased to $10.9 billion.

Another user, Bharath, saw their bill rise an impressively large 745,728,201,771% from the previous month to $1.5 trillion. "I just saw $1.5tn on my AWS bill and my soul left my body," they wrote.

But the record seems to belong to the user who racked up a $7.1 trillion bill in service fees since July 1, which is more than double Amazon's market cap.

Amazon said that after 30 minutes of investigating, it determined that the root cause was "an issue with unit pricing within the estimated billing computation subsystem." The company provided no further details but paused the bill estimation system.

AWS later said it was "rolling back a recent change to the billing computation subsystem" and attempting to restore the last known good version of its estimated billing calculations.

It appears that the issue wasn't an easy fix. A few hours after that initial message, AWS wrote that "Our efforts to backfill corrected estimated cost and usage data are still underway. We are progressing slower than anticipated."

While most people obviously knew this was an error on Amazon's part, some have pointed out that this could really have alarmed customers and may have even caused health issues. "Good morning to everyone enjoying their heart attacks," wrote X user Gerred

Even Amazon tried to make light of the situation, though it's easy to imagine that not everyone was laughing about it . The company wrote on X, "Typo alert: Some customers saw quadrillion-dollar AWS billing estimates today. Slight miscalculation on our end (very slight ). We're fixing it now. No action needed on your end. Sorry for the confusion. Real question: what will you do with those trillions instead?"

Permalink to story:

 
My apps run all from a cheap dedicated machine with some old Intel CPU am 16 GB of RAM. I haven't updated it in 10 years and nobody hacked it. Though people certainly are trying judged by the access logs.

Anyway, I can't serve 10 trillion requests per second, but if my apps reached that popularity, I'll find a better solution then.

On the other hand the company I work for has some 50 people only for maintaining the Cloud platform and every week there's something broken. I understand that these Cloud services were made to remove the need to keep servers up to date and all that, so why do we still have armies of people to maintain them?
 
Methinks there was a price increase and someone missed a decimal point. It'll be interesting to see how much the bills AWS sends these people will be. Bet ya while not nearly as much they'll still be much more than a few pennies. The other AI shoe drops...
 
Obviously nobody was going to pay these bills, but receiving an automated demand for trillions still demonstrates why cloud cost controls matter. The system that creates the charge should not be the only system deciding whether the charge makes sense.
 
I hope nobody paid those trillion or billion dollar bills as those numbers seem very fishy and are most likely a mistake. Best thing to do would be to call amazon and get a detailed breakdown of the costs and read the contract to make sure they are correct charges before paying
 
Amazon: Hey! I got an idea! Lets send a bunch of customers bills in the trillions of dollars,
then when they complain, we'll drop it to a few thousands!

Another Amazon person: BRILLIANT!

LOL
 
May be it is not a mistake. Amazon and big techs need all the money they can get to fund their AI dreams. It is clear that all of them are running out of money funding each other.
 
Back