Toyota's "insufficient disk space" blunder brings car manufacturing plants to a halt

Alfonso Maruccia

Posts: 1,025   +302
Staff
Why it matters: Recurring security incidents have highlighted the poor state of Toyota's IT operations. Even when an external threat is missing, the Japanese giant is working in self-sabotage mode with potentially significant consequences on its finances and daily operations.

Toyota suffered a recent "production order system malfunction" in its domestic manufacturing plants, an issue that forced the company to put everything offline for a few days. Unlike past incidents, however, this time there was no security attack or unwanted data exposure. Toyota was simply relying on very bad IT practices and was forced to sacrifice a good chunk of its growing domestic output as a consequence.

Apologizing to customers, suppliers and other interested parties, Toyota has now confirmed that the system malfunction happened because "some multiple servers" that process part orders suddenly became unavailable. Routine maintenance work was performed on those servers on August 27, a day before the malfunction event. Toyota says that while the maintenance procedure was deleting and organizing some data that had accumulated in the database, an "insufficient disk space" error occurred.

The system suddenly crashed, and the error was replicated in the "backup function" because both servers were running on the same system. The double error forced Toyota to halt production at all of its 14 manufacturing plants in Japan, the BBC reports. It was just a one-day disruption, but Toyota experienced an output loss of around 13,000 cars.

The part ordering system and the crashed databases were seemingly restored on August 29, after IT managers transferred the data to a server with larger storage capacities. Manufacturing operations were resumed the following day, and countermeasures are now in place to avoid any new issues with database operations. Toyota was eager to reassure the public that the system malfunction wasn't caused by a cyberattack.

According to internal sources quoted by Reuters, Toyota's database blunder will be financially expensive as domestic manufacturing output increased by 29% in the first half of the year. Toyota business was growing for the first time in two years, but the company was seemingly unable (or unwilling) to properly manage its database backups on secondary servers away from production environments.

Going forward, the Japanese carmaker is assuring a complete review of its "maintenance procedures" to prevent a database error like the one experienced this August. The company wants to deliver "as many vehicles as possible" to its customers, but it also should strengthen its IT and security practices a bit. In February 2022, Toyota's Japanese factories were shut down because of a supply chain attack. Hackers compromised the company in 2021 through a US subsidiary, and they did the same at least three more times in 2019.

Permalink to story.

 
So Toyota's IT department is no better than the majority of IT departments. Is this really news?

At this point, however, I bet Toyota will take action to correct the situation.
 
"insufficient disk space" I have noticed that since their last CEO retired there have been more and more errors and quality issues. Apparently the new CEO isn't aware of the long standing reputation Toyota has for quality and consistency in their product lines. They need to take a serious look at their senior management team and bring in or bring back some of the old reliable managers to re-educate current management or risk falling off the car buyers radar .....
 
Back