The Microsoft Azure DevOps service in South Brazil experienced a problem for about ten hours due to a simple code mistake. Eric Mattingly, a manager at Microsoft, apologized for the issue on Friday.
A few days later, some software changes were made to a group of computers in the South Brazil unit that perform specific tasks. However, the computers had an outdated snapshot database, which caused the error. This mistake led to a background task accidentally deleting the entire Azure SQL Server and all seventeen production databases.
Mattingly stated that there were several reasons for this problem. First, the customer was unable to fix the Azure SQL Server themselves, so they required assistance from a Microsoft engineer. It took approximately an hour to receive this assistance. Second, the databases had different backup settings, with some being regionally redundant backups and others being geo-redundant backups. This mismatch resulted in a longer time to fix everything.
To prevent this problem from reoccurring, Mattingly mentioned that Microsoft has made some software changes. He also apologized once again to everyone who experienced difficulties due to this problem.