Amazon tightens code guardrails after outages rock retail business
Amazon’s e-commerce platform has faced significant challenges recently, marked by a series of outages that have disrupted operations and led to substantial losses. These incidents have prompted the company to implement stricter controls on its coding practices, particularly in light of issues linked to its AI coding assistant, known as Q.
The Recent Outages
In the past few weeks, Amazon has experienced a trend of incidents that have raised concerns about the reliability of its systems. According to Dave Treadwell, Amazon’s Senior Vice President of e-commerce services, several major outages occurred, some of which were directly tied to the use of the AI coding tool Q. Internal documents suggest that these problems were exacerbated by a lack of adequate safeguards in the company’s control planes, which are essential for managing data flow across its networks.
Understanding the Impact
One of the most notable outages occurred on March 2, when customers reported incorrect delivery times while adding items to their carts. This incident resulted in nearly 120,000 lost orders and approximately 1.6 million errors on the website. The internal review indicated that the AI tool Q played a significant role in triggering this event, highlighting the potential risks associated with the rapid deployment of AI technologies in critical operational areas.
Another significant outage on March 5 led to a staggering 99% drop in orders across Amazon’s North American marketplaces, resulting in a loss of 6.3 million orders. This disruption was primarily attributed to a production change that was made without following the formal documentation and approval processes established by the company.
New Code Controls and Safety Measures
In response to these challenges, Amazon is implementing tighter code controls aimed at enhancing the reliability of its systems. Treadwell announced a 90-day reset period during which engineers will be required to document their code changes more thoroughly and secure additional approvals before deployment. This initiative is designed to introduce “controlled friction” into the code-change review process, ensuring that significant changes undergo rigorous scrutiny.
Key Features of the New Policy
- Engineers must obtain two approvals for any code changes.
- All changes must be documented using an internal tool that adheres to Amazon’s reliability engineering rules.
- Audits of production code change activities will be conducted by Tier-1 system owners and their leadership teams.
The Role of AI in Software Development
The recent outages underscore the complexities introduced by generative AI in the software development process. While AI coding services, such as Amazon’s Q and Kiro, can significantly increase the volume of code produced, they also necessitate careful oversight to prevent errors and ensure quality. Treadwell emphasized the need for a combination of AI-driven tools and deterministic systems to address the inherent unpredictability of AI outputs.
Agentic vs. Deterministic Systems
Amazon’s new approach aims to balance the use of “agentic” AI tools, which can produce varied outputs, with more predictable, rules-based “deterministic” systems. This dual approach is crucial for maintaining the integrity of core systems that require consistent accuracy, such as those managing product listings, pricing, and transactions on the e-commerce platform.
Looking Ahead
The implementation of these new safety measures reflects Amazon’s commitment to improving its operational reliability and preventing future outages. By reinforcing its coding practices and introducing additional layers of oversight, the company aims to mitigate risks associated with rapid technological advancements and ensure a seamless shopping experience for its customers.
Frequently Asked Questions
The recent outages at Amazon were primarily caused by issues related to the use of its AI coding assistant, Q, and a lack of adequate safeguards in its control planes, which manage data flow across the network.
Amazon is introducing tighter code controls that require engineers to document code changes more thoroughly, secure additional approvals, and undergo audits of production code change activities to enhance operational reliability.
Generative AI can significantly increase the volume of code produced but also introduces challenges related to unpredictability and quality assurance, necessitating careful oversight and a balance between AI-driven and deterministic systems.
Note: The information provided in this article is based on internal documents and statements from Amazon officials and is subject to change as the company continues to refine its operational strategies.
