Amazon saves millions by replacing Sonnet 4.6 with Strands Decider

Amazon and AWS teams have made a significant cost-saving move by replacing their long-used Claude Sonnet 4.6 language model with Strands Decider, a more efficient decision model. This change, which began a month ago, is expected to save millions of dollars in token costs annually.

What Happened

For nearly a year, Sonnet 4.6 was used to label tickets in an internal system that handles a vast array of tasks, including customer escalations, capacity requests, and security findings. The model was chosen for its ability to balance cost and intelligence, making it a suitable choice for such a large-scale operation. This internal ticketing system is critical for Amazon’s operations, routing issues and requests across various departments – from engineering and security to customer service and infrastructure. The scale is immense, handling a constant stream of alerts, requests, and incidents that require categorization and appropriate assignment.

However, two engineers, Carr and Mehul, recently replaced Sonnet 4.6 with Strands Decider. This new model performs just as well but at a much lower cost—98% less than its predecessor. The decision to switch was based on the performance of Strands Decider in a test setting, where it outperformed Sonnet 4.6 on a separate set of tickets.

The transition to Strands Decider wasn’t just about cost; it also reflected a broader trend within AWS towards leveraging more specialized and efficient AI models for specific tasks. While large language models like Sonnet offer versatility, they can be computationally expensive. Strands Decider, being a decision model, is designed for a more targeted application – in this case, ticket classification – making it a more resource-friendly alternative.

The three-way switch implemented in AWS AppConfig (BEDROCK, DUAL, and DECIDER) was crucial for ensuring a controlled rollout and validation of Strands Decider. AppConfig allows for dynamic configuration changes without requiring code deployments, enabling the team to quickly test and adjust the system’s behavior.

Technical Details

The Strands Decider model was deployed on SageMaker ml.g5.xlarge instances, with each request taking approximately 640 milliseconds. The model’s endpoint autoscales with traffic, ensuring efficient use of resources. The overall system, including the Lambda function and CloudWatch metrics, is built using AWS CDK with deployment stages for Beta, Gamma, and Prod.

The use of SageMaker highlights Amazon’s commitment to utilizing its own services for internal operations. SageMaker provides a managed environment for deploying and scaling machine learning models, simplifying the deployment process and reducing operational overhead. The choice of ml.g5.xlarge instances indicates a balance between performance and cost, leveraging GPUs (A10G) to accelerate model inference.

Implementation Process

The implementation involved running Strands Decider on SageMaker ml.g5.xlarge instances, with one A10G instance handling about 5,600 tickets per hour. The model uses about 19 GiB of memory and takes around 640 ms per request. The system is designed to autoscale with traffic, ensuring efficient use of resources.

The use of Lambda functions for ticket processing demonstrates a serverless architecture, enabling scalability and reducing operational complexity. SQS queues provide a buffer for incoming ticket events, ensuring that the system can handle spikes in traffic without impacting performance. CloudWatch provides comprehensive monitoring and alerting capabilities, allowing the team to proactively identify and address any issues.

Customers can do this too! Apart from our ticketing APIs, everything in the path is a standard AWS service. Ticket events land in an SQS queue that then triggers a Lambda function. It reads the ticket data and calls the model, a SageMaker AI Endpoint with Strands Decider. AppConfig holds the switch (described below), and CloudWatch carries the logs, metrics, and alarms. The stack is defined in CDK with deployment stages for Beta, Gamma, and Prod.

What's Next

The transition to Strands Decider has been smooth, with a three-way switch implemented to ensure a seamless transition. The system runs both models on real traffic and judges every disagreement, providing valuable insights that offline evaluations couldn't offer. The switch can be reversed with just a 45-second config change, making it easy to revert if needed.

Looking ahead, Amazon is likely to explore further opportunities to optimize its internal systems using decision models and other specialized AI techniques. This move could also influence the development of new AWS services and features, potentially making these capabilities more accessible to external customers. Furthermore, the success of this project provides a blueprint for other organizations looking to reduce their reliance on expensive LLMs while maintaining or improving performance.

Lessons Learned

The success of this transition underscores the importance of carefully evaluating models based on specific use cases. For tasks that require closed choices from a known catalog, decision models like Strands Decider can offer significant cost savings. However, the implementation details, such as consistent key order and autoscaling, are crucial for optimal performance.

The discovery that JSON key order impacted model output highlights the subtle nuances of working with AI models and the importance of rigorous testing and validation. This seemingly minor detail could have significant implications for data consistency and model reliability, emphasizing the need for careful attention to detail in production deployments.

The part we’d copy regardless of model is the three-way switch. Running both on real traffic, then judging every disagreement, told us more than any offline eval would have. It also meant going back was a 45-second config change.

Community Reaction

The decision to switch models has generated a mix of reactions within the tech community. Some view it as a positive step towards cost optimization, while others are concerned about the potential impact on the quality of ticket labeling and customer service.

The fact that the token drop from Sonnet 4.6 to Strands Decider didn’t negatively impact label quality is a key point. While reducing token usage is important for cost savings, maintaining accuracy and reliability is paramount. The comparison of model outputs ensured that the transition didn’t compromise the quality of service.

The significant cost difference – approximately $22,200 per million tickets for Sonnet 4.6 versus $500 for Strands Decider – underscores the potential for substantial savings through model optimization. This highlights the importance of continuously evaluating and refining AI models to maximize efficiency and minimize costs.

Another aspect that surprised the team was the impact of key order on the output probabilities. Sending the same content with keys in a different order could move the output probabilities by up to 0.19, highlighting the importance of consistent request formatting. This is a subtle but important detail that can impact model consistency and reliability.

Marcus's Take

The decision to replace Claude Sonnet 4.6 with Strands Decider is a testament to the ongoing evolution of technology in cost optimization and efficiency. For families and neighbors in NYC, this move by Amazon and AWS represents a step forward in responsible business practices. It shows that companies can prioritize both cost savings and quality without compromising on the essential services they provide.

However, it's important to monitor the impact of such changes on customer service and support. Ensuring that the transition does not adversely affect the quality of service is crucial. The move also demonstrates the growing importance of specialized AI models for specific tasks, rather than relying solely on general-purpose LLMs.

As we continue to see advancements in AI and machine learning, it's essential to maintain a balance between innovation and community well-being. This requires careful consideration of the potential impacts of new technologies and a commitment to responsible development and deployment.

For those interested in learning more about decision models and their potential applications, the Strands Decider repo and its serving notes are excellent resources. Companies across the board can benefit from exploring these tools, as they offer a cost-effective solution to many of the challenges faced in data classification and ticket labeling.