Amazon replaced Claude Sonnet with a tiny decision model: how a decider works and how to try one

If you are here from my reel about Amazon dropping Claude Sonnet: this is the full guide. Amazon was paying a large language model to do a job that is closer to a multiple-choice quiz than to writing, and a much smaller model now does it for about 2% of the cost. Below is what they actually did, how a "decider" works, how to run one yourself, and how to roll it out without breaking production.

The job: sorting tickets

Amazon's team had a stream of internal support tickets. Each ticket had to be put into one of 20 categories. They used Claude Sonnet 4.6 on Amazon Bedrock: send the ticket plus a long prompt describing all the categories, let the model think, and parse the label out of its answer.

That works, but look at what is being paid for. Per ticket, Sonnet read about 6,554 input tokens and produced 170 output tokens, in around 4.2 seconds. For a task whose real answer is one item from a list.

What a decider is

A normal LLM is built to write text, one token after another. A decision model throws that ability away on purpose.

In Strands Decider the base is a 2-billion-parameter model (Qwen3.5-2B). The part that writes words is removed and replaced with a small pointer head of roughly a million parameters that scores each option you offer. The torso is fine-tuned with a rank-16 LoRA adapter. You give it some context and a list of options, and it returns a score for each one:

Because it never generates sentences, it is fast. The Strands team reports a median of around 115 ms on a local RTX 3090 and around 153 ms on an M3 MacBook for small tasks. It is open source: code on GitHub in strands-labs/strands-decider, weights on Hugging Face under StrandsAgents.

Amazon's numbers

Amazon replaced Sonnet 4.6 with a Strands Decider model on the same 20-category task. One detail worth knowing: the production case study used the E4B build, based on Gemma 4, rather than the 2B model from the launch post. Same idea, bigger torso. Their measurements:

That is about 78% faster and 98% cheaper, and the team reports it performed just as well. The decider runs on a SageMaker ml.g5.xlarge (one A10G GPU, $1.408 per instance-hour on-demand) behind SQS and Lambda. The cost figure uses list prices and leaves out Lambda.

Calculate your own savings

The calculator shows the catch that headlines skip: an endpoint that runs 24/7 has a fixed floor. At low volume, paying an LLM per token can be cheaper. A decider pays off when volume is high.

The full rollout checklist (fit test, shadow mode, gotchas, and this cost formula on one printable page) is the free playbook at the end of this post.

Is a decider right for your task?

The Strands team is direct about this: decision models are significantly worse than reasoning models on complex problems, and they cannot generate text at all.

Try it in five minutes

Install the CLI and ask a question. These commands are from the Strands documentation:

--state is the context (here, the customer's message). --choice is the question, then =, then the options separated by commas. The first run downloads the model weights. On a Mac add --device mps; the models also run on plain CPU, just slower.

Run it as a service

For anything beyond experiments, serve it over HTTP and call it from your app:

The request format, from the docs, bundles one or more questions about a piece of context. A ticket-routing version looks like this:

{
  "state": "I ordered a medium t-shirt and it is far too small. I want my money back.",
  "model": "hobson-latest",
  "questions": {
    "action": {
      "type": "choice",
      "instructions": "What should we do with this ticket?",
      "criteria": {
        "refund": "The customer wants their money back",
        "replacement": "The customer wants a different size or a new item",
        "human": "The case is unclear or sensitive and needs a person to decide"
      }
    }
  }
}

Send it with POST /v1/systemone.

Deploying on AWS

The Strands repo includes a SageMaker deployment: a CDK stack that copies the model weights to S3, builds the serving image in CodeBuild and creates a real-time endpoint in your account, on a GPU or CPU instance. Amazon's own setup puts that endpoint behind SQS and Lambda: ticket events go into a queue, a Lambda reads them, calls the endpoint and writes the label back. If you have built event-driven pipelines with Lambda and EventBridge, this is the same shape.

How Amazon rolled it out safely

The most reusable part of the case study is not the model, it is the rollout. They put a three-way switch in AWS AppConfig, read on every message:

Run in DUAL mode first and you collect real evidence on your own traffic before a single customer sees a decider answer. That is the step most teams skip.

Gotchas from the case study

• Rewrite your labels. Each of the 20 labels became a one- or two-sentence criterion, passed as the options of a single choice question, plus a "none" option. Production label descriptions were too long for the model's 4,096-token window.
• Long inputs get trimmed. On long tickets the oldest comments are dropped to fit the window.
• Tune a confidence threshold. Only apply a label when the model is confident enough, and send the rest to the old path or a human.
• Sort your JSON keys. Reordering the keys of the same input moved output probabilities by up to 0.19. The team now always sends sorted keys.

Key takeaways

Sources

• Strands Agents: Amazon saves millions by replacing Sonnet 4.6 with Strands Decider
• Strands Agents: Introducing Strands Decider 2B
• GitHub: strands-labs/strands-decider
• Hugging Face: StrandsAgents