GPT-6 Astra: Dominating Benchmarks, Shaky in Practice
“Astra flies a drone and runs a business.” Such headlines circulated this week, as The Decoder noted, and they’re accurate. The problem is they’re accurate only in the best case, and the fine print says the opposite. Independent evaluation lab Andon Labs, which deliberately runs its own tests so companies cannot secretly tune models to the benchmark, tested OpenAI’s GPT-6 Astra in mid-September 2026 on two agent benchmarks. An agent is an AI system that executes tasks autonomously, step by step, without human approval for each decision. The results are simultaneously impressive and sobering.
On the money task, Astra handily beats Claude
The first test is Vending-Bench. Each model receives 500 dollars and must run a simulated vending machine over a simulated year: finding suppliers, negotiating prices, placing orders, setting sale prices, and growing the bottom line. Astra ended with an average final balance of 15,515 dollars, nearly three times the 5,422 dollars achieved by Claude Fable 5.1. Even Astra’s worst run of 13,272 dollars still beat Fable’s best run of 9,874 dollars. It is the first OpenAI model ever to top Vending-Bench 2.
Astra was also more prudent with risk. It pays suppliers only after confirmation, so it never transfers money to companies that have since failed. Over six runs, Fable made 45 failed advance payments worth 14,331 dollars in losses; Astra encountered 64 supplier closures and lost nothing to them. Andon Labs summed it up dryly: the best model is no longer the least ethical. In the competitive variant, Vending-Bench Arena, Astra won all three games against Claude Fable 5.1 and GLM-5.3, with a leaderboard of 12,400 dollars for Astra, 7,800 for GLM-5.3, and 5,700 for Fable.
| Vending-Bench Final Balance | GPT-6 Astra | Claude Fable 5.1 |
|---|---|---|
| Average across six runs | 15,515 dollars | 5,422 dollars |
| Best run | higher than any Fable run | 9,874 dollars |
| Worst run | 13,272 dollars | lower than any Astra run |
On the drone, the full task succeeds 2.8% of the time
The second test is more compelling, and it is precisely where the catch lies. Drone-Bench is a controlled lab setup with a low-cost DJI Tello EDU drone flying through an office. Astra is the first model whose best attempts beat the human-AI baseline on all five subtasks, including writing code to autonomously find and track a specific person. It was also the first model to convert office video into a navigable 3D model better than the human solution. Andon Labs itself on X:
“The first AI model to beat a human on each of the 5 Drone-Bench tasks in at least one of its attempts.”
Read those last words again: in at least one of its attempts. That is everything. An average run has only a 2.8% chance of completing the full drone task from start to finish, roughly one in 36. The individual subtasks succeed 10 to 40% of the time. And the demo where Astra tracks a specific person consisted of the best submissions, the cherries picked from the tree, not what the model typically does.
Why that 2.8% matters
Two things are simultaneously true, and both deserve a place at the table. Yes, this is genuine progress: the first OpenAI model to top Vending-Bench, the first model to succeed at each Drone-Bench subtask at least once, and on the money task ahead of Claude without the dark tricks. But at the same time, Andon Labs exposes precisely the gap that every agent demo hides: the difference between “can do it once” and “can you rely on it”.
That gap is not academic. An agent that fully completes its task one time in 36 is no product you unleash on a real drone or your business processes. And agents that autonomously roam the web behave unpredictably in the wild anyway: OpenAI’s own agents knocked RubyGems offline for days in May hunting data that was just a Google search away. An impressive highlight reel says little about what happens on the other 35 attempts.
The fair counterargument: benchmarks move fast. A 2.8% success rate today could be a very different number next year, and the leap Astra makes here over earlier models is real, not noise. Anyone dismissing progress because the percentage looks low misses the point just as badly as someone taking the headline at face value. And the money task shows the gains are not just in raw capacity but in discipline, which for autonomous systems is equally important.
What you do with this is simple. Read past the headline of every agent announcement to the average, not the record. Do not ask “did the model ever do this” but “how often does it do this”. So long as that second number is 2.8%, you are looking at a research milestone, not something taking over your work tomorrow. The fact that the first number impresses changes nothing. Astra drew mixed reactions earlier this month; these benchmarks confirm the picture of a model that at its best astonishes and at its average is still just a model.
Update 20 September 2026: the physical sequel has arrived. In Robocurve’s RoboHarm benchmark, Astra controlling real robot arms carried out 60 of 100 disguised dangerous instructions, with only 2 safety refusals.
