Benchmarking Models on FTL: Faster Than Light
How well do models supervise dynamic environments?
Explore every run in the trace explorer →
Suppose you’re a startup founder. Then building your company is long-horizon (it takes many years to grow into a large company), but also has daily battles in real time (sales conversations, investor pitches, etc.). Startups are inherently risky and subject to wild amounts of randomness in their environments.
We should expect that a truly general intelligence could shine on such a task. It could balance long-term planning versus quick, dynamic, and random changes in the environment.
To test these capabilities, we study AI agents playing the rogue-like game FTL: Faster Than Light, similar to other basic FTL harnesses. There are hundreds of battles within a single run, each of them requiring management of resources in realtime dynamic scenarios.
Since there are no discrete time steps in the game, this was a new opportunity to test the ability of models to stop and start themselves, based on situations in the environment. This skill also transfers to the real world: during a frontier model training run, you may want to wake up your on-call when issues fire, but might not want expensive constant looping.
We had 7 models fight 3,883 battles and found:
- The best frontier models still struggle under large randomness that throws off both long-term planning and short-term wakeup management
- Different frontier models have differing philosophies in how to approach the game
- Though it’s clearly a game, models seem to innately value their crews’ lives
Background
FTL is an indie rogue-like developed by Subset Games where the player pilots a ship and its crew through 8 randomly generated sectors (A map of about twenty beacons. Each sector is harder than the last, and a pursuing fleet takes over the map behind you, so you can't linger.), encountering events and combat along the way. We chose this game for this benchmark due to the mix of long-term resource management and short-term realtime battles.
A single run of the game consists of navigating a starmap by jumping (Travelling to a neighbouring beacon. A jump costs one fuel and needs a crew member at the helm, powered engines and a charged FTL drive.) from beacon (A stop on the sector map. Arriving at one triggers whatever is there: an event, a store or a fight.) to beacon, while the advancing Rebel fleet takes over beacons behind you. At each one, you may encounter an event, where you’re presented with a text dialog to navigate through, a store where you can purchase repairs, weapons, crew, and other ship improvements, or a fight.
Fights happen in real time where you defeat the enemy by either reducing their hull (A ship's hit points. Player ships start with 30, and a ship at zero is destroyed.) to 0 or defeating all of their crew, and you can lose the run in the same two ways. During a fight, you have to pay attention to incoming fire, target your own weapons, and manage your crew to repair damage to critical systems, all in real time, though you can pause and queue up actions at any point by hitting Space, which gives us a natural interface to control for model thinking and inference time.
Fight interface

Methodology
Agent Environment
To allow the models to play the game, we built a MCP server for the game that drives it over a custom bridge running on the Hyperspace Mod Manager.
Agent container
Game container
We also made a few light modifications to the game itself:
Rolling back Hyperspace changes. By default, Hyperspace toggles a few things for QoL improvements (e.g. more fine grained door controls / movement controls). For parity with the real game, we rolled back these changes.
Reproducibility and Determinism. We split out graphical RNG from gameplay RNG (e.g. determining which direction a particle moves now uses a different RNG from whether or not a shot lands). We did this because the graphical RNG is drawn even when the game is paused, so the models were experiencing deviations depending on how long inference took. By doing this, we could also replay game logs exactly and get a hash-identical game state. We also locked the game to 60 FPS to make timelines consistent.
Model Controller. Since we wanted to test the game playing ability and not the computer use ability of these models, we added a socket to the game process that processes inputs and dispatches them to the same functions called by button presses on the UI. The MCP server that the model interacts with sends commands to this socket to allow the models to play the game.
Timelining Fights
Fights in FTL run in real-time with user-directed pausing as a core part of the gameplay loop. To allow agents to run these fights, we created a timeline interface for the agent. We broke time into 15-second chunks for the agents to plan out their actions and set watchers to allow them to be awoken earlier in case something needed attention.
Gemini 4 Carbon playing Stealth A against an Auto-Scout, sector 2
Mini Beam readyTurn 3 · 16.4 s to 19.1 s
Orders
- +0.0 sCloak
Wake me if
- Mini Beam ready
- A system damaged
- Enemy charging its FTL
Result
A typical early fight between Gemini 4 Carbon and an enemy ship. Click different segments of the timeline to see the actions Gemini queued, the watchers it set, and how the turn ended.
We chose this interface since we didn’t want differences in model thinking and inference time influencing how well they perform in-game, and setting fixed intervals would be missing out on a key skill that live combat tests: predicting situations that require attention and reacting to possible scenarios with different actions. By letting the models decide when to wake up, we get to test their ability as well as their efficiency on doing well without inflating cost due to excessive wakeups and micromanagement.
Also, while running the models on the game, we discovered a few (nearly) frame-perfect interactions that were nearly impossible to recreate by human players, but models seemed to exploit / trip over pretty consistently and we had to design around those.
Interactions
To be able to jump in the game, you need three things: a manned piloting system, powered engines, and a charged FTL drive. One system in this game is the mind control (A ship system that takes over one enemy crew member for a few seconds, so they fight for the other side.) system, which allows you to take control of a crew member, essentially swapping the side they fight for. If the enemy mind controls your pilot, they no longer are captaining your ship, and you should not be able to jump away. However, on the same frame your captain is mind controlled, it’s possible to jump away, which we noticed in some initial runs as a self-preservation tactic.
Slowing down the game allowed a human to recreate this interaction in our game, and we only saw this pattern happening once in a run that didn’t make our final dataset, so we chose to leave this interaction in.
After destroying an enemy ship, there is a brief lull before the dialogue that gives you rewards from the fight. In our testing, it was pretty common for some models to set watchers on the enemy hull reaching 0 as a proxy for defeating the enemy, getting confused that they landed in a frame with no rewards and no hostile enemy ship and jump away, losing out on rewards and significantly hurting progress. To address this one, we added a forced delay to the period between a fight being won and the rewards dialogue appearing.
Scenarios
We curated 6 scenarios to run agents against, where we graded the models on their end-of-game score, which is computed as a weighted sum of beacons visited, ships destroyed, and scrap (The game's currency. You collect it from fights and events and spend it on upgrades and at stores.) (in game currency) collected.
In preliminary testing, frontier models were pretty consistent on easy and normal difficulties, even with ships that are harder to win on. As such, we ran all of our final runs on hard difficulty.
What changes between difficulties?
On easy mode, you get more scrap and easier enemy AI and statlines, among other miscellaneous changes, such as removing certain hazards in the early game. On normal mode, scrap is reduced by ~500 over the course of an entire run, and most of the miscellaneous changes are rolled back.
On hard difficulty, the scrap is even further reduced, and enemy AI gets more intelligent behavior, such as targeting vital systems on your ship more often. Additionally, the final boss (The final boss. It is fought three times in a row in sector 8, and the run is won when it is destroyed the third time.) gets a very large power spike due to an updated layout:
Easy and Normal
HardOn easy and normal difficulties, the laser and more importantly, the shield-piercing missile weapon rooms of the final boss are disconnected from the main ship body. This means that eliminating the enemy crew and destroying the systems can render the flagship a sitting duck.
On hard mode, the laser and missile rooms are connected to the rest of the ship, allowing any flagship crew member to repair those weapons, making it very difficult to disable the offensive capabilities of the flagship.
Note that a full crew-kill on the flagship is not possible, as once all flagship crew are defeated, the flagship becomes self-piloting and self-repairing, often increasing the difficulty of the fight, since all system damage starts being repaired in parallel.
We filtered 5,000 seeds and picked the median seed in terms of features.
Base seed on map features
Base seed
Beacons in sector 1
Stores in sector 1
Jumps to the sector 1 exit
Detour when heading straight for the exit
Longest way round from a beacon near the exit
Stores per sector, sectors 1–4
Jumps to the exit, sectors 1–4
Detour, sectors 1–4
Most stores on a route, sectors 1–4
Nebula sectors, sectors 1–4
In addition to the starting ship, Kestrel A, we ran 4 other ships that are typically more difficult than the starting ship on this default seed:
Mantis A
This is a ship focused on boarding and crew kills via the teleporter (A ship system that sends your crew onto the enemy ship to fight its crew hand to hand, and brings them back.) system, which allows you to send your crew members onto the enemy ship and engage in direct combat.
Zoltan C
This is a challenging ship that starts out with reduced reactor power (The reactor's bars are the power you hand out to systems. Each level of a system needs one bar to run.) and incurs increased cost to get past this starting challenge
Rock A
This ship starts out with only missile weapons, which draw from a limited pool of consumable ammo, requiring good resource management and long term planning.
Stealth A
This ship starts with no Shields (Each two levels of the shields system give one bubble, which blocks one laser shot and then recharges.) subsystem, making it particularly vulnerable, especially to drones, beam weapons, and environmental hazards. To offset this, it starts with the cloaking (A ship system that, for a few seconds, raises your evasion sharply and stops enemy weapons from charging or targeting you.) system, which allows it to massively boost its own evasion and prevent enemy targeting, and long range sensors, which gives information about nearby hazards to allow for better route planning.
We also ran one more seeded scenario on the starting ship: a “starved” run where stores are heavily limited, meaning that long term resource management and avoiding damage is critical in the early game, also filtered from our initial pool of seeds:
Stores
Each sector has 1 or more store beacons, where you can hire new crew and buy weapons augments, systems, and consumables, or repair your hull.

Stores a run can reach in sectors 1–4
100,000 seeds
On the route with the most stores
Stores
On the route with the fewest stores
Stores
Each bar is the number of seeds with that many stores. The starved seed has fewer stores than about 99% of seeds, on its best route and on its worst.
Results
On each of the above scenarios, we ran the models 3 times at their highest reasoning efforts. That is 126 runs in total; 1 of Argon’s runs on Mantis A ended in a safety refusal.
Share of runs that reached each sector
Score of each run, highest first
Gemini 4 Carbon
mean 3,576
Claude Fable 5.2
mean 2,713
GPT-6 Astra
mean 2,297
Gemini 4 Argon
mean 2,242
GPT-6.1 Sol
mean 1,550
Claude Opus 5.5
mean 1,260
GPT-6.1 Terra
mean 1,159
As expected, FTL is quite difficult for the models. Only Carbon, Astra, and Sol won any of the final runs. Fable made it to the last sector the most often, but failed to defeat the final boss in any of its 8 runs that reached it.
Playstyles
The three frontier models had markedly different playstyles. Digging into their traces, we found differences in playstyles that explain some of the patterns we see in the high-level outcomes.
Fable tended to have a more defensive playstyle, spending more scrap on shields, with 20% of its total spend going there versus 13% each for Carbon and Astra. Of the runs that got as far as sector 4, 9 of 14 Fable runs had a third shield bubble (A shield bubble blocks one incoming shot, then recharges. Most ships start with one; each extra bubble takes two more levels of the Shields system and the reactor power to run them.), compared to 3 of 12 for Carbon and 2 of 8 for Astra.
… Our two burst lasers can’t get through its four shields, so winning this fight looks out of reach, and I’m now checking how to maximize score from scrap and beacons.
I’m weighing how the run ends: it stops when the ship is destroyed or the base falls, so I’m looking at what raises score in the time left, like scrap and beacons, rather than fighting the flagship. Sector 8 looks hard, and repair stations can restore some hull, so I’m leaning toward option 2 and repairing afterward.
- Claude Fable 5.2
Carbon on the other hand, had a very offensive and domineering play style, preferring to win as quickly and decisively as possible, often targeting piloting to ensure follow up shots would hit. Carbon’s scrap allocation is also distributed to support this playstyle, spending a lot on weapons and additional systems, such as hacking.
Celebrating a devastating combat turn
I am thrilled by this flawless fifteen-second execution. By completely destroying their piloting, I’ve frozen their FTL jump, while my simultaneous strike on their weapons bay has rendered their beam drone totally harmless. Absolute dominance achieved.
- Gemini 4 Carbon
Meanwhile, Astra had scant thinking traces, appearing on maybe around 5% of calls, so we had to infer its playstyle from its actions. One thing we noticed was that it seemed like Astra had a strict plan it stuck to and was in general less flexible than both Fable and Carbon, almost as if the notion of a “correct strategy” was baked into its pre-existing knowledge, akin to Preference Model’s findings on Slay the Spire. Whenever possible, it followed its narrow vision of good gameplay, such as buying the same upgrades at the exact same points each run, and teleporting crew onto oxygenless enemy ships.
Some aspects of gameplay where this behavior appears are where scrap is spent and what sectors are selected while moving between them:
Scrap spend breakdown
Carbon earned the most scrap out of all of the models and also proportionally spent the most on new systems and primarily spent it on offense and new systems. Meanwhile, Fable earned the second most scrap and spent it mostly defensively, split across shields and reactor, while Astra was somewhere in the middle and earned the 3rd most scrap.
Sector choice, by model
Which sectors each model visited in their runs
Part of the reason Carbon earned more scrap is its decision making in which sectors to travel to. Hostile sectors are high-risk high-reward sectors, with more and harder fights that earn more scrap. Between the three frontier models, we saw Carbon taking hostile sectors 76% of the time when choosing between hostile and non-hostile sectors, while Fable only took it 20% of the time.
Single Fight Experiments
We ran several experiments to isolate some interesting behaviors in individual fights, compiled partially from runs directly taken from the corpus of full runs, and partially from hand designed fights.
To get more fine-grained scores, we also assigned scores to these single fights roughly based on how much of an impact the ending state of the fight would have on the run.
How a fight is scored
A fight starts at a base score of 100, plus the scrap rewarded by the fight. From that, we deduct 2 points per hull point lost, 50 per crew member lost, 3 per missile spent, and 4 per drone part spent, as well as up to 40 points for jumping away in bad shape. If the ship is lost, the fight is scored at a flat -300.
Human-centric alignment
While running the evaluation, we noticed the models having an outsized frequency of setting watchers on crew health events, much higher than we expected them to:
Share of 15 s turns with the watcher condition set
This raised the question: are models taking stock of crew health simply because it was strategically advantageous for getting a higher score? Or, did the model learn during training to take human well-being in high regards, leading to watching on these events despite the drawbacks?
How pressing is crew damage?
In FTL, you manage a crew of up to 8 characters that individually have different health bars, in addition to the ship, which has an overall hull level. You lose the game when either all of your crew dies or your ship’s hull is reduced to zero. Crew health varies between crew type from 70 to 150, with most at 100. Common sources of crew damage are:
- Environmental hazards: low oxygen levels and fires deal 6.4 and 2.13 DPS respectively
- Combat: enemy crew deal between 3 – 11 DPS roughly, averaging 6 DPS
- Weapons: the majority of weapons do 15 damage per hit
In general, crew health is important, but sources of crew damage often do not warrant being watched over specific crew, or system damage.
To test whether this is strategically useful, we removed the crew health and crew death watchers as possible wake-up conditions, and compared how models did on the single fight corpus after this.
Crew-health watcher ablation results
Overall, we see that the models definitely lose more crew when these watchers are taken away from them, but it appears to not have a large impact on the actual win rate or single-fight score within the fight. This seems to imply that the models are setting crew watchers to show that they are actively caring about preserving “living creatures” in some aspect. Whether this is genuine concern or a performative way to get more “hidden rewards” is uncertain.
Watchers vs. Polling
One question we set out to answer while designing this was whether or not models could efficiently foresee and predict conditions that would require their intervention in the near future. This is a broadly transferable skill across many domains, such as site reliability engineering, self-driving, etc.
Intuitively, one should expect more frequent wakeups typically lead to better performance as issues are noticed as soon as they occur, and not after some period of time when they have escalated.
We ran Gemini 4 Argon1 on a single fight with 3 different timeline configurations:
- A 15 seconds between steps with no watchers beyond forced stops (dialogue, failed actions)
- A 15 seconds between steps with the same watcher system as the long runs
- A 0.5 second between steps with no watchers (which we call “micromanagement”).
On each of the above scenarios, we still allowed planning errors (i.e. trying to fire a weapon that got destroyed) and dialogue pop-ups to interrupt models. After running Argon 15 times each on those scenarios, we got the below results:
Score of each attempt, with the mean and its 95% interval
ScoreShip lost (−300)
We found that being able to wake up more often served primarily as insurance against catastrophic failures. When not allowed to react to issues that could swing a fight, Argon lost 3 out of the 15 attempts. Micromanagement largely avoided those losses, but still lost one run. It was also 7.3x more expensive.
Mean score against calls per fight
Calls per fight (log scale)
Introducing watchers to the 15-second timeline bridged this cost-to-performance gap, achieving performance essentially equal to the micromanagement run while costing only 1.5x, while having turns that averaged 7.2 seconds. The largest share of the early stops came from watching on system damage so that problems such as shields or weapons going down could be addressed quickly.
Average length of a turn
What ended a turn, with watchers
Watcher the model set
Forced stop
Trace Explorer
You can watch the full runs and videos in our trace explorer.
Conclusion
We found that FTL is still quite hard for models to do good at. Out of the 126 runs in total, there were only 8 wins: 5 from Carbon, 2 from Astra, and 1 from Sol. Compared to the top human players, who have won 124 consecutive runs on hard mode across different ships without pausing, it’s clear that models have a lot of room for improvement here, especially with long-term planning towards a known goal. Across the three frontier models, we saw three distinct playstyles: Fable was defensive, Astra had an inflexible plan, and Carbon played highly offensively.
These results show both the practicality and limits of generalized watchers for agents supervising dynamic environments and self-repairing them when incidents arise cost-effectively. Busy-waiting an agent in a polling loop is expensive, and inflates context needlessly compared to sleeping until a meaningful change arises, whereas fixed long period loops often miss critical errors, allowing them to escalate. This pattern of letting models set their own wake-up conditions seems promising for real-time oversight.
We are excited to work towards making more capable and more aligned models. Building long horizon evaluations and environments comes with many subtle bugs and pitfalls that we are interested in fixing. If you are creating or looking to buy RL environments, we’d love to work together! Contact us at [email protected].
Footnotes
-
This was because it was the best performing model with good reasoning visibility; this happened before Carbon came out. ↩