11.3 Decision-Making (Behavior Trees, FSM, GOAP, Utility)

This chapter is about how a game AI agent decides what to do. Earlier chapters showed you how to move an object smoothly and how to organize gameplay code with components, events, and the State pattern. None of that answers the actual question a designer asks about an enemy: "why did it just do that?" This chapter covers four ways to answer that question, from simplest to most flexible: Finite State Machines (FSM), Behavior Trees (BT), Goal-Oriented Action Planning (GOAP), and Utility AI. You will write real C# for the first two, understand the concept and see a worked example for the last two, and finish knowing which tool to reach for on a real project.

1. How a Game AI Agent Decides What To Do

Every game AI, no matter how it is built, runs the same three-step loop every frame: look at the world (perception), decide what to do about it (decision-making), then actually do it (action). You already know how to write the "action" part — moving a Transform, playing an animation, firing a projectile. This chapter is entirely about the middle step: the part of the code that picks which action runs.

A "decision-making system" is just the piece of code that turns perception into a choice of action. The four systems in this chapter are four different ways to write that piece of code. They are not different AI "brands" competing for the same job — they solve the decision problem with different trade-offs, and real games often mix them:

• FSM — the agent is always in exactly one named mode, with explicit rules for switching modes. Simple, but does not scale well.
• Behavior Tree — a tree of small, reusable nodes checked in priority order every frame. The default choice at most studios today.
• GOAP — the agent is given a goal and a list of actions with requirements and effects, and a planner searches for a sequence of actions that reaches the goal.
• Utility AI — every possible action gets a numeric score, and the agent picks whichever scores highest right now.

2. Finite State Machines: States and Transitions

A finite state machine (FSM) is a system that is always in exactly one named state out of a fixed, known set, with explicit rules — called transitions — for switching from one state to another. You already met this shape as the "State pattern" when organizing gameplay code; here we use the exact same idea specifically to decide enemy behavior.

A classic enemy has three states: Patrol (walk a fixed route), Chase (run toward the player), and Attack (deal damage in melee range). The rules for switching between them:

Only four transitions connect three states, and every one of them is easy to justify out loud: "if it can see the player, it should chase; if it's close enough, it should attack." That clarity is the whole appeal of an FSM — you can point at any line of behavior and explain exactly when it applies.

3. Implementing an FSM in C#

A clean FSM implementation gives each state its own small class instead of one giant method full of if statements. Each state needs to know what to do when it starts (Enter) and what to do every frame while active (Tick):

public interface IEnemyState
{
    void Enter(EnemyAI enemy);
    void Tick(EnemyAI enemy);
}

Each concrete state implements that interface and decides, inside Tick, whether it is time to switch to a different state:

using UnityEngine;

public class PatrolState : IEnemyState
{
    public void Enter(EnemyAI enemy)
    {
        Debug.Log("Enter Patrol");
    }

    public void Tick(EnemyAI enemy)
    {
        enemy.MoveAlongPatrolRoute();

        if (enemy.CanSeePlayer())
        {
            enemy.ChangeState(new ChaseState());
        }
    }
}

public class ChaseState : IEnemyState
{
    public void Enter(EnemyAI enemy)
    {
        Debug.Log("Enter Chase");
    }

    public void Tick(EnemyAI enemy)
    {
        enemy.MoveTowardPlayer();

        if (enemy.DistanceToPlayer() < enemy.attackRange)
        {
            enemy.ChangeState(new AttackState());
        }
        else if (!enemy.CanSeePlayer())
        {
            enemy.ChangeState(new PatrolState());
        }
    }
}

public class AttackState : IEnemyState
{
    public void Enter(EnemyAI enemy)
    {
        Debug.Log("Enter Attack");
    }

    public void Tick(EnemyAI enemy)
    {
        enemy.AttackPlayer();

        if (enemy.DistanceToPlayer() >= enemy.attackRange)
        {
            enemy.ChangeState(new ChaseState());
        }
    }
}

The EnemyAI component just holds the current state and hands control to it every frame:

using UnityEngine;

public class EnemyAI : MonoBehaviour
{
    public float attackRange = 2f;
    private IEnemyState currentState;

    void Start()
    {
        ChangeState(new PatrolState());
    }

    void Update()
    {
        currentState.Tick(this);
    }

    public void ChangeState(IEnemyState newState)
    {
        currentState = newState;
        currentState.Enter(this);
    }

    public bool CanSeePlayer() { /* raycast toward the player */ return false; }
    public float DistanceToPlayer() { /* Vector3.Distance to the player */ return 999f; }
    public void MoveAlongPatrolRoute() { /* walk to the next waypoint */ }
    public void MoveTowardPlayer() { /* walk toward the player */ }
    public void AttackPlayer() { /* deal damage on a cooldown */ }
}

Expected output: when the enemy spawns, the console prints Enter Patrol. Once CanSeePlayer() returns true, it prints Enter Chase, and once DistanceToPlayer() drops below attackRange, it prints Enter Attack. A new state object is created on every ChangeState call, so any per-state field (a timer, a target position) always starts fresh.

4. Why Big FSMs Turn Into Spaghetti

Three states and four transitions are easy to reason about. Real enemies eventually need more: Flee when health is low, Search when the player was seen but is now hidden, Stunned after taking a heavy hit, Dead. Six states do not need six transitions — they need up to 6 × 5 = 30 possible directed transitions, because in principle any state might need to interrupt any other:

The pain is not just counting arrows — it is that every state class ends up needing to know about every other state that might interrupt it:

// FleeState now has to know about every other state that
// might need to interrupt fleeing -- coupling grows with
// every new feature anyone adds to the game.
public class FleeState : IEnemyState
{
    public void Tick(EnemyAI enemy)
    {
        if (enemy.Health <= 0)
        {
            enemy.ChangeState(new DeadState());
        }
        else if (enemy.IsCornered())
        {
            enemy.ChangeState(new StunnedState());
        }
        else if (enemy.Health > enemy.fleeThreshold)
        {
            enemy.ChangeState(new PatrolState());
        }
        else if (enemy.CanSeePlayer() && enemy.HasBackup())
        {
            enemy.ChangeState(new ChaseState());
        }
        // ...one more line for every future state, forever.
    }
}

This is the well-known problem with FSMs at scale: the number of transitions to think about grows roughly with the square of the number of states, every state class gets tightly coupled to many others, and the same guard conditions (is the enemy dead? stunned? out of ammo?) get copy-pasted into every single state. This tangled, hard-to-follow growth is exactly the spaghetti code problem from gameplay architecture, now inside your AI. It is not a sign you wrote a bad FSM — it is a structural limit of the FSM shape itself, and it is the reason the next tool exists.

5. Behavior Trees: Nodes and the Tick

A behavior tree (BT) is a tree of small nodes, evaluated top to bottom, left to right, once per frame — that single evaluation pass is called a tick. Every node returns one of three results when ticked:

• Success — the node finished, and it worked.
• Failure — the node finished, and it did not work.
• Running — the node is still in progress (a multi-frame action, like walking somewhere), ask again next tick.

public enum NodeStatus
{
    Success,
    Failure,
    Running
}

public interface IBTNode
{
    NodeStatus Tick();
}

There are two families of nodes. Composite nodes have children and combine their results; leaf nodes do the actual checking or acting and have no children. The two composite nodes you need to know first:

• Sequence — like a checklist ("AND"). Ticks its children left to right. If a child fails, the Sequence stops immediately and returns Failure. Only if every child succeeds does the Sequence return Success.
• Selector — like a priority list ("OR"). Ticks its children left to right. If a child succeeds (or is Running), the Selector stops immediately and returns that result. Only if every child fails does the Selector return Failure.

Notice both composites can short-circuit: a Sequence never ticks children after the first failure, and a Selector never ticks children after the first success. This "stop early" behavior is what lets a tree express priority — the leftmost branch of a Selector is tried first, and only if it fails does the tree even look at the next option.

6. Building a Behavior Tree Framework in C#

The composite nodes: Sequence and Selector

public class Sequence : IBTNode
{
    private readonly IBTNode[] children;

    public Sequence(params IBTNode[] children)
    {
        this.children = children;
    }

    public NodeStatus Tick()
    {
        foreach (var child in children)
        {
            NodeStatus status = child.Tick();
            if (status != NodeStatus.Success)
            {
                return status; // Failure or Running: stop here
            }
        }
        return NodeStatus.Success; // every child succeeded
    }
}

public class Selector : IBTNode
{
    private readonly IBTNode[] children;

    public Selector(params IBTNode[] children)
    {
        this.children = children;
    }

    public NodeStatus Tick()
    {
        foreach (var child in children)
        {
            NodeStatus status = child.Tick();
            if (status != NodeStatus.Failure)
            {
                return status; // Success or Running: stop here
            }
        }
        return NodeStatus.Failure; // every child failed
    }
}

The leaf nodes: Condition and Action

A Condition node checks something about the world and returns Success or Failure immediately — it is never Running. An Action node actually does something, and can return Running while the action is still in progress:

using System;

public class ConditionNode : IBTNode
{
    private readonly Func<bool> condition;

    public ConditionNode(Func<bool> condition)
    {
        this.condition = condition;
    }

    public NodeStatus Tick()
    {
        return condition() ? NodeStatus.Success : NodeStatus.Failure;
    }
}

public class ActionNode : IBTNode
{
    private readonly Func<NodeStatus> action;

    public ActionNode(Func<NodeStatus> action)
    {
        this.action = action;
    }

    public NodeStatus Tick()
    {
        return action();
    }
}

That is the whole framework: two composites, two leaf types, one shared Tick() contract. Every behavior tree you will ever build, no matter how large, is assembled from just these four building blocks.

7. A Full Behavior Tree: Patrol, Chase, Attack

Here is the same enemy from Section 3 — Patrol, Chase, Attack — rebuilt as a behavior tree instead of an FSM, so you can compare the two directly:

Read the tree the same way you read the Selector semantics from Section 5: try the leftmost branch first, and only fall through to the next one if it fails.

using UnityEngine;

public class EnemyAI : MonoBehaviour
{
    public float attackRange = 2f;
    private IBTNode root;

    void Start()
    {
        root = new Selector(
            new Sequence(
                new ConditionNode(CanSeePlayer),
                new Selector(
                    new Sequence(
                        new ConditionNode(InAttackRange),
                        new ActionNode(AttackPlayer)
                    ),
                    new ActionNode(ChasePlayer)
                )
            ),
            new ActionNode(Patrol)
        );
    }

    void Update()
    {
        root.Tick();
    }

    bool CanSeePlayer() { /* raycast toward the player */ return false; }
    bool InAttackRange() { /* distance check against attackRange */ return false; }

    NodeStatus AttackPlayer()
    {
        Debug.Log("Attack!");
        return NodeStatus.Running; // the attack animation is still playing
    }

    NodeStatus ChasePlayer()
    {
        Debug.Log("Chasing...");
        return NodeStatus.Running; // still closing the distance
    }

    NodeStatus Patrol()
    {
        Debug.Log("Patrolling...");
        return NodeStatus.Running; // walking to the next waypoint
    }
}

Expected output, traced by hand for three situations:

• Player not visible: root Selector ticks its first child, the outer Sequence. That Sequence ticks CanSeePlayer, which returns Failure. The Sequence stops immediately and returns Failure — ChasePlayer and AttackPlayer are never even checked. The root Selector moves on to its next child, Patrol, which prints Patrolling....
• Player visible but far away: CanSeePlayer succeeds, so the outer Sequence ticks the inner Selector. That Selector tries its first child, the inner Sequence, which ticks InAttackRange — Failure, since the player is far. The inner Sequence stops and returns Failure. The inner Selector falls through to ChasePlayer, which prints Chasing... and returns Running. Running propagates straight up through both Selectors, unchanged, to the root.
• Player visible and close: both conditions succeed, so AttackPlayer runs, prints Attack!, and returns Running.

Compare this to the FSM in Section 3: same three behaviors, same rules, but here nothing is hardcoded as "state X transitions to state Y." The priority order and the fallback logic live entirely in the shape of the tree.

8. Why Behavior Trees Scale Better Than FSMs

Go back to the Flee/Search/Stunned/Dead problem from Section 4 and picture it as a tree instead. Adding "flee when health is low" means adding one new Sequence branch near the top of the Selector:

root = new Selector(
    new Sequence(new ConditionNode(IsCritical), new ActionNode(Flee)),   // new
    new Sequence(new ConditionNode(CanSeePlayer), /* chase/attack subtree */ null),
    new ActionNode(Patrol)
);

Nothing inside the chase/attack subtree changed. PatrolState, ChaseState, and AttackState did not need a single new line added to check for "is the enemy fleeing right now" — the tree structure itself guarantees the Flee branch is checked first, and if it fails (health is fine), everything below it runs exactly as before. This is the core reason behavior trees scale better than FSMs:

• No transition table to maintain. The tick order and short-circuiting already encode "try this first, fall back to that" — there is no separate N × N grid of allowed transitions to keep in sync.
• New behavior means adding a node, not editing existing ones. Existing branches are untouched, so you cannot accidentally break Patrol while adding Flee.
• Subtrees are reusable. A "LookAround" or "PlayHurtAnimation" subtree, once built, can be dropped into several different parent branches or several different enemy types.
• Running handles multi-frame actions naturally. An FSM needs a manual bool or timer to track "is this action still happening"; a behavior tree gets that for free from the three-value return.

9. GOAP: Goal-Oriented Action Planning

FSMs and behavior trees both require a person to hand-author every path the agent can take. GOAP (Goal-Oriented Action Planning) takes a different approach: instead of authoring paths, you describe individual actions the agent can take, and a planner searches, at run time, for a sequence of those actions that reaches a goal. This is the technique the soldiers in the 2005 shooter F.E.A.R. made famous — instead of a hand-scripted "take cover, then reload, then shoot" routine, the game just told them the goal ("kill the target") and let each soldier work out its own plan from whatever actions were available and useful in the moment.

Actions: preconditions and effects

Each action has a precondition (what must already be true in the world for this action to be allowed) and an effect (what becomes true in the world once the action finishes), plus a cost the planner will try to minimize:

using System.Collections.Generic;

public class GoapAction
{
    public string Name;
    public float Cost = 1f;
    public Dictionary<string, bool> Preconditions = new Dictionary<string, bool>();
    public Dictionary<string, bool> Effects = new Dictionary<string, bool>();
}

GoapAction moveToCover = new GoapAction
{
    Name = "MoveToCover",
    Cost = 1f,
    // no preconditions -- the agent can always try to move to cover
    Effects = { ["InCover"] = true }
};

GoapAction reload = new GoapAction
{
    Name = "Reload",
    Cost = 1f,
    Preconditions = { ["AmmoInReserve"] = true },
    Effects = { ["HasAmmo"] = true }
};

GoapAction attackFromCover = new GoapAction
{
    Name = "AttackFromCover",
    Cost = 2f,
    Preconditions = { ["InCover"] = true, ["HasAmmo"] = true },
    Effects = { ["EnemyDead"] = true }
};

The planner: searching for a plan

Given a starting world state and a goal world state, the planner searches backward: "which action's effects would satisfy the goal? what does that action need as a precondition? keep chaining backward until every remaining requirement is already true right now." This is the same kind of graph search as pathfinding, except the "map" is made of world states instead of physical locations, and the "roads" between them are actions:

// Conceptual pseudocode for how the plan gets used once found.
// Real planners run an A*-style search with a priority queue --
// this sketch shows the idea, not a production implementation.
WorldState goal = new WorldState { EnemyDead = true };
WorldState current = agent.GetCurrentWorldState();

List<GoapAction> plan = Planner.FindPlan(current, goal, availableActions);

foreach (GoapAction step in plan)
{
    Debug.Log("Plan step: " + step.Name);
}
// Prints, for the example above:
//   Plan step: MoveToCover
//   Plan step: Reload
//   Plan step: AttackFromCover

The payoff is adaptability: if the world changes mid-plan (the cover gets destroyed, ammo runs out somewhere else), the agent simply re-plans from its new current state — nobody had to hand-author a rule for that specific situation, because the planner searches fresh every time it is asked.

10. Utility AI: Scoring Every Option

Utility AI skips both trees and planning. Every possible action gets a numeric score (also called its utility) computed from the current situation, using a small scoring function per action, and the agent simply runs whichever action scored highest this frame. It is the right tool when the "correct" choice genuinely depends on blending several continuous factors at once, rather than a clean yes/no rule.

public interface IUtilityAction
{
    string Name { get; }
    float Score(EnemyAI enemy);
    void Execute(EnemyAI enemy);
}

public class FleeAction : IUtilityAction
{
    public string Name => "Flee";

    public float Score(EnemyAI enemy)
    {
        // Full health never wants to flee; near-death wants it badly.
        float missingHealthFraction = 1f - (enemy.Health / enemy.MaxHealth);
        return missingHealthFraction * 10f; // ranges 0 .. 10
    }

    public void Execute(EnemyAI enemy) { enemy.RunAwayFromPlayer(); }
}

public class AttackAction : IUtilityAction
{
    public string Name => "Attack";

    public float Score(EnemyAI enemy)
    {
        if (enemy.Ammo <= 0) return 0f; // can't attack with no ammo

        float closeness = Mathf.Clamp01(1f - enemy.DistanceToPlayer() / enemy.MaxAttackDistance);
        return closeness * 6f; // ranges 0 .. 6, higher when the player is close
    }

    public void Execute(EnemyAI enemy) { enemy.AttackPlayer(); }
}

public class PatrolAction : IUtilityAction
{
    public string Name => "Patrol";

    public float Score(EnemyAI enemy) => 1f; // a low, constant fallback

    public void Execute(EnemyAI enemy) { enemy.MoveAlongPatrolRoute(); }
}

using System.Collections.Generic;
using UnityEngine;

public class UtilityAI : MonoBehaviour
{
    public EnemyAI enemy;
    private List<IUtilityAction> actions;

    void Awake()
    {
        actions = new List<IUtilityAction>
        {
            new FleeAction(),
            new AttackAction(),
            new PatrolAction()
        };
    }

    void Update()
    {
        IUtilityAction best = null;
        float bestScore = float.MinValue;

        foreach (var action in actions)
        {
            float score = action.Score(enemy);
            if (score > bestScore)
            {
                bestScore = score;
                best = action;
            }
        }

        best.Execute(enemy);
    }
}

Worked trace: say Health = 20, MaxHealth = 100, Ammo = 3, DistanceToPlayer = 4, MaxAttackDistance = 10:

Flee wins, even though the player is well within attack range and the enemy still has ammo, because health is critically low. This is exactly the kind of "it depends on several things at once" nuance an FSM would need an extra explicit rule for, and a behavior tree would need an extra Condition node for — Utility AI gets it for free, because the score functions already blend every factor together.

11. Choosing Among FSM, BT, GOAP, and Utility AI

None of these four tools is strictly "better" — each fits a different shape of decision problem:

• FSM — best for a small, genuinely fixed number of clearly distinct modes: a traffic light, a door, a menu screen, a simple enemy with three or four behaviors. Cheap to write and to explain. Avoid it once you are past roughly five or six states, or once states start needing to interrupt each other in many different ways (Section 4).
• Behavior Tree — the default choice for most enemy and NPC AI in shipped games today. Good whenever behavior is naturally "try this, then fall back to that," reuses well across many enemy types, and has strong tool support (Unreal's built-in editor, Unity's Behavior package and third-party tools).
• GOAP — best when you want genuinely adaptive, emergent sequences of actions in a world with many possible actions and world states, and hand-authoring every path would be impractical (F.E.A.R.'s squad tactics). Costs more to build, tune, and debug, and is usually reserved for higher-budget or systemically deep games.
• Utility AI — best when the decision is fundamentally "which of several valid options is best right now," blended from continuous factors: a Sims-style character weighing hunger against tiredness against fun, or an RTS unit picking a target from several in range. Avoids the abrupt, all-or-nothing flips a state machine or tree can produce when a value crosses a hard threshold.

A practical rule of thumb: start with the simplest tool that honestly fits — often an FSM for a small prototype — and move to a behavior tree the moment you feel yourself fighting the transition-table problem from Section 4. Reach for GOAP or Utility AI only once you have a concrete decision that the simpler tools genuinely cannot express cleanly, not because they sound more advanced.

12. Combining Techniques: A Realistic Hybrid

Shipped games rarely use exactly one of these in isolation. A common, practical combination is: use Utility AI (or a lightweight GOAP goal-picker) once, at a coarse grain, to choose a high-level goal, then hand that goal to a behavior tree that knows how to execute it step by step:

Utility AI is good at the coarse, nuanced "what should my priority be" question; a behavior tree is good at reliably executing the concrete steps of whatever priority won. Neither tool has to do the other's job. The same pattern works with GOAP standing in for the top layer: GOAP decides the sequence of high-level actions, and each individual action can itself be implemented as a small behavior tree. There is no rule that says a project must pick exactly one of these four systems — pick per problem, and combine them where it genuinely simplifies the code.

13. Glossary

• AI agent — the game object making decisions: an enemy, an NPC, a companion.
• Decision-making — the part of an agent's code that chooses which action to perform.
• Finite state machine (FSM) — a system always in exactly one named state, with explicit transition rules between states.
• State — one named mode of behavior (Patrol, Chase, Attack, ...).
• Transition — a rule that switches the FSM from one state to another.
• Behavior tree (BT) — a tree of nodes, ticked in priority order, that returns Success, Failure, or Running.
• Tick — one evaluation pass through a behavior tree (or one node), usually once per frame.
• Composite node — a behavior tree node with children that combines their results (Sequence, Selector).
• Leaf node — a behavior tree node with no children that actually checks or acts (Condition, Action).
• Sequence node — runs children left to right; stops and fails at the first failing child; succeeds only if all children succeed.
• Selector node — runs children left to right; stops and succeeds at the first succeeding child; fails only if all children fail.
• Running — a node status meaning "still in progress, tick me again next frame."
• GOAP (Goal-Oriented Action Planning) — a system where a planner searches a set of actions (with preconditions and effects) for a sequence that reaches a goal.
• Precondition — what must already be true in the world before an action is allowed to run.
• Effect — what becomes true in the world once an action finishes.
• Planner — the algorithm that searches for a valid, low-cost sequence of actions reaching a goal.
• World state — the set of relevant true/false facts a GOAP planner reasons about (HasAmmo, InCover, ...).
• Utility AI — a system that scores every possible action with a function and runs whichever scores highest.
• Score / utility function — the formula that turns the current situation into a single number for one action.
• Spaghetti code — code so tangled that tracing one part forces you to pull in many unrelated others.

14. Exercises

public class FleeState : IEnemyState
{
    public void Enter(EnemyAI enemy)
    {
        Debug.Log("Enter Flee");
    }

    public void Tick(EnemyAI enemy)
    {
        enemy.RunAwayFromPlayer();

        if (enemy.Health > 50)
        {
            enemy.ChangeState(new PatrolState());
        }
    }
}

// And inside EACH of PatrolState.Tick, ChaseState.Tick, AttackState.Tick,
// as the very first check:
if (enemy.Health <= 20)
{
    enemy.ChangeState(new FleeState());
    return;
}

root = new Selector(
    new Sequence(
        new ConditionNode(HeardNoise),
        new ActionNode(Investigate)
    ),
    new Sequence(
        new ConditionNode(CanSeePlayer),
        new ActionNode(Chase)
    ),
    new ActionNode(Patrol)
);

bool HeardNoise() { /* check a recent noise event */ return false; }
bool CanSeePlayer() { /* raycast toward the player */ return false; }

NodeStatus Investigate()
{
    Debug.Log("Investigating noise...");
    return NodeStatus.Running;
}

NodeStatus Chase()
{
    Debug.Log("Chasing...");
    return NodeStatus.Running;
}

NodeStatus Patrol()
{
    Debug.Log("Patrolling...");
    return NodeStatus.Running;
}

Flee   : (1 - 60/100) * 10 = 0.4 * 10 = 4.0
Attack : Ammo <= 0, so Score() returns 0 immediately, regardless of distance
Patrol : constant                                                    = 1.0

Winner: Flee, with a score of 4.0