13.3 Adaptive & Interactive Music

So far, game music in this course has meant one thing: press play on an AudioClip, set loop to true, and let it run underneath the game. That is fine for a title screen. It falls apart the moment the game itself is unpredictable — a fight that could last five seconds or five minutes, a boss that could appear the instant the player walks through a door, or a player who runs from combat and wanders back three times before actually fighting. No composer can write music timed to events that have not happened yet.

This chapter covers adaptive music (also called interactive music): music built, from the start, to change in response to what is happening in the game, instead of just playing through one fixed arrangement. Two techniques do almost all of the real work — horizontal re-sequencing and vertical layering — plus two supporting ideas, stingers and quantized (beat-aligned) transitions, that make both techniques sound intentional instead of glitchy. By the end you will have built a beat-quantized segment switcher and a layered-intensity mixer in C#, and you will understand what audio middleware like Wwise or FMOD is doing for you when a studio has the budget to use one.

1. Why a fixed song breaks in a game

A movie composer knows, to the frame, when the hero draws their sword. They write sixty seconds of music timed exactly to that moment. A game composer has no such luxury — the player decides when to draw the sword, and might do it during dialogue, mid-jump, or not for another ten minutes. Any system that just plays a fixed track and swaps to another fixed track the instant something happens will, almost always, swap in the middle of a musical phrase.

using UnityEngine;

public class NaiveMusicSwitcher : MonoBehaviour
{
    public AudioSource musicSource;
    public AudioClip exploreClip;
    public AudioClip combatClip;

    public void EnterCombat()
    {
        musicSource.clip = combatClip;
        musicSource.Play();   // starts immediately, wherever exploreClip happened to be
    }
}

Here musicSource is one AudioSource looping exploreClip. The moment EnterCombat() runs, whatever exploreClip happens to be doing gets cut off — mid-note, mid-drum-fill, wherever the player's timing landed — and combatClip starts fresh from its own beat one, with a completely different rhythm underneath it while the ear catches up.

This is the entire problem this chapter solves. Everything from here on is really one question asked in different forms: how do you change what the player hears without it sounding like a mistake?

2. Two answers: horizontal re-sequencing and vertical layering

Two techniques, used by nearly every game with dynamic combat (including HoYoverse titles like Genshin Impact and Honkai: Star Rail), solve this in different ways:

• Horizontal re-sequencing — the music is written as separate segments (an explore segment, a combat segment, a boss segment), and the game plays one at a time, choosing which segment comes next and when the switch is allowed to happen. It is called "horizontal" because you are moving along the timeline, picking which piece of music plays next.
• Vertical layering (also called stems) — the music is one ongoing piece split into separate instrument tracks (drums, bass, strings, melody) that all play at once, all the time, and the game mixes their volumes based on how intense things are right now. It is called "vertical" because at any single instant you are stacking layers on top of each other.

Neither technique is "better" — they solve different problems. Horizontal re-sequencing is for moments where the music itself needs to be a different piece (a village theme has nothing in common with a dungeon theme). Vertical layering is for moments where the music is still fundamentally the same idea, just more or less intense (the same boss theme, thin while its health is high, full once it drops low). Most real game scores use both at once: horizontal re-sequencing to move between big states like explore/combat/boss, and vertical layering inside each of those states to react to smaller changes, like how many enemies are left.

3. Musical time: beats, bars, and BPM

Both techniques depend on one idea: knowing exactly where you are in the music, measured in musical time instead of plain seconds. Three terms, used constantly in this chapter:

• BPM (beats per minute) — the tempo: how many beats occur in one minute of music.
• Beat — one pulse of the music, the thing you would tap your foot to.
• Bar (also called a measure) — a fixed group of beats, most commonly 4 beats per bar in "4/4 time" (a time signature meaning nothing more exotic than "count to four, then repeat"). A bar is usually the smallest chunk of music that still sounds like a complete musical idea, and segments are almost always authored in whole multiples of one.

Converting BPM into real seconds is arithmetic used in every remaining section of this chapter:

using UnityEngine;

public class TempoMath : MonoBehaviour
{
    public float bpm = 120f;
    public int beatsPerBar = 4;

    void Start()
    {
        double secondsPerBeat = 60.0 / bpm;
        double secondsPerBar = secondsPerBeat * beatsPerBar;
        Debug.Log("One beat = " + secondsPerBeat + "s, one bar = " + secondsPerBar + "s");
    }
}

One beat = 0.5s, one bar = 2s

At 120 BPM, one beat is exactly half a second, and one bar (4 beats) is exactly 2 seconds. This is part of why 120 BPM is such a common tempo for game composers to pick — it makes the seconds-per-bar math land on a clean, easy number.

One more piece: Unity's frame-based Time.time is not accurate enough for this. Time.time only updates once per rendered frame, and frame timing jitters — one frame might take 16ms, the next 19ms. Audio hardware runs on its own, much steadier clock, exposed in Unity as AudioSettings.dspTime (a double giving the current time of the audio system in seconds, unrelated to the frame rate). Every scheduling calculation in this chapter uses dspTime, never Time.time, because even a fraction of a frame's worth of drift is audible as a transition landing slightly off the beat.

4. Horizontal re-sequencing: swapping segments by game state

Horizontal re-sequencing starts by splitting the score into segments — separate audio clips, one per game state, all written at the same BPM and the same bar length so any one of them can follow any other. A typical setup has several interchangeable segments per state (so the player is not stuck hearing the exact one Explore clip on a loop — more on that in section 11), tied to a small state graph.

public enum MusicState { Explore, Combat, Boss }

[System.Serializable]
public class MusicStateSegments
{
    public MusicState state;
    public AudioClip[] variants;   // several clips per state, same tempo and bar length
}

Two things matter about that graph. First, states are usually mutually exclusive — the game is in exactly one of Explore, Combat, or Boss at a time, never a blend of two (that is what vertical layering is for, inside one state). Second, and this is the point of the rest of this section: every arrow is quantized. A request to move from Explore to Combat does not cut the instant it is requested — it gets queued, and only actually swaps on the next bar line. That single rule is what keeps horizontal re-sequencing from sounding like the naive switcher in section 1.

5. Quantized transitions: waiting for the bar line

Quantization, in the musical sense, means snapping a moment to the nearest point on a musical grid — usually the nearest beat or the nearest bar — instead of letting it happen at the exact, arbitrary instant it was requested. A quantized transition is a segment swap that has been snapped to a bar line this way.

The reasoning is straightforward once you see the diagram: every segment sharing the same BPM and the same bar length means beat one of any bar in Combat lines up rhythmically with beat one of any bar in Explore. If the switch always happens exactly on a bar line, the transition sounds like the composer wrote it that way on purpose — because, in effect, they did: they wrote every segment able to follow any other segment, as long as the switch respects the beat grid.

There is a real cost: quantizing adds latency. At 120 BPM with a 4-beat bar, a request arriving right after a bar line just started has to wait almost the full 2 seconds for the next one. For music, that small delay is worth it — a 2-second-late but clean transition sounds far better than an instant but jarring one. Section 9 revisits this trade-off for stingers, where the opposite choice is usually correct.

6. Building it: a beat-quantized transition scheduler

Turning section 5's diagram into code needs two Unity features: AudioSource.PlayScheduled(double dspTime) starts a clip at a precise, future point on the audio clock instead of "as soon as possible," and AudioSource.SetScheduledEndTime(double dspTime) stops a currently-playing clip at a precise future point. Both take the same dspTime units as AudioSettings.dspTime, which is what makes them line up exactly.

using UnityEngine;

// Horizontal re-sequencer: only one segment plays at a time, but a
// requested switch always waits for the next bar line before it happens.
public class HorizontalMusicPlayer : MonoBehaviour
{
    public AudioSource activeSource;    // currently playing segment
    public AudioSource standbySource;   // holds the next segment, cued up in advance

    public float bpm = 120f;
    public int beatsPerBar = 4;

    private double segmentStartDsp;     // dspTime the active segment's beat 1 landed on
    private AudioClip queuedClip;
    private bool hasQueuedClip = false;

    public void PlayFirstSegment(AudioClip clip)
    {
        segmentStartDsp = AudioSettings.dspTime + 0.1;   // small safety offset
        activeSource.clip = clip;
        activeSource.PlayScheduled(segmentStartDsp);
    }

    // called by gameplay code: "switch to this segment, whenever the beat allows it"
    public void RequestSegment(AudioClip clip)
    {
        queuedClip = clip;
        hasQueuedClip = true;
    }

    void Update()
    {
        if (!hasQueuedClip) return;

        double secondsPerBeat = 60.0 / bpm;
        double secondsPerBar = secondsPerBeat * beatsPerBar;

        double now = AudioSettings.dspTime;
        double elapsed = now - segmentStartDsp;
        double barsSoFar = System.Math.Ceiling(elapsed / secondsPerBar);
        double nextBarDsp = segmentStartDsp + barsSoFar * secondsPerBar;

        standbySource.clip = queuedClip;
        standbySource.PlayScheduled(nextBarDsp);
        activeSource.SetScheduledEndTime(nextBarDsp);

        Debug.Log("Queued switch to " + queuedClip.name + " at dsp time " + nextBarDsp
                   + " (in " + (nextBarDsp - now).ToString("F2") + "s)");

        AudioSource temp = activeSource;   // swap roles for the next request
        activeSource = standbySource;
        standbySource = temp;
        segmentStartDsp = nextBarDsp;

        hasQueuedClip = false;
    }
}

activeSource and standbySource are two separate AudioSource components holding the same role in turn — never both playing at once, but always ready to hand off. Two are needed, not one, because you cannot safely change the clip on a source that is mid-playback without stopping it first; the incoming segment has to be cued up on a second source in advance, scheduled to start the instant the outgoing one is scheduled to end.

RequestSegment just records which clip is wanted and sets a flag — it does no math itself, because the request can arrive at any arbitrary moment and should not block anything. The real work happens next time Update() runs: elapsed is how far into the active segment playback currently is, in seconds; dividing by secondsPerBar and rounding up with Math.Ceiling finds how many whole bars have to pass before reaching a bar line at or after right now; multiplying back by secondsPerBar and adding to segmentStartDsp gives the exact dspTime of that bar line.

Trace it with bpm = 120, beatsPerBar = 4 (so secondsPerBar = 2.0), a segment that started at segmentStartDsp = 10.0, and a RequestSegment call that lands right when Update() runs at dspTime = 11.3:

Queued switch to CombatA at dsp time 12 (in 0.70s)

elapsed came out to 1.3 seconds into a 2-second bar (bar one spans dsp 10.0 to 12.0). Math.Ceiling(1.3 / 2.0) rounds 0.65 up to 1, meaning "wait for one more bar boundary," which lands at 10.0 + 1 * 2.0 = 12.0 — the very next bar line, 0.7 seconds after the request came in. If the request had instead arrived at dsp 11.9, only 0.1 seconds before that same bar line, the wait would have been a mere 0.1 seconds instead of nearly a full bar — the wait length depends entirely on how close the request lands to the next bar line, never longer than one full bar.

7. Vertical layering (stems): same loop, different mix

Vertical layering starts from a different premise: instead of many different pieces of music, there is one ongoing piece, exported from the composer's session as several stems (separate audio files for each instrument group — drums, bass, strings/pads, melody — all the exact same length, all the exact same tempo, all meant to be played together). At any moment every stem is playing; what changes is how loud each one is.

The single most important rule of vertical layering: never stop or restart a layer once the piece begins. All four AudioSources start together, at the same instant, and then keep running for the entire time the player is in that state — sometimes for minutes. Only their volume changes. This guarantees the four layers can never drift apart rhythmically, because from the audio engine's point of view they are not four separate pieces of music being coordinated — they are four copies of one piece of music, always at the exact same playback position, just mixed at different loudness.

Compare this directly with horizontal re-sequencing: horizontal changes which melody the player hears (Explore vs Combat are different musical ideas). Vertical changes how much of one ongoing musical idea the player hears (the same boss theme, thin and quiet at first, full and loud once the fight escalates). A single float — an intensity value from 0 to 1 — is usually enough to drive it.

8. Building it: a layered-intensity mixer

using UnityEngine;

// All four layers share ONE clip length and ONE start time. This code
// never starts or stops a layer -- it only changes volume, so the
// layers can never drift out of sync with each other.
public class VerticalMusicMixer : MonoBehaviour
{
    public AudioSource drums;
    public AudioSource bass;
    public AudioSource strings;
    public AudioSource melody;

    [Range(0f, 1f)] public float intensity = 0f;
    public float fadeSpeed = 1.5f;   // volume units per second

    void Start()
    {
        double startDsp = AudioSettings.dspTime + 0.1;

        drums.volume = 0f;
        bass.volume = 0f;
        strings.volume = 0.3f;
        melody.volume = 1f;

        drums.PlayScheduled(startDsp);
        bass.PlayScheduled(startDsp);
        strings.PlayScheduled(startDsp);
        melody.PlayScheduled(startDsp);
    }

    void Update()
    {
        float drumsTarget   = intensity >= 0.6f  ? 1f : 0f;
        float bassTarget    = intensity >= 0.35f ? 1f : 0f;
        float stringsTarget = Mathf.Lerp(0.3f, 0.7f, intensity);
        float melodyTarget  = 1f;   // melody is always present, at full volume

        drums.volume   = Mathf.MoveTowards(drums.volume, drumsTarget, fadeSpeed * Time.deltaTime);
        bass.volume    = Mathf.MoveTowards(bass.volume, bassTarget, fadeSpeed * Time.deltaTime);
        strings.volume = Mathf.MoveTowards(strings.volume, stringsTarget, fadeSpeed * Time.deltaTime);
        melody.volume  = Mathf.MoveTowards(melody.volume, melodyTarget, fadeSpeed * Time.deltaTime);
    }

    // called from gameplay: enemy distance, combat state, boss phase, etc.
    public void SetIntensity(float value)
    {
        intensity = Mathf.Clamp01(value);
    }
}

Start() schedules all four layers with PlayScheduled at the exact same startDsp, with a small 0.1-second safety offset (enough time for Unity to actually prepare the clips before the deadline arrives). Scheduling all four from the same dspTime value, in the same frame, is what guarantees they begin on the same sample — four separate Play() calls made across even two or three different frames could each land a few milliseconds apart, which is not a huge gap, but is exactly the kind of tiny desync (sometimes called phasing) a trained ear can pick out over a few minutes of listening.

Update() is the only place the actual mixing decision happens, and it never touches the intensity field directly — gameplay code only ever calls SetIntensity(value), and Update() reads it every frame. Each layer gets a target volume computed from intensity: drums and bass are simple on/off thresholds (drums only above 0.6, bass only above 0.35), strings ramp smoothly across the whole range with Mathf.Lerp, and melody stays at full volume throughout, because a melody that vanishes at low intensity would make the music feel incomplete rather than calm. Mathf.MoveTowards eases each layer's actual volume toward its target at a fixed speed (fadeSpeed, in volume units per second) instead of snapping instantly, so a sudden intensity change fades in smoothly rather than jumping.

Trace a scenario with fadeSpeed = 1.5: intensity starts at 0 (settled), then SetIntensity(0.5) is called at t = 2.0s, and SetIntensity(0.8) is called at t = 5.0s:

Notice bass never needed to move the second time (it was already at its target of 1), and drums only started moving at t = 5.0s, because 0.5 never crossed its 0.6 threshold but 0.8 did. Nothing snaps — every change is a smooth ramp at the same fixed speed, which is why a sudden jump from 0 to 0.8 intensity still sounds like a fade, not a cut.

9. Stingers: short musical flourishes on events

A stinger is a short, one-shot musical phrase — a brass hit, a choir stab, a rising sweep — layered on top of whatever music is already playing, to punctuate a single instant: a boss appearing, the player leveling up, a rare item drop. Unlike sections 4-8, a stinger does not represent an ongoing state; it fires once and is done.

using System.Collections;
using UnityEngine;

public class MusicStingerPlayer : MonoBehaviour
{
    public AudioSource stingerSource;    // separate channel, layered on top of the music
    public AudioSource[] musicLayers;    // duck these briefly while the stinger plays

    public float duckAmount = 0.4f;      // fraction to lower music volume by
    public float duckTime = 0.15f;
    public float minCooldown = 1.0f;     // ignore repeats faster than this

    private float lastPlayTime = -999f;

    public void PlayStinger(AudioClip stingerClip)
    {
        if (Time.time - lastPlayTime < minCooldown)
        {
            Debug.Log("Stinger " + stingerClip.name + " suppressed (too soon)");
            return;
        }
        lastPlayTime = Time.time;

        stingerSource.PlayOneShot(stingerClip);
        StartCoroutine(DuckMusic());
        Debug.Log("Stinger: " + stingerClip.name);
    }

    private IEnumerator DuckMusic()
    {
        float[] original = new float[musicLayers.Length];
        for (int i = 0; i < musicLayers.Length; i++)
            original[i] = musicLayers[i].volume;

        for (int i = 0; i < musicLayers.Length; i++)
            musicLayers[i].volume *= (1f - duckAmount);

        yield return new WaitForSeconds(duckTime);

        for (int i = 0; i < musicLayers.Length; i++)
            musicLayers[i].volume = original[i];
    }
}

PlayOneShot plays the stinger clip without disturbing anything else stingerSource might play later, layered on top of the ongoing music. DuckMusic briefly ducks (temporarily lowers one sound's volume so another sound is clearly audible on top of it) every layer's volume by duckAmount for duckTime seconds, then restores each layer's original volume exactly, so a big stinger is not fighting the full mix for the listener's attention.

Trace two PlayStinger calls for the same clip, 0.3 seconds apart, with minCooldown = 1.0:

Stinger: BossReveal
Stinger BossReveal suppressed (too soon)

The cooldown exists because game events do not politely space themselves out — three enemies could die within the same half-second, each one wanting to fire the same "enemy defeated" stinger. Without a cooldown, three overlapping copies of the same short phrase turn into noise instead of three clean sting hits.

Unlike section 5's segment transitions, PlayStinger fires immediately, with no wait for a bar line. That is a deliberate choice: a stinger usually represents a sudden, surprising event, and instant feedback matters more here than rhythmic alignment — the player should hear the boss reveal the moment it happens, not up to two seconds later. Some games do quantize certain stingers to the nearest beat for a punchier, more "musical" hit; which way to go is a call the composer and the gameplay designer make together, event by event.

10. How middleware (Wwise/FMOD) does all this

Everything in sections 5 through 8 is real, useful code — and also exactly the kind of code a dedicated tool exists to do for you. Audio middleware (external tools used alongside Unity or Unreal, most commonly Audiokinetic Wwise and FMOD Studio) lets a composer or sound designer build the entire adaptive music system visually, without writing C#, and hand the game programmer a tiny API surface to call into.

Three ideas map directly onto what you already built:

• States — a single, global, named value (our MusicState enum, Explore/Combat/Boss) that the game sets with one call. The middleware's authoring tool already knows which segments belong to which state and how they are allowed to follow each other — this is horizontal re-sequencing, authored visually instead of in a C# switch statement.
• RTPCs (real-time parameter control) — a continuous value, exactly like our intensity float, that the game updates every frame. The composer draws a curve, per layer, mapping the parameter to that layer's volume inside the authoring tool — this is vertical layering, with the fadeSpeed/Mathf.Lerp math from section 8 replaced by a curve the composer can see and tweak without asking a programmer to change code.
• Sync points — markers (Bar, Beat, Entry Cue, Exit Cue) the composer places directly on the audio file's timeline in the authoring tool. When the game requests a state change, the middleware automatically quantizes it to the next marker of whatever type the composer chose — this is section 6's Math.Ceiling bar-line calculation, done for you.

// Hand-rolled (sections 5-8 of this chapter): the game computes the
// next bar line itself and schedules the switch/fade by hand.
horizontalPlayer.RequestSegment(combatClip);
verticalMixer.SetIntensity(0.8f);

// Wwise-style: the game only announces WHAT changed. The sound designer
// already placed Bar/Beat/Exit Cue markers and wrote the transition
// rules inside the Wwise authoring tool -- no C# math needed for it.
AkSoundEngine.SetState("MusicState", "Combat");

// FMOD-style: a continuous parameter (an RTPC) drives the vertical
// layers, mixed by the composer in FMOD Studio instead of by our
// fadeSpeed/Mathf.Lerp code.
musicInstance.setParameterByName("Intensity", 0.8f);

Notice how small the C# side becomes: SetState and setParameterByName are one line each. All the scheduling, ducking, and curve math from this chapter still happens — it just happens inside the middleware's own engine, driven by data the composer authored in a visual tool, instead of inside your MonoBehaviour.

11. Avoiding repetition fatigue

Repetition fatigue is what happens when a player notices, and gets tired of, hearing the exact same loop, transition, or stinger over and over. It is close to unavoidable in principle — a player can easily spend twenty minutes in one area that a composer wrote three minutes of music for — but a few habits keep it from being obvious:

• Multiple segment variants per state (E1, E2, E3 for Explore instead of one clip), picked so the same one rarely plays twice in a row.
• Stinger cooldowns (section 9) so a common event cannot spam the same short phrase.
• Deliberate quiet or sparse sections written into the arrangement itself, so the ear gets a rest instead of nonstop density.
• Small authored variation inside a loop — an extra fill, a slightly different melody line on the second pass — so consecutive loops are not audibly identical, even though they are the same length and tempo.

using UnityEngine;

public class SegmentPicker
{
    private AudioClip[] variants;
    private int lastIndex = -1;

    public SegmentPicker(AudioClip[] variants)
    {
        this.variants = variants;
    }

    // picks a random variant, but never repeats the same one twice in a row
    public AudioClip PickNext()
    {
        int index;
        do
        {
            index = Random.Range(0, variants.Length);
        }
        while (variants.Length > 1 && index == lastIndex);

        lastIndex = index;
        return variants[index];
    }
}

Trace five calls to PickNext() with three variants, [E1, E2, E3]. One possible run (the exact sequence is random, but the rule is not):

Your own run will very likely land on a different sequence, because Random.Range is genuinely random — the only guarantee the do/while loop enforces is that index is never equal to lastIndex when the loop exits, so two identical segments can never play back to back.

12. Glossary

• Adaptive music / interactive music — music built to change in response to gameplay, instead of playing through a fixed arrangement.
• Linear (fixed) music — one track played straight through or looped, with no reaction to game state.
• Horizontal re-sequencing — splitting music into segments and choosing which one plays next based on game state.
• Segment — one self-contained clip of music used by horizontal re-sequencing (an Explore clip, a Combat clip).
• Quantization (musical) — snapping an event to the nearest beat or bar instead of letting it happen at an arbitrary instant.
• BPM — beats per minute; the tempo of a piece of music.
• Beat / Bar (measure) — one pulse of music, and a fixed group of beats (commonly four) that forms one musical unit.
• AudioSettings.dspTime — Unity's steady audio-hardware clock, in seconds, used for precise scheduling instead of the frame-based Time.time.
• PlayScheduled / SetScheduledEndTime — Unity AudioSource methods that start or stop a clip at an exact future dspTime.
• Vertical layering / stems — one ongoing piece split into separate instrument tracks, mixed live by volume instead of swapped.
• Intensity parameter — a single value (often 0 to 1) that drives how loud each vertical layer is.
• RTPC (real-time parameter control) — middleware's term for a continuous value, like intensity, that the game updates every frame to drive a mix.
• Stinger — a short, one-shot musical phrase layered on top of the music to punctuate a single event.
• Ducking — temporarily lowering one sound's volume so another sound is clearly audible on top of it.
• Middleware — a dedicated audio tool (Wwise, FMOD) used alongside a game engine, letting a composer author adaptive music without writing engine code.
• State (middleware) — a single, global, named value the game sets to select which horizontal segment set is active.
• Sync point (Entry Cue / Exit Cue) — a marker placed on an audio file's timeline that middleware snaps transitions to.
• Repetition fatigue — a player noticing and tiring of hearing the same music over and over.

13. Exercises

That is adaptive music end to end: a fixed loop breaks because the game does not know its own timing in advance, horizontal re-sequencing solves it by swapping between authored segments on a quantized bar line, vertical layering solves a different problem by mixing always-running stems with one intensity value, stingers punctuate single instants on top of either, and middleware like Wwise or FMOD lets a composer author states, RTPCs, and sync points visually so the C# side shrinks to a couple of calls. The one idea that ties all of it together is musical time — BPM, beats, and bars, measured in dspTime instead of frame time — because every technique in this chapter is really just different ways of respecting it.