Audioreactive Shaders

Resonon can drive GPU graphics from the same code that drives your sound. You write a fragment shader as a string, hand it to gfx.Visual(...) (from use "std/visuals" as gfx;), bind its uniforms to live signals and audio meters with .param("name") << …, and route it to the display with >>. The shader renders in a browser canvas, fed live over Resonon’s WebSocket server — so the visuals move in lockstep with the music.

The whole loop is live: edit the shader in VSCode, re-execute, and the picture hot-swaps without dropping a beat.

use "std/visuals" as gfx;

let frag = "#version 300 es
precision highp float;

uniform float iTime;        // transport time in seconds (built-in)
uniform vec2  iResolution;  // canvas size in pixels      (built-in)

out vec4 fragColor;

void main() {
  vec2 uv = gl_FragCoord.xy / iResolution;            // 0..1 across the canvas
  float wave = 0.5 + 0.5 * sin(iTime + uv.x * 6.2831);
  fragColor = vec4(uv.x, uv.y, wave, 1.0);
}
";

let v = gfx.Visual(frag);
v >> screen;
PLAY;

Run that and nothing appears yet — there’s no display attached. Let’s set one up, then take the example apart.

The Workflow

Visuals render in a browser, served by Resonon’s own server. So before the shader can show up, two processes need to be running alongside your editor. Open two terminals:

# Terminal 1 — start the engine + WebSocket server
resonon server

# Terminal 2 — open the browser render page (must be run after the server is up)
resonon visuals

resonon visuals checks that a server is listening (default 127.0.0.1:5555, override with --host/--port) and opens your default browser to the render page. Press f or click the canvas to go fullscreen.

With both running, the primary workflow is the same as the rest of Resonon: select the shader block in VSCode and execute it (Cmd/Ctrl+Enter). The shader and its bindings are pushed to the browser and start rendering. Tweak the shader string, select, and execute again — the picture hot-swaps live.

Note

The visual operators live in the std/visuals module — use "std/visuals" as gfx; to get gfx.Visual, gfx.load_shader, and the operator builders. screen is global (no import needed), like master. The shader source itself is a string, so you can build it however you like (a literal, a function that returns one, etc.).

Your First Shader

Here is the minimal shape again, in isolation:

use "std/visuals" as gfx;

let v = gfx.Visual(frag);   // compile a fragment-shader string into a Visual
v >> screen;                // route it to the display
PLAY;                       // run the transport (drives iTime)

gfx.Visual(source) takes a GLSL-ES 3.00 fragment shader as a string and returns a Visual. The shader is compiled in the browser, not by Resonon, so a syntax error surfaces later (see Live Editing) rather than at construction.

v >> screen routes the visual to the display, exactly as track >> master routes audio. There is one screen, so routing a new visual replaces whatever was showing.

PLAY matters here: the built-in iTime uniform follows the engine transport, so without PLAY the shader sits frozen at time zero.

Loading Shaders From Files

Embedding GLSL in a Resonon string is convenient for short shaders, but longer ones are easier to write in a real .glsl file — your editor’s shader tooling (highlighting, linting, completion) works on it directly. In VSCode, install a GLSL extension such as Shader languages support (slevesque.shader) to highlight .glsl files. Read the file with load_file:

use "std/visuals" as gfx;

let v = gfx.Visual(load_file("shaders/plasma.glsl"));   // read the .glsl beside this file
v >> screen;
PLAY;

// or, equivalently:
let v = gfx.load_shader("shaders/plasma.glsl");

load_file(path) returns the file’s contents as a string; the path is resolved relative to the .non file being run. gfx.load_shader(path) reads and compiles the .glsl in one step. To share GLSL snippets between shaders, compose the source strings in Resonon — e.g. gfx.Visual(load_file("shaders/lib.glsl") + frag).

There is no filesystem watcher: after editing the .glsl, re-execute the routing line to recompile — load_file re-reads the file each time, so the new source hot-swaps in.

Built-in Uniforms

Four uniforms are supplied to every shader automatically — declare them and use them, no binding required. They come from the engine transport, so they stay beat-synced and freeze when you STOP (and resume in sync on the next PLAY).

Uniform	Type	Meaning
iTime	float	Transport time in seconds
iResolution	vec2	Canvas size in pixels (supplied by the browser)
iBeat	float	Continuous beat position
iCycle	float	Continuous cycle position

These names are reserved — trying to bind one (v.param("iTime") << ...) is an error, because the engine already supplies it.

Binding Uniforms

Any other uniform you declare in the shader is yours to drive. Name it with .param("…") and bind it with <<, just like a track parameter. The value can be a constant, a signal, or a pattern:

v.param("brightness") << 0.7;                  // a constant
v.param("energy")     << Sine(8).range(0.2, 1.0);   // an LFO, remapped to 0.2..1.0
v.param("hue")        << Saw(16);              // a slow ramp, 0..1 over 16 cycles

Signal periods are measured in cycles, so Sine(8) completes one oscillation every 8 cycles. Use .range(lo, hi) to remap a signal’s default range to whatever your shader expects. Re-binding the same uniform replaces the previous source.

Every bound uniform is sampled once per rendered frame, off the audio thread — binding a signal never touches the real-time path.

Going Audioreactive

The point of all this is to react to sound. Every audio track and the master bus expose two live meter feeds — .rms (average level) and .peak — and you can bind them straight to a uniform:

v.param("kick")  << drums.rms;     // a track's level
v.param("level") << master.peak;   // the whole mix

Raw meter feeds are spiky — bound straight to brightness they make the shader strobe on every transient. Add .smooth(ms) to apply a one-pole envelope (same time-constant idea as Signal.smooth), so the value glides up on a hit and eases back down:

v.param("kick") << drums.rms.smooth(280);   // soft 280ms envelope — glides instead of flashing

Larger values are smoother/slower (≈100–200 ms feels punchy, ≈300–500 ms feels like a swell). Then drive smooth shader responses with it — nudge hue, saturation, or glow rather than multiplying brightness by a sharp transient.

Fast attack, slow release

One time constant smooths the rise and the fall equally — but musically you almost always want to catch the transient fast and let it decay slowly. Give .smooth two times, smooth(attackMs, releaseMs), and the rising edge uses the attack, the falling edge the release (the TouchDesigner Lag CHOP model):

v.param("kick") << drums.rms.smooth(20, 300);   // snap up in 20ms, decay over 300ms

Set the attack to 0 for an instant rise — a peak follower that jumps to every hit and eases back down:

v.param("kick") << drums.rms.smooth(0, 400);    // instant attack, 400ms release

Single-arg smooth(ms) is just the symmetric case (attack == release).

Here is a complete, runnable example: a CR-78 beat, two LFO-driven uniforms, and a smoothed kick uniform wired to the drum track’s RMS so the shader pulses with the groove.

use "std/signals" { Sine, Saw };
use "std/instruments" { Sampler, Kit };
use "std/visuals" as gfx;

let frag = "#version 300 es
precision highp float;

uniform float iTime;        // transport seconds   (built-in)
uniform vec2  iResolution;  // canvas pixels        (built-in)
uniform float energy;       // 0..1 brightness      (LFO-driven)
uniform float hue;          // 0..1 color shift      (LFO-driven)
uniform float kick;         // smoothed drum RMS     (audio-driven)

out vec4 fragColor;

vec3 hsv2rgb(vec3 c) {
  vec3 p = abs(fract(c.xxx + vec3(0.0, 2.0 / 3.0, 1.0 / 3.0)) * 6.0 - 3.0);
  return c.z * mix(vec3(1.0), clamp(p - 1.0, 0.0, 1.0), c.y);
}

void main() {
  vec2 uv = (gl_FragCoord.xy * 2.0 - iResolution) / min(iResolution.x, iResolution.y);
  float r = length(uv);
  float rings = 0.5 + 0.5 * sin(r * 10.0 - iTime * 2.0);
  // The smoothed kick lifts brightness gently and nudges the hue — a glide, not a flash.
  float bright = mix(0.1, 1.0, energy) * rings + kick * 0.7;
  fragColor = vec4(hsv2rgb(vec3(hue + r * 0.2 + kick * 0.1, 0.8, bright)), 1.0);
}
";

let v = gfx.Visual(frag);

// LFO-driven uniforms — periods are in cycles.
v.param("energy") << Sine(8).range(0.2, 1.0);
v.param("hue")    << Saw(16);

// A beat, so the transport runs and there is audio to react to.
let drums = AudioTrack("drums");
drums.load_instrument(Sampler(Kit("CR-78")));
drums << [bd [sd _ _ bd] bd sd];

// Live audio: the drum RMS, smoothed into a soft envelope so it glides.
v.param("kick") << drums.rms.smooth(280);

v >> screen;
PLAY;

With resonon server and resonon visuals running, executing this block plays the beat and renders concentric rings that breathe with the LFOs and flash on every kick.

The Frequency Spectrum

.rms and .peak give you one number — how loud a channel is. Sometimes you want to see which frequencies are playing: bass in the low bins, hats in the high ones. Every track and the master bus also expose .fft, a live FFT magnitude spectrum.

There are two ways to consume it. Bind .fft whole and it arrives as a 1-D data texture your shader samples bin by bin. Or slice out a Hz range with .fft.band(lo, hi) and it collapses to a single float — usually all you want.

The Whole Spectrum as a Texture

A spectrum is an array, not a scalar, so bound whole it arrives as a 1-D data texture (sampler2D) rather than a float uniform — the same idea as ShaderToy’s audio input. Declare the sampler, bind .fft to it, and read bin x ∈ [0,1) (low → high frequency) with a texture lookup:

uniform sampler2D spectrum;   // an FFT feed, low freq at x=0
// ...
float mag = texture(spectrum, vec2(uv.x, 0.5)).r;   // this column's bin

Bins are linear in frequency and normalized so a full-scale sine reads ≈ 1.0 at its bin.

Read raw, the texture strobes: an FFT bin is spiky, the texture is NEAREST-filtered, and it jumps every FFT hop. Add .smooth(ms) — the same one-pole time-constant idea as .rms.smooth, applied per bin — so the whole spectrum glides frame to frame instead of flickering:

use "std/instruments" { Sampler, Kit };
use "std/visuals" as gfx;

let frag = "#version 300 es
precision highp float;
uniform sampler2D spectrum;   // an FFT feed, low freq at x=0
uniform vec2 iResolution;
out vec4 fragColor;
void main() {
  vec2 uv = gl_FragCoord.xy / iResolution;
  float mag = texture(spectrum, vec2(uv.x, 0.5)).r;   // this column's bin
  float bar = step(uv.y, mag * 4.0);                  // simple bar graph
  fragColor = vec4(vec3(bar), 1.0);
}";
let v = gfx.Visual(frag);

let drums = AudioTrack("drums");
drums.load_instrument(Sampler(Kit("CR-78")));
drums << [bd [hh sd hh] bd sd];

v.param("spectrum") << master.fft.smooth(120);   // per-bin 120ms glide
v >> screen;
PLAY;

Because each .fft binding is its own texture, you can wire different channels to different samplers and react to them independently — drive one part of the image from the drums’ highs and another from the bass:

v.param("drumSpec") << drums.fft.smooth(80);
v.param("bassSpec") << bass.fft.smooth(200);

The two-time form works here too: smooth(attackMs, releaseMs) gives each bin a fast attack and a slow release, so the spectrum snaps up on a hit and decays gently — the same envelope idea as .rms, applied per bin:

v.param("spectrum") << master.fft.smooth(20, 300);   // per-bin fast attack, slow release

An analyzer runs on a channel only while its .fft is bound to a routed visual, so unused channels cost nothing. Binding two smoothings of the same channel (master.fft and master.fft.smooth(120)) still runs one analyzer — the smoothing happens downstream, and each distinct (attack, release) is its own glided texture over the shared tap.

Scalar Bands

Sampling the texture yourself is powerful but fiddly: to react to “the bass” you have to average a range of bins in GLSL. .fft.band(loHz, hiHz) does that for you — it integrates the bins in a Hz range (Hann-weighted, so the band edges roll off softly) down to a single float. Because the result is a scalar, the ordinary meter pipeline chains straight on: .smooth(ms) to glide it, .range(lo, hi) to remap it into whatever your shader wants.

use "std/instruments" { Sampler, Kit };
use "std/visuals" as gfx;

let frag = "#version 300 es
precision highp float;
uniform vec2  iResolution;
uniform float bass;     // 20-200 Hz   energy, smoothed
uniform float treble;   // 4-12 kHz    energy, smoothed
out vec4 fragColor;
void main() {
  vec2 uv = (gl_FragCoord.xy * 2.0 - iResolution) / min(iResolution.x, iResolution.y);
  float r = length(uv);
  // Bass swells the disc; treble sparkles a ring around it — both glide, no flash.
  float disc  = smoothstep(0.6 + bass, 0.55 + bass, r);
  float ring  = treble * smoothstep(0.02, 0.0, abs(r - 0.7));
  fragColor = vec4(vec3(disc) + ring * vec3(1.0, 0.7, 0.3), 1.0);
}";
let v = gfx.Visual(frag);

let drums = AudioTrack("drums");
drums.load_instrument(Sampler(Kit("CR-78")));
drums << [bd [hh sd hh] bd sd];

v.param("bass")   << master.fft.band(20, 200).smooth(150).range(0, 0.4);
v.param("treble") << master.fft.band(4000, 12000).smooth(80).range(0, 4);
v >> screen;
PLAY;

.band reads the channel’s raw FFT tap, so a spectrum-level .smooth() before it has no effect — to smooth a band, put the time constant after the integration: .band(...).smooth(ms), not .fft.smooth(...).band(...).

Beyond Raw Shaders: Operators

Writing GLSL by hand is one way to make a picture. The other is to compose ready-made operators — gfx.Noise(), gfx.Blur(), gfx.Displace(), and more — into a TouchDesigner-style node graph, wired with the same >>/<< you have used here. That is a topic of its own: see Visual Operators.

Live Editing & Errors

Re-running v >> screen is the moment everything updates — it pushes the current shader source and bindings to the browser. That is what makes live editing work: tweak the shader string or change a binding, then re-execute the block to hot-swap it.

Because the browser compiles the shader, a GLSL mistake can’t crash the engine. When a shader fails to compile, Resonon reports a diagnostic on the v >> screen line with the GLSL error log, the audio keeps playing, and the last shader that compiled keeps rendering. Fix it and re-execute to recover.

Notes & Limits

• Shaders are GLSL-ES 3.00 fragment shaders rendered with WebGL2. Write #version 300 es and a precision qualifier at the top, and output to an out vec4.
• Bound uniforms are scalars (float) or vectors (vec2/vec3/vec4, bound from a 2–4 element array). The built-in iResolution is always a vec2. A whole .fft feed binds as a sampler2D data texture instead; a .fft.band(lo, hi) slice is a scalar float (see The Frequency Spectrum).
• This page covers writing shaders, not GLSL itself — for the shading language, see any GLSL-ES reference.

Next Steps

• Signals & Automation — the LFOs and envelopes you bind to uniforms
• Tracks & Master — where .rms / .peak meter feeds come from
• Live Workflow — the select-and-execute loop in depth