The above image remixes the Hydra code "Filet Mignon" from AFALFL and GLSL shader "Just another cube" from mrange. Licensed under CC BY-NC-SA 4.0 and CC0 respectively.
Patchies is a tool for building interactive audio-visual patches in the browser with JavaScript and GLSL. It's made for live creative coding -- for building visualizations, soundscapes and simulations of all kinds.
Try it out at patchies.app - it's open source 😎
Patchies lets you use the audio-visual tools that you know and love, together in one place. For example:
Patchies lets you "patch" multiple objects together using Message Passing, Video Chaining and Audio Chaining. It is very heavily inspired by tools such as Max/MSP, Pure Data, TouchDesigner, VVVV, and many others.
"What I cannot create, I do not understand. Know how to solve every problem that has been solved." - Richard Feynman
If you haven't used a patching environment before, patching is a visual way to program by connecting objects together. Each object does something e.g. generate sound, generate visual, compute some values. You can connect the output of one object to the input of another object to create a flow of data. We call the whole visual program a "patch" or "patcher".
Here's a visual example:
This patch takes in audio from an audio file, analyzes the frequency spectrum using the fft~
object, and sends the frequency data to various visual objects to create audio-reactive visuals and meters. You can create your own patches by combining different objects in different ways 😝
Enter
to create a new object.hydra
or glsl
or p5
.Arrow Up/Down
navigates the list.Enter
inserts the object.Esc
closes the menu.Enter
, but it lets you see all objects at a glance 👀Delete
to delete an object.Shift + Enter
while in a code editor to run the code again. This helps you to make changes to the code and see the results immediately.Ctrl/Cmd + K
brings up the command palette.To create shareable links, click on the "Share Link" button on the bottom right. You can also use "Share Patch" from the command palette.
You can use the Shortcuts button on the bottom right to see a list of shortcuts. Here are some of the most useful ones:
Click on object / title
: focus on the object.Drag on object / title
: move the object around.Scroll up
: zoom in.Scroll down
: zoom out.Drag on empty space
: pan the canvas.Enter
: create a new object at cursor position.Ctrl/Cmd + K
: open the command palette to search for commands.Shift + Enter
: run the code in the code editor within the selected object.Delete
: delete the selected object.Ctrl + C
: copy the selected object.Ctrl + V
: paste the copied object.Each object can send message to other objects, and receive messages from other objects. Here are some examples to get you started:
button
objects, and connect the outlet of one to the inlet of another.{type: 'bang'}
message to the second button, which will flash.msg
object with the message hello world
(you can hit Enter
and type m hello world
). Then, hit Enter
again and search for the message-console.js
preset. Connect them together.hello world
to the console object, which will log it to the virtual console.In JavaScript-based objects such as js
, p5
, hydra
, strudel
and canvas
, you can use the send()
and recv()
functions to send and receive messages between objects. For example:
// In the source `js` object
send('Hello from Object A')
// In the target `js` object
recv((data) => {
// data is "Hello from Object A"
console.log('Received message:', data)
})
This is similar to the second example above, but using JavaScript code.
The recv
callback also accepts the meta
argument in addition to the message data. It includes the inlet
field which lets you know which inlet the message came from.
You can combine this with send(data, {to: inletIndex})
to send data to only a particular inlet, for example:
recv((data, meta) => {
send(data, {to: meta.inlet})
})
In the above example, if the message came from inlet 2, it will be sent to outlet 2.
In js
, p5
and hydra
objects, you can call setPortCount(inletCount, outletCount)
to set the exact number of message inlets and outlets. Example: setPortCount(2, 1)
ensures there is 2 message inlets and 1 message outlet.
See the Message Passing with GLSL section for how to use message passing with GLSL shaders to pass data to shaders dynamically.
You can chain video objects together to create complex video effects, by using the output of a video object as an input to another. For example: P5 -> Hydra -> GLSL -> output
. This is similar to shader graphs in programs like TouchDesigner.
To use video chaining, connect the orange inlets and outlets on the patch. For example, connect the orange video outlet of a p5
to an orange video inlet of a hydra
object, and then connect the hydra
object to a glsl
.
See the glsl
object section for how to create uniforms to accept video inputs.
Similar to video chaining, you can chain many audio objects together to create complex audio effects.
You can use these objects as audio sources: strudel
, chuck
, ai.tts
, ai.music
, soundfile~
, sampler~
, video
as well as the web audio objects (e.g. osc~
, sig~
, mic~
)
dac~
to hear the audio output, otherwise you will hear nothing. Audio sources do not output audio unless connected to dac~
. Use gain~
to control the volume.You can use these objects to process audio: gain~
, fft~
, +~
, lowpass~
, highpass~
, bandpass~
, allpass~
, notch~
, lowshelf~
, highshelf~
, peaking~
, compressor~
, pan~
, delay~
, waveshaper~
, convolver~
, expr~
, dsp~
.
Use the fft~
object to analyze the frequency spectrum of the audio signal. See the Audio Analysis section on how to use FFT with your visual objects.
You can use dac~
to output audio to your speakers.
Here are the non-exhaustive list of objects that we have in Patchies.
These objects support video chaining and can be connected to create complex visual effects:
p5
: creates a P5.js sketchP5.js is a JavaScript library for creative coding. It provides a simple way to create graphics and animations, but you can do very complex things with it.
If you are new to P5.js, I recommend watching Patt Vira's YouTube tutorials on YouTube, or on her website. They're fantastic for both beginners and experienced developers.
Read the P5.js documentation to see how P5 works.
See the P5.js tutorials and OpenProcessing for more inspirations.
You can also use ML5.js in your P5 sketch to add machine learning capabilities. Call loadML5()
at the top of your sketch to load the ML5 library.
You can call these special methods in your sketch:
noDrag()
disables dragging the whole canvas. You must call this method if you want to add interactivity to your sketch, such as adding sliders or mousePressed events. You can call it in your setup()
function.noDrag()
is enabled, you can still drag the "p5" title to move the whole object around.send(message)
and recv(callback)
, see Message Passing.hydra
: creates a Hydra video synthesizersetVideoCount(ins = 1, outs = 1)
creates the specified number of Hydra source ports.setVideoCount(2)
initializes s0
and s1
sources with the first two video inlets.h
o0
, o1
, o2
, and o3
.send(message)
and recv(callback)
works here, see Message Passing.pipe.hydra
: passes the image through without any changesdiff.hydra
, add.hydra
, sub.hydra
, blend.hydra
, mask.hydra
: perform image operations (difference, addition, subtraction, blending, masking) on two video inputsfilet-mignon.hydra
: example Hydra code "Filet Mignon" from AFALFL. Licensed under CC BY-NC-SA 4.0.glsl
: creates a GLSL fragment shaderp5
, hydra
, glsl
, swgl
, bchrn
, ai.img
or canvas
) to the GLSL object via the four video inlets.uniform float iMix;
, it will create a float inlet for you to send values to.sampler2D
such as uniform sampler2D iChannel0;
, it will create a video inlet for you to connect video sources to.glsl
, as they accept the same uniforms.red.gl
: solid red colorpipe.gl
: passes the image through without any changesmix.gl
: mixes two video inputsoverlay.gl
: put the second video input on top of the first onefft-freq.gl
: visualizes the frequency spectrum from audio inputfft-waveform.gl
: visualizes the audio waveform from audio inputswitcher.gl
: switches between six video inputs by sending an int message of 0 - 5.You can send messages into the GLSL uniforms to set the uniform values in real-time. First, create a GLSL uniform using the standard GLSL syntax, which adds two dynamic inlets to the GLSL object.
uniform float iMix;
uniform vec2 iFoo;
You can now send a message of value 0.5
to iMix
, and send [0.0, 0.0]
to iFoo
. When you send messages to these inlets, it will set the internal GLSL uniform values for the object. The type of the message must match the type of the uniform, otherwise the message will not be sent.
If you want to set a default uniform value for when the patch gets loaded, use the loadbang
object connected to a msg
object or a slider. loadbang
sends a {type: 'bang'}
message when the patch is loaded, which you can use to trigger a msg
object or a slider
to send the default value to the GLSL uniform inlet.
Supported uniform types are bool
(boolean), int
(number), float
(floating point number), vec2
, vec3
, and vec4
(arrays of 2, 3, or 4 numbers).
swgl
: creates a SwissGL shadercanvas
: creates a JavaScript canvasYou can use HTML5 Canvas to create custom graphics and animations. The rendering context is exposed as canvas
in the JavaScript code, so you can use methods like canvas.fill()
to draw on the canvas.
You can call these special methods in your canvas code:
noDrag()
to disable dragging the whole canvas. this is needed if you want to add interactivity to your canvas, such as adding sliders. You can call it in your setup()
function.send(message)
and recv(callback)
, see Message Passing.bchrn
: render the Winamp Milkdrop visualizer (Butterchurn)hydra
and glsl
) to derive more visual effects.img
: display imagesstring
: load the image from the given url.video
: display videosbang
: restart the videostring
: load the video from the given url.{type: 'play'}
: play the video{type: 'pause'}
: pause the video{type: 'loop', value: false}
: do not loop the videobg.out
: background outputjs
: A JavaScript code blockconsole.log()
to log messages to the virtual console.setInterval(callback, ms)
to run a callback every ms
milliseconds.setInterval
that automatically cleans up the interval on unmount. Do not use window.setInterval
from the window scope as that will not clean up.requestAnimationFrame(callback)
to run a callback on the next animation frame.requestAnimationFrame
that automatically cleans up on unmount. Do not use window.requestAnimationFrame
from the window scope as that will not clean up.send()
and recv()
to send and receive messages between objects. This also works in other JS-based objects. See the Message Passing section above.setRunOnMount(true)
to run the code automatically when the object is created. By default, the code only runs when you hit the "Play" button.setPortCount(inletCount, outletCount)
to set the number of message inlets and outlets you want. By default, there is 1 inlet and 1 outlet.meta.inlet
in the recv
callback to distinguish which inlet the message came from.send(data, { to: inletIndex })
to send data to a specific inlet of another object.expr
: mathematical expression evaluatorEvaluate mathematical expressions and formulas.
Use the $1
to $9
variables to create inlets dynamically. For example, $1 + $2
creates two inlets for addition, and sends a message with the result each time inlet one or two is updated.
This uses the expr-eval library from silentmatt under the hood for evaluating mathematical expressions.
There are so many mathematical functions and operators you can use here! See the expression syntax section.
Very helpful for control signals and parameter mapping.
You can also create variables and they are multi-line. Make sure to use ;
to separate statements. For example:
a = $1 * 2
b = $2 + 3
a + b
This creates two inlets, and sends the result of (inlet1 * 2) + (inlet2 + 3)
each time inlet one or two is updated.
You can also define functions to make the code easier to read, e.g. add(a, b) = a + b
.
python
: creates a Python code environmentbutton
: a simple button{type: 'bang'}
message when clicked.any
: flashes the button when it receives any message, and outputs the {type: 'bang'}
message out.msg
: message objectEnter
and type m <message>
to create a msg
object with the given message.m {type: 'start'}
creates a msg
object that sends {type: 'start'}
when clicked.100
sends the number 100hello
or "hello"
sends the string "hello"{type: 'bang'}
sends the object {type: 'bang'}
. this is what button
does.{type: 'bang'}
: outputs the messageslider
: numerical value sliderEnter
and type in these short commands to create sliders with specific ranges:slider <min> <max>
: integer slider control. example: slider 0 100
fslider <min> <max>
: floating-point slider control. example: fslider 0.0 1.0
{type: 'bang'}
: outputs the current slider valuenumber
: sets the slider to the given number within the range and outputs the valuetextbox
: multi-line text input{type: 'bang'}
: outputs the current textstring
: sets the text to the given stringstrudel
: Strudel music environmentCtrl/Cmd + Enter
to re-evaluate the code.dac~
object to hear the audio output.recv
only works with a few functions, e.g. setcpm
right now. Try recv(setCpm)
to automate the cpm value.chuck
: creates a ChucK audio programming environmentCtrl/Cmd + \
: replaces the most recent shredCtrl/Cmd + Enter
: adds a new shredCtrl/Cmd + Backspace
: removes the most recent shredobject
: textual object systemEnter
, and type in the name of the object you want to create.gain~
object's gain value (e.g. 1.0
) to see the tooltip.Audio Processing:
gain~
: Amplifies audio signals with gain controlosc~
: Oscillator for generating audio waveforms (sine, square, sawtooth, triangle)lowpass~
, highpass~
, bandpass~
, allpass~
, notch~
: Various audio filterslowshelf~
, highshelf~
, peaking~
: EQ filters for frequency shapingcompressor~
: Dynamic range compression for audiopan~
: Stereo positioning controldelay~
: Audio delay line with configurable delay time+~
: Audio signal additionsig~
: Generate constant audio signalswaveshaper~
: Distortion and waveshaping effectsconvolver~
: Convolution reverb using impulse responsesfft~
: FFT analysis for frequency domain processingexpr~
: Audio expression evaluator for simple math expressionsdsp~
: Dynamic JavaScript DSP processor for custom audio algorithmsSound Sources:
soundfile~
: Load and play audio files with transport controlssoundurl~ <url>
to load audio files and streams from URLs directly.soundurl~ http://stream.antenne.de:80/antenne
to stream Antenne Bayern live radio.sampler~
: Sample playback with triggering capabilitiesmic~
: Capture audio from microphone inputdac~
: Send audio to speakersControl & Utility:
mtof
: Convert MIDI note numbers to frequenciesloadbang
: Send bang on patch loadmetro
: Metronome for regular timingdelay
: Message delay (not audio)adsr
: ADSR envelope generatorexpr~
: audio-rate mathematical expression evaluatorexpr
but runs at audio rate for audio signal processing.expr
, so the same mathematical expression will work in both expr
and expr~
.sig~
if you just need a constant signal.$1
to $9
: control inletss
: current sample value, a float between -1 and 1i
: current sample index in buffer, an integer starting from 0channel
: current channel index, usually 0 or 1 for stereobufferSize
: the size of the audio buffer, usually 128samples
: an array of samples from the current channelinput
: first input audio signal (for all connected channels), a float between -1 and 1inputs
: every connected input audio signalrandom()
creates white noises
outputs the input audio signal as-iss * $1
applies gain control to the input audio signals ^ 2
squares the input audio signal for distortion effectcompressor~
object with appropriate limiter-esque setting after expr~
to avoid loud audio spikes that can and will damage your hearing and speakers. You have been warned!dsp~
: dynamic JavaScript DSP processorThis is similar to expr~
, but it takes in a single process
JavaScript function that processes the audio. It essentially wraps an AudioWorkletProcessor
. The worklet is always kept alive until the node is deleted.
Here's how to make white noise:
function process(inputs, outputs) {
outputs[0].forEach((channel) => {
for (let i = 0; i < channel.length; i++) {
channel[i] = Math.random() * 1 - 1
}
})
}
Here's how to make a sine wave oscillator at 440Hz:
function process(inputs, outputs) {
outputs[0].forEach((channel) => {
for (let i = 0; i < channel.length; i++) {
let t = (currentFrame + i) / sampleRate
channel[i] = Math.sin(t * 440 * Math.PI * 2)
}
})
}
You can use $1
, $2
, ... $9
to dynamically create control inlets:
const process = (inputs, outputs) => {
outputs[0].forEach((channel) => {
for (let i = 0; i < channel.length; i++) {
channel[i] = Math.random() * $1 - $2
}
})
}
You can use the counter
variable that increments every time process
is called. There are also a couple more variables from the worklet global that you can use.
const process = (inputs, outputs) => {
counter // increments every time process is called
sampleRate // sample rate (e.g. 48000)
currentFrame // current frame number (e.g. 7179264)
currentTime // current time in seconds (e.g. 149.584)
}
midi.in
: MIDI inputmidi.out
: MIDI outputnetsend
: network message sendernetsend <channelname>
to create a netsend
object that sends messages to the specified channel name. Example: netsend drywet
netrecv
: network message receivernetrecv <channelname>
to create a netrecv
object that receives messages from the specified channel name. Example: netrecv drywet
These objects can be hidden via the "Toggle AI Features" command if you prefer not to use AI:
ai.txt
: AI text generationai.img
: AI image generationai.music
: AI music generationai.tts
: AI text-to-speechmarkdown
: Markdown rendererThe fft~
audio object gives you an array of frequency bins that you can use to create visualizations in your patch.
First, create a fft~
object. Set the bin size (e.g. fft~ 1024
). Then, connect the purple "analyzer" outlet to the visual object's inlet.
Supported objects are glsl
, hydra
, p5
, canvas
and js
.
sampler2D
GLSL uniform inlet and connect the purple "analyzer" outlet of fft~
to it.Enter
to insert object, and try out the fft-freq.gl
and fft-waveform.gl
presets for working code samples.uniform sampler2D waveTexture;
. Using other uniform names will give you frequency analysis.You can call the fft()
function to get the audio analysis data in the supported JavaScript-based objects: hydra
, p5
, canvas
and js
.
IMPORTANT: Patchies does NOT use standard audio reactivity APIs in Hydra and P5.js. Instead, you must use the fft()
function to get the audio analysis data.
fft()
defaults to waveform (time-domain analysis). You can also call fft({type: 'wave'})
to be explicit.
fft({type: 'freq'})
gives you frequency spectrum analysis.
Try out the fft.hydra
preset for Hydra examples.
Try out the fft-capped.p5
, fft-full.p5
and rms.p5
presets for P5.js examples.
The fft()
function returns the FFTAnalysis
class instance which contains helpful properties and methods:
fft().a
fft().getEnergy('bass') / 255
. You can use these frequency ranges: bass
, lowMid
, mid
, highMid
, treble
.fft().getEnergy(40, 200) / 255
fft().rms
fft().avg
fft().centroid
Where to call fft()
:
p5
: call in your draw
function.
canvas
: call in your draw
function that are gated by requestAnimationFrame
js
: call in your setInterval
or requestAnimationFrame
callback
setInterval(() => {
let a = fft().a
}, 1000)
hydra
: call inside arrow functions for dynamic parameters
let a = () => fft().getEnergy('bass') / 255
src(s0).repeat(5, 3, a, () => a() * 2)
Q: Why not just use standard Hydra and P5.js audio reactivity APIs like a.fft[0]
and p5.FFT()
?
p5-sound
and a.fft
APIs only lets you access microphones and audio files. In contrast, Patchies lets you FFT any dynamic audio sources 😊Converting Hydra's Audio Reactivity API into Patchies:
Replace a.fft[0]
with fft().a[0]
(un-normalized int8 values from 0 - 255)
Replace a.fft[0]
with fft().f[0]
(normalized float values from 0 - 1)
Instead of a.setBins(32)
, change the fft bins in the fft~
object instead e.g. fft~ 32
Instead of a.show()
, use the below presets to visualize fft bins.
Using the value to control a variable:
- osc(10, 0, () => a.fft[0]*4)
+ osc(10, 0, () => fft().f[0]*4)
.out()
Converting P5's p5.sound API into Patchies:
p5.Amplitude
with fft().rms
(rms as float between 0-1)p5.FFT
with fft()
fft.analyze()
with nothing - fft()
is always up to date.fft.waveform()
with fft({ format: 'float' }).a
, as P5's waveform returns a value between -1 and 1. Using format: 'float'
gives you Float32Array.fft.getEnergy('bass')
with fft().getEnergy('bass') / 255
(normalize to 0-1)fft.getCentroid()
with fft().centroid
If you dislike AI features (e.g. text generation, image generation, speech synthesis and music generation), you can hide them by activating the command palette with CMD + K
, then search for "Toggle AI Features". This will hide all AI-related objects and features, such as ai.txt
, ai.img
, ai.tts
and ai.music
.