Part I : Foundations of Form

vega is a fluent interface: a convenience layer that takes MayaFlux’s power and hides the verbosity without hiding the machinery.

Making complex logic less verbose is a good way to encourage more people to explore. But complexity that is presented well is not an obstacle. It is what gives you something to choose between, and choices are where agency and creativity come from. So vega shortens the path to the first result and leaves every door on that path open.

If you didn’t have vega, loading a sound file would look like this:

auto reader = std::make_shared<IO::SoundFileReader>();

if (!reader->can_read("path/to/file.wav")) {
    return;
}

reader->set_target_sample_rate(Config::get_sample_rate());
reader->set_audio_options(IO::AudioReadOptions::DEINTERLEAVE);

auto options = IO::FileReadOptions::EXTRACT_METADATA | IO::FileReadOptions::EXTRACT_REGIONS;
if (!reader->open("path/to/file.wav", options)) {
    return;
}

auto container = std::dynamic_pointer_cast<Kakshya::SoundFileContainer>(reader->create_container());

if (!reader->load_into_container(container)) {
    return;
}

container->set_memory_layout(Kakshya::MemoryLayout::ROW_MAJOR);

auto processor = std::dynamic_pointer_cast<Kakshya::ContiguousAccessProcessor>(container->get_default_processor());
if (!processor) {
    processor = std::make_shared<Kakshya::ContiguousAccessProcessor>();
    container->set_default_processor(processor);
}
processor->set_output_size({ Config::get_buffer_size(), container->get_num_channels() });
processor->set_auto_advance(true);

Depending on your exposure to programming, this can feel complex or liberating. Being explicit means you decide what is created, when, and with which settings.

vega says: “You just want to load a file? Say so.”

auto container = vega.read_audio("path/to/file.wav");

Same machinery underneath: the same reader, the same resampling to your project’s sample rate, the same processor setup.

What vega does:

• Picks the reader from the file
• Applies sensible defaults
• Checks for errors
• Builds and configures the container
• Returns the real object

What vega doesn’t do:

• Hide the idea. It removes the verbosity, not the machinery.
• Make the object less capable. What you get back is the full container.
• Remove your ability to do it explicitly. You can always write the long version when you need control.

The short syntax is convenience. The long syntax is control. MayaFlux gives you both.

A container is a large collection of data held as a whole: every sample of a recording, every frame of a film. It knows its own shape and size, and it has a processor that knows how to hand out pieces of it.

A buffer holds one step of that, and the information needed to use it:

• For sound, a buffer holds one block of samples.
• For a picture, a buffer holds the data for one frame, plus the rendering information that goes with it: which shaders draw it, which window it goes to, and the fields those need, whether they apply to one frame or to all of them.

Containers are where material rests. Buffers are where it gets used.

This is why every example above has two parts: a loader that makes a container (or, for an image, a buffer that already holds its pixels), and a registration that gives a buffer somewhere to go.

The | operator is quick registration. It hands the object to a domain and gives you the same object back, so the line keeps working as a value:

auto image = vega.read_image() | Graphics;

Audio and Graphics are domains. A domain tells the engine when and where to process the object: audio at the sound card’s pace, graphics once per frame.

Registration is the part that is the same every time, so | does it for you. Written by hand:

• container | Audio is get_io_manager()->hook_audio_container_to_buffers(container)
• video | Graphics is get_io_manager()->hook_video_container_to_buffer(video), and the same for the sound that came with it. A camera takes hook_camera_to_buffer(camera)
• buffer | Graphics is register_graphics_buffer(buffer)
• network | Graphics is register_node_network(network, Nodes::ProcessingToken::VISUAL_RATE), after switching the network to graphics output if it is not already

Whenever the choice is load bearing, you do it by hand. That is why the image and the model need a second line to choose a window, and why the video is wired by hand: one file can feed more than one domain, and only you know where each part should go.

vega.read_audio() on its own does this:

• Opens the file and decodes it
• Resamples to your project’s sample rate
• Builds a container with a processor attached
• Returns the container

It does not start playback, create buffers, or connect to your audio hardware. Try it:

void compose() {
    auto container = vega.read_audio();
}

The file loads and nothing plays. The container sits in memory, and you decide what happens next: route it to your speakers with | Audio, or analyse it, or send its numbers somewhere else.

Loading is separate from routing. You can load a file and send it to hardware immediately, or spend the next twenty lines building something before it ever plays.

vega.read_audio() opened the dialog, then decoded the whole file into memory:

1. Created a reader and checked that the file is readable
2. Resampled to the engine’s sample rate
3. Deinterleaved the samples, so each channel is its own array
4. Created a SoundFileContainer and loaded every sample into it
5. Set the memory layout to row major
6. Configured a ContiguousAccessProcessor: the container’s default processor, which knows how to hand out the samples block by block

That processor does the work of access. It:

• Sets its output size to one buffer’s worth of samples, for every channel
• Tracks where in the file you are
• Auto-advances: each time it is asked, it moves forward
• Gives channels separately, so each can be processed on its own

The processor is why a container is more than a data holder. It has built in logic for how it should be consumed.

| Audio hooked the container to buffers. By hand:

auto container = vega.read_audio();
auto buffers = get_io_manager()->hook_audio_container_to_buffers(container);

That call creates one buffer per channel, and the buffer manager owns them. Inside it is this loop:

auto buffer_manager = get_buffer_manager();

for (uint32_t channel = 0; channel < container->get_num_channels(); ++channel) {
    auto buffer = buffer_manager->create_audio_buffer<Buffers::SoundContainerBuffer>(
        Buffers::ProcessingToken::AUDIO_BACKEND, channel, container, channel);
    buffer->initialize();
}

Step by step, for each channel:

1. Create a SoundContainerBuffer, a buffer that reads from a container
2. Tag it with AUDIO_BACKEND: it feeds the sound card at the audio pace
3. Put it on the output channel with that number
4. Tell it which channel of the container to read
5. Initialize it

A stereo file’s left channel feeds output 0 and its right channel feeds output 1. Channels stay separate so that each can later have its own processing.

Every audio cycle, each buffer asks the container’s processor for the next block of samples and passes it on. Get the buffers back later with get_io_manager()->get_audio_buffers(container).

input.enabled and input.channels ask the engine to open the sound card’s input before anything runs. The engine then keeps one input buffer per hardware channel and fills it every audio cycle with the samples that arrived.

create_input_listener_buffer(0, true) does this by hand:

auto buffer = std::make_shared<Buffers::AudioBuffer>(0);
register_audio_buffer(buffer, 0);
read_from_audio_input(buffer, 0);

1. Make an audio buffer for channel 0
2. Register it on output channel 0, which is the true: it is why you hear it
3. Register it as a listener of input channel 0

Each cycle the input buffer copies its samples into every listener. With false the listener exists and receives sound, but nothing sends it to the speakers.

If you leave input disabled there is no input channel to listen to, and the call logs an error that the channel is out of range.

This is the one sound example with no container. Live sound has no whole to hold: it exists one block at a time, and the buffer is all there is.

Everything in this card ran on the engine’s defaults: 48 kHz, 512 frames per block, stereo output and 60 frames per second. settings() is where you change them.

settings() is a function in your src/user_project.hpp. It runs before the engine starts, and what you set there is fixed for the life of the program. By the time compose() runs the engine is already going, and changing a setting then logs a warning and has no effect. That is why the microphone needs settings(): the sound card’s input is opened when the engine starts, so it has to be asked for beforehand.

Settings come in five sections:

• stream: sample rate, block size, and the input and output channels and devices
• graphics: frame rate, the window surface, which GPU to use, the default font
• input: MIDI, OSC and other devices
• network: UDP and TCP
• journal: how much the engine tells you, and where

In code:

void settings() {
    auto& stream = Config::get_global_stream_info();
    stream.buffer_size = 256;
    stream.input.enabled = true;
    stream.input.channels = 1;

    Config::get_global_graphics_config().target_frame_rate = 60;
}

The same settings in a file, mayaflux.json, in your project’s source folder:

{
  "stream": {
    "buffer_size": 256,
    "input": { "enabled": true, "channels": 1 }
  },
  "graphics": {
    "target_frame_rate": 60
  }
}

The file is read every time the program starts, so changing it needs no rebuild. The launcher picks up mayaflux.json by itself when it exists, so settings() can stay empty. The keys are the names of the config fields, and any field you leave out keeps its default.

Do not confuse "stream" and the "input" inside it, which is your sound card’s input, with the top level "input" section, which is for MIDI, OSC and other devices.

Choosing the file. To use a file from somewhere else, start the program with --config path/to/file.json.

Which one wins. The file is read before settings(), so when both set the same value, settings() wins. Start the program with --config-override to turn that around: the file is then read after settings() and replaces all of it, so anything the file does not mention goes back to its default.

Reading settings back. Once the engine is running you can read what was set:

void compose() {
    uint32_t rate = Config::get_sample_rate();
    uint32_t block = Config::get_buffer_size();
    uint32_t channels = Config::get_num_out_channels();
}

Logging is the one exception to the rule that settings are fixed at start: the journal settings take effect at once.

docs/Settings.md lists every field.

vega.read_image() decoded the file into pixels with four channels: red, green, blue and alpha. Common formats are eight bits per channel. An .exr file stays floating point. The pixels went into a TextureBuffer, which also holds a flat rectangle for the picture to be drawn on.

An image never becomes a container. It goes straight to a buffer, because a picture is one frame. By hand:

IO::ImageReader reader;
reader.open("path/to/image.png");
auto image = reader.create_texture_buffer();

| Graphics is then register_graphics_buffer(image). It initializes the buffer on the GPU and adds it to the buffers the engine processes each frame.

At that point the buffer exists and is processed, but nothing draws it, because nothing says where.

setup_rendering says where. It is the line that turns a buffer holding a picture into a picture on screen. With the defaults written out:

image->setup_rendering({
    .target_window = window,
    .fragment_shader = "texture.frag.spv",
});

The TextureBuffer already carries its defaults: the vertex shader texture.vert.spv, the fragment shader texture.frag.spv, and a texture binding named texSampler. What setup_rendering does with them:

1. Keeps the shaders, using yours where you give them
2. Records the window the buffer draws to
3. Declares the texture binding on the render pipeline
4. Binds the buffer’s picture to that binding
5. Adds a render processor to the buffer’s processing chain, aimed at the window

A TextureBuffer draws as a triangle strip and ignores any other topology. There is no default window: you choose one, and that is the second line.

The same call, with the same meaning, appears on every buffer that draws. MeshBuffer and MeshNetworkBuffer have their own, and the video buffer uses the TextureBuffer one.

vega.read_mesh() read every mesh in the file. By hand:

const auto folder = std::filesystem::path("path/to/model.glb").parent_path();
auto resolver = [folder](const std::string& name) {
    return IO::ImageReader::load_texture((folder / name).generic_string());
};

IO::ModelReader reader;
reader.open("path/to/model.glb");
auto meshes = reader.create_mesh_buffers(resolver);
reader.close();

for (auto& mesh : meshes) {
    register_graphics_buffer(mesh);
}

1. The reader imports the file, triangulating faces and generating normals when the file has none
2. Each mesh becomes a MeshBuffer holding its vertices, with position, normal, tangent, texture coordinates and color, and its triangle indices
3. The resolver finds each mesh’s diffuse texture next to the model, and the texture is bound to that buffer when it is found
4. Registering each buffer is what | Graphics does on the group

setup_rendering then adds the render processor, with depth testing on so nearer surfaces hide farther ones. It picks the textured shader when the mesh has a texture and the plain one when it does not.

vega.read_mesh_network() read the same file, but kept its parts together. Each mesh became a slot in a MeshNetwork: a name, a node holding that mesh’s geometry, a transform of its own, and the diffuse texture when one was found.

| Graphics registered the network as a visual network, one the engine advances every frame, after switching it to graphics output if it was not already. A network has no way to draw itself, because drawing needs a buffer, so something still has to make one. By hand:

register_node_network(network, Nodes::ProcessingToken::VISUAL_RATE);

auto form = std::make_shared<Buffers::MeshNetworkBuffer>(network);
register_graphics_buffer(form);
form->setup_rendering({ .target_window = window });

vega.mint does that whole sequence: it registers the network if it is not registered, wraps it in a MeshNetworkBuffer, and sets up rendering aimed at the window you give it. Then it returns the real buffer. That is why the window goes inside the StructureConfig::Model and not in a separate line.

The network buffer draws every slot with its own transform, and uses each slot’s texture when it has one.

choose_video returns a pair: the video container first, the audio second. Destructuring it with auto [video, audio] gives each half a name. With a path instead of the dialog:

auto [video, audio] = get_io_manager()->load_video(
    "path/to/film.mp4", { .video_options = IO::VideoReadOptions::EXTRACT_AUDIO });

What that load does:

1. Checks the file is readable
2. Applies the options: video options, audio options, and the size to decode at
3. When EXTRACT_AUDIO is set, asks for audio at the engine’s sample rate
4. Opens the file and registers the reader with the IO manager, which gives it an id the decode machinery uses to find it
5. Creates a VideoFileContainer and loads it
6. Configures its processor: a FrameAccessProcessor that auto-advances at your project’s frame rate
7. When audio was asked for, takes the audio track, configures it as for any sound file, and returns it as the second half

The video container holds the film’s frames and information about them. The processor advances through the frames, a little each graphics cycle, scaled by the file’s own frame rate against the engine’s, so the picture moves at the speed the file was made for.

The audio is also kept by the IO manager, keyed by its video: get_io_manager()->get_extracted_audio(video) finds it later.

video | Graphics hooks the picture and, when the file has sound you asked the reader to extract, hooks that sound for playback too. By hand, for each:

auto picture = get_io_manager()->hook_video_container_to_buffer(video);
picture->setup_rendering({ .target_window = window });

auto buffers = get_io_manager()->hook_audio_container_to_buffers(audio);

The last line is audio | Audio. This fragment assumes the film has a soundtrack, as in the video-with-sound block. If the file has none, load_video returns an empty audio result and no audio buffer is produced. Afterwards get_associated_buffer(video) finds the picture’s buffer, and get_associated_buffers(video) finds the sound’s, one per channel.

hook_video_container_to_buffer made a VideoContainerBuffer, which is a TextureBuffer that copies the current frame into its texture each cycle. That is why setup_rendering is the same line as for an image. When the container reaches its end the buffer removes itself, so the video plays once.

One pipe registers both, but they are still two objects. Nothing in this card ties their clocks together.

A recording only belongs to audio, and a picture only belongs to graphics, so one | is enough for each. A video file belongs to both. A shortcut would have to guess where the picture and the sound should go, and a wrong guess is invisible until it sounds or looks wrong.

So here you write both lines. Wiring by hand is not the hard way. It is the way that shows what the machinery is doing, and it is the way you keep when you want to send the sound somewhere else.

vega.read_camera is the camera version of vega.read_audio and the video steps:

auto camera = vega.read_camera();
auto buffer = get_io_manager()->hook_camera_to_buffer(camera);

The first line asks which camera and which mode, and opens it. It gives back the camera itself, the way vega.read_audio gives back the sound, and nothing is shown yet. The second hooks it to a buffer, and camera | Graphics does the same, after which get_associated_buffer(camera) finds the buffer. That is the same kind of hook as hook_video_container_to_buffer, so what you get back is the same kind of buffer and the same setup_rendering line applies. The buffer is already registered, so there is no | Graphics.

The camera is opened through FFmpeg. A camera has no file and no end: frames arrive as the device makes them, and a separate thread decodes one when the graphics cycle asks for it, so the device never holds up drawing. The mode you choose is a request, and the device falls back to the nearest one it accepts, so it may give something else. Frames arrive as four channels: red, green, blue and alpha.

To skip the windows, pass a CameraConfig with the device name: vega.read_camera({ .device_name = "/dev/video0" }). /dev/video0 is the first camera on Linux. On macOS use "0", and on Windows use "video=Integrated Camera" or the name your camera reports. choose_camera(false) skips only the mode window. If you close a window or the device cannot be opened, the call returns nothing.

A camera gives picture only. Its sound counterpart is the microphone.

vega.read_audio(), read_image() and read_mesh() with no argument open the dialog and load what you choose. With a path they skip it.

choose_video is the dialog form for video. It takes the options for loading: an empty {} is the defaults, and EXTRACT_AUDIO asks for the sound as well. Its path form is get_io_manager()->load_video(path, options), shown above.

If you cancel, nothing is loaded and the call returns an empty result. The examples in this card assume you chose a file.

Fluent (What happens behind the scenes)

vega.read_audio() | Audio;

This single line loads the file into a container, creates a buffer for every channel, and registers them with the sound card. Nothing plays until the | Audio, which is when the connection happens.

Explicit (What’s actually happening)

auto container = vega.read_audio();
auto buffers = get_io_manager()->hook_audio_container_to_buffers(container);

Understanding the difference:

• The fluent version loads and hooks in one line
• The explicit version separates the two, so you can inspect or change the container before it is hooked
• With a chosen, readable file, both create the same per-channel playback buffers

create_sampler loads a recording for reading from, and gets a sampler ready to play it. Without it:

auto stream = get_io_manager()->load_audio_bounded("path/to/your/file.wav", 48000 * 5, true);

auto sampler = std::make_shared<Kriya::SamplingPipeline>(
    stream, *get_buffer_manager(), *get_scheduler(), 0, Config::get_buffer_size());
sampler->build();

1. load_audio_bounded reads the file into a DynamicSoundStream, resampled to your project’s sample rate and laid out so any frame can be reached directly
2. The SamplingPipeline is made with the stream, the engine’s managers, an output channel and the block size
3. build() supplies the sampler’s buffer to that output channel and starts the pipeline that drives it every audio cycle

The numbers are the ones create_sampler uses by default: 48000 * 5 frames, and true to cut the file off there. That is five seconds at 48 kHz, 2.5 seconds at 96 kHz, or about 5.44 seconds at 44.1 kHz. To request a duration at the current rate, calculate its frame count: create_sampler("file.wav", Config::get_sample_rate() * 20) asks for twenty seconds.

A sampler plays one channel of the file. By default that is the channel with the same number as its output channel, which is channel 0 for a stereo file’s left side. create_sampler(path, frames, true, 1) plays channel 1 on output 1, and create_samplers(path) makes one sampler per channel.

store keeps the sampler alive. A sampler stops sounding when the last handle to it goes away, and the handle in compose() goes away when compose() returns.

In the last card the sound lived in a SoundFileContainer: one reader walking through it from start to end. A sampler’s sound lives in a DynamicSoundStream. It holds the same samples, but any number of voices can read it at once, because each voice keeps its own position and its own slot.

That is why the card can play two loops of one recording together, and why the tap set can read the same samples three ways. Nothing is copied. The recording exists once.

A sampler and a tap set can even share one stream:

auto sampler = store(create_sampler());
auto taps = create_tap_set_from_stream(sampler->get_stream()).tap().speed(2.0).start();
store(taps);

create_sampler_from_stream and create_tap_set_from_stream take a stream you already have. The forms that load a file open a dialog, or take a path as their first argument, because they have to load first.

Each voice of a sampler is a slice: a region of the stream together with how to play it. slice_from_range(start, end) makes one, and the end frame is included.

A slice carries its own settings:

• looping and loop_count: whether it repeats, and how many times
• scale: its level
• speed: see the next expansion
• time_map: a function from time to position, in the expansion after that
• source_channel: which channel of the stream it plays

play_continuous(0, slice) is the short form of three steps:

auto slice = sampler->slice_from_range(0, rate - 1);
slice.looping = true;
sampler->load(0, slice);
sampler->play(0);

1. load puts the slice in voice slot 0 and gives it its own cursor
2. play rewinds that cursor to the start of the region and switches the voice on
3. Every audio cycle, each voice that is on reads its next block, scales it, and adds it to the sampler’s buffer. A voice that is off adds silence

A voice that does not loop switches itself off when it reaches the end of its region. Edits to a playing voice are picked up at the next block: sampler->slice(0).looping = false;.

A plain voice reads one block of the recording, then moves its position forward by one block times speed. At speed = 1.0 that is exactly one block, so it plays the recording as it is.

At any other speed it skips or overlaps blocks. Above 1.0 it jumps over part of the recording. Below 1.0 each block starts before the previous one has finished, so stretches are heard twice. What you hear is not a change of pitch but a rhythmic skipping. Speed must also be above zero, so a plain voice cannot play backwards.

If you want the pitch to change, or to read backwards, give the voice a time map instead. That is what the tap set does.

A time map is a function from the seconds since a voice started to a position in the recording, in frames. A voice with a time map no longer steps through blocks. Each audio cycle it works out where the map says it should be at the start and at the end of the block, and reads every frame in between, interpolating between neighbouring frames. Because it reads frames at a different rate than they were recorded, the pitch changes, and a map that runs backwards plays backwards.

A voice at half speed, looping a two second region:

#include "MayaFlux/Kinesis/Tendency/TimeMap.hpp"

void compose() {
    const auto rate = Config::get_sample_rate();

    auto sampler = store(create_sampler());
    sampler->play_continuous(0, sampler->slice_from_range(0, rate * 2 - 1)
        .with_time_map(Kinesis::TimeMaps::quadratic(0.0, rate * 0.5, 0.0)));
}

The map is quadratic(start, velocity, acceleration): it starts at frame 0 and moves at half the recording’s own rate, which is rate * 0.5 frames per second. The factories in Kinesis::TimeMaps:

• linear(from, to, duration): a straight path, then stays
• quadratic(start, velocity, acceleration): constant acceleration, so a speed that rises or falls
• triangle(lo, hi, velocity, start): back and forth between two positions
• piecewise_linear(points, duration): through a list of positions, such as a hand drawn curve
• exponential(from, to, duration): a constant ratio each step, for values that shrink or grow geometrically

A map replaces speed. While a voice has a map, loop_count is ignored.

A tap is a voice with a time map that moves at a speed you can change while it plays. Without the builder, one tap looks like this:

#include "MayaFlux/Kinesis/Tendency/TimeMap.hpp"

void compose() {
    const auto rate = static_cast<double>(Config::get_sample_rate());

    auto sampler = store(create_sampler());
    auto velocity = std::make_shared<double>(1.5 * rate);

    sampler->play_continuous(0, sampler->slice_from_range(0, static_cast<uint64_t>(rate * 4.0) - 1)
        .with_time_map(Kinesis::TimeMaps::integrated(0.0, velocity)));
}

integrated adds up a velocity as time passes, and reads that velocity from a value you keep. Write *velocity = rate * 0.5; while it plays and the voice slows from where it is, with no jump. Zero holds it still, and a negative number plays it backwards.

create_tap_set does this for every tap you declare, and keeps the velocities for you:

• tap() begins a new tap, and the calls that follow describe it
• speed(ratio), with a negative ratio meaning backwards, and backward() for reading from the end of the region
• enters_after(seconds), level(gain), on_channel(n) and on_channels({ 0, 1 })
• region(start_frame, end_frame), which applies to every tap and uses an exclusive end frame (unlike slice_from_range), and loop(false) to play once

Taps loop by default. After start() the set can still be changed:

taps.level(1, 0.3);
taps.speed(0, 0.5);
taps.stop();

speed(tap, ratio) carries on from where the tap is, so it never jumps. stop() silences every tap, including ones that have not entered yet.

One sampler is made for each output channel a set uses, and each tap is a voice on it. A mono file plays on whichever output a tap is given.

A tap reads the recording by interpolating between frames, so reading it faster or slower changes the pitch. A speed of 1.5 is a frequency ratio of 3 to 2, a fifth. A speed of 2.0 is an octave.

Two taps at 1.0 and 1.001 read the same samples a thousandth apart in speed. In the example above, the second enters two seconds after the first, so the readings start at different places in the recording. From then on, their positions drift by a thousandth of a second per second. How that sounds depends on the recording.

Voices in one sampler add together inside its buffer; taps assigned to the same output channel share a sampler. By default the output channel divides its mix by the number of active buffers feeding it. Adding another voice to the same sampler does not increase that divisor. Adding a separate sampler buffer can make the existing buffers quieter.

To turn that off for the two stereo channels:

set_mix_normalization_for_channels({ 0, 1 }, false);

With it off, buffers add at full level. Lower a voice’s or tap’s level to keep the total under control; normalization by buffer count does not prevent several voices within one buffer from adding up.

repeat_every(seconds) restarts a tap from the start of its region every so many seconds, so a long recording becomes a short loop. Given a time map instead of a number, the length is read at the start of each repeat. A length that shrinks gives a stutter.

Each repeat restarts the tap’s own clock, and the cut between repeats is exact to the frame, not the block.

.repeat_every(Kinesis::TimeMaps::exponential(0.5, 0.03, 8.0))

This repeat starts at half a second and shrinks by the same ratio every step until it is 30 milliseconds, reached after eight seconds. Because each repeat is shorter than the one before, the sound speeds up into a buzz. linear(0.4, 0.05, 6.0) shrinks by the same amount every step instead, a more even slowing of the bounces.

A tap with lag ignores repeat_every and speed. A lagged tap trails the write head of a recording that is still being made, as in the last example of this section.

On tape, position follows a motor. One head sits at one place, the speed changes pitch and length together, turning around means stopping and reversing the spool, a canon needs one machine per voice, and an echo needs a second head at a fixed distance. A sound that is still happening cannot be played back until it has finished and been rewound.

In memory there is no head and no motor. A voice with a time map has a position that is whatever function of time you give it, returning a frame number. That is the whole mechanism, and it is why the examples above are possible:

• A sine gives a smooth reversal, with no stop of the spool. The sound slows to a halt at the turn and glides away again
• A list of positions, such as the random walk, gives any path you can compute or measure. The list can come from anywhere
• A different function on each voice gives each voice its own time

You write the function directly, as a Kinesis::TimeMap:

Kinesis::TimeMap { .fn = [](const double& t) -> double { return 48000.0 * t; } }

t is the seconds since the voice started and the result is a position in frames. This one is just normal speed at 48 kHz. Every factory in Kinesis::TimeMaps is a function like this one.

A tap’s map can be replaced while it plays: taps.slice(0).with_time_map(map). Setting a new map restarts that voice’s clock at zero.

The live case. create_ring(8.0) makes a stream that keeps only the most recent eight seconds of whatever is written into it, overwriting the oldest, and record_into writes a live signal into it every audio cycle. A tap with lag reads that ring a certain distance behind the write head, and because the lag is a time map too, the distance itself can change. A lag that grows is a tap falling behind, so it plays slower and lower. A lag that swings is a pitch that wobbles. A lagged tap stays at least two blocks behind the head, so it never reads what is being written.

ring->snapshot(frames) copies the most recent frames into a new stream, and any tap set can read it. That is the fork: a stretch of the past, taken out of the flow and given to a voice that keeps going without it.

A granular call does four things in a row:

1. Segment: cut the recording into grains of grain_size frames, starting a new one every hop_size frames
2. Attribute: measure each grain and write the number onto it under the name feature_key
3. Sort: order the grains by that number, lowest first or highest first
4. Reconstruct: write the grains out in that order as a new recording

process_to_container runs all four. Granular::process runs the first three and gives you the ordered list, so you can look at it:

#include "MayaFlux/WorkflowIncludes.hpp"

void compose() {
    auto sound = vega.read_audio();
    auto config = Granular::GranularConfig { .grain_size = 4800, .hop_size = 4800, .feature_key = "rms" };

    auto grains = Granular::process(sound, AnalysisType::FEATURE, config, "rms");

    for (size_t i = 0; i < 4 && i < grains.data.regions.size(); ++i) {
        auto onset = grains.data.regions[i].start_coordinates[0];
        auto rms = grains.data.regions[i].get_attribute<double>("rms");
        std::cout << "grain at frame " << onset << ", rms " << rms.value_or(-1.0) << "\n";
    }
}

This prints the four quietest grains: where each came from in the original, and how loud it was. The ordered list is the whole composition. Reconstruction only reads it out.

A grain is a region of the recording. It starts at a frame, called its onset, and covers grain_size frames across every channel. It carries its measurements as named attributes: the onset, and whatever you measured under feature_key.

Segmentation makes whole grains only. The last stretch that is too short for a full grain is dropped. With the default hop of half a grain, every sample lands in two grains. With hop_size equal to grain_size every sample lands in exactly one.

Grains are measured on one channel, channel in the config, which is 0 by default. The grains themselves are moved with all their channels together, so a stereo recording stays stereo.

There are two ways to measure.

A named measurement. Pass an AnalysisType and a name. AnalysisType::FEATURE with "rms" is the loudness of each grain. AnalysisType::STATISTICAL with "variance" is how much its samples vary. The number is written on each grain under feature_key, and the sort reads that same name.

Your own measurement. Pass an AttributeExecutor: a function that receives one grain’s samples and returns one number. The second block above measures zero-crossing rate. The analyzer is made once, before the grains are measured, with its settings named, and the function reuses it for every grain. Change the method and the order changes:

auto analyzer = std::make_shared<StandardEnergyAnalyzer>(
    EnergyAnalyzerConfig { .window_size = 512, .hop_size = 256, .method = EnergyMethod::ZERO_CROSSING });

window_size and hop_size are how the analyzer walks along the grain. With ZERO_CROSSING, extract_scalar_energy(analysis, "mean_energy") takes the average zero-crossing rate across those windows; mean_energy is the analyzer’s generic field name. Other names it understands are "max_energy", "min_energy", "variance" and "dynamic_range". Zero crossings count how often the signal changes sign. They can rise with pitch or noise; they are not a reliable measure of brightness on their own. EnergyMethod::RMS measures signal level. Any function you can write over a span of samples will do.

The measure changes the order of the grains. RMS orders by level, zero crossings by sign changes, and variance by the spread of sample values. How each ordering sounds depends on the recording.

ascending = true puts the lowest measurement first. false puts the highest first. That one switch reverses the whole arc.

The sort runs on the CPU by default. Set gpu_sort_threshold to a grain count, and any call with at least that many grains sorts on the GPU instead:

Granular::GranularConfig { .grain_size = 512, .hop_size = 256, .feature_key = "rms", .gpu_sort_threshold = 1024 }

A minute of audio cut into grains of 256 frames is more than ten thousand of them, which is where the GPU path pays for itself. The default of 0 means never.

Concatenative output (CONTAINER, STREAM) places the grains end to end, with no overlap. hop_size then only decides which parts of the recording are cut, and a hop smaller than the grain repeats material. This is the first block above: tenth-second grains, each used once, in a new order.

Overlap-add output (CONTAINER_ADDITIVE, STREAM_ADDITIVE) places grain number i at i times hop_size and adds the grains together where they overlap. The second and third blocks use it. The grain taper decides how the overlaps sound:

• A Hann window, Kinesis::Discrete::apply_hann, fades each grain in and out smoothly, reducing abrupt edges at the overlaps
• A trapezoid, Kinesis::Discrete::apply_trapezoid(grain, grain.size() / 8), keeps the body of each grain and fades only the edges, which keeps more of the grain’s own character
• No taper at all sums the grains raw. Where they overlap the level builds up, which is what makes the third block a saturated cloud rather than a smooth one

With a hop of a quarter of the grain, as in the second block, each moment of the output is four grains laid over each other.

process_to_container returns a SoundFileContainer, the same kind of thing vega.read_audio() gives you. | Audio plays it from start to end, exactly as in the first card.

process_to_stream returns a DynamicSoundStream, the same kind of thing the samplers of the last card read from. That makes everything from that card available to the result:

auto stream = Granular::process_to_stream(
    sound, AnalysisType::FEATURE,
    Granular::GranularConfig { .grain_size = grain, .hop_size = grain, .feature_key = "rms" },
    "rms");

auto sampler = store(create_sampler_from_stream(stream, 0));
sampler->play_continuous(0, sampler->slice_from_stream());

create_sampler_from_stream takes the stream you already have, so no file path is needed. Loop it, read it at any speed or backwards, or give a tap set several readings at once. The next section does that.

Every field of GranularConfig, with its default:

• grain_size = 1024, frames per grain
• hop_size = 512, frames between the starts of grains
• feature_key = "feature", the name the measurement is written under
• channel = 0, which channel is measured
• ascending = true, the order
• gpu_sort_threshold = 0, grain count at which the sort moves to the GPU, 0 for never
• attribution_context = ComputationContext::SPECTRAL, the context used to select the attribute rule
• taper, empty by default, which means no taper

Fields you leave out keep their defaults. Fields you do name have to be written in the order above.

The three first steps are rules in a grammar, not fixed code, and each can be replaced on its own while the others carry on. A rule that cuts at every rising zero crossing, instead of at fixed hops, gives grains that begin where the waveform crosses zero. A second measurement rule can add a roughness number to every grain alongside the first, to use later or to order by.

It is where the workflow stops being a function call and becomes a place to compose: the segmentation, the measure and the order are the three decisions of a piece, and each can be anything.

Segmenting and measuring a long recording takes time, and a call that returns a recording makes you wait for it. The asynchronous forms run in the background and hand you the result when it is ready, so something else can play meanwhile:

#include "MayaFlux/WorkflowIncludes.hpp"

void compose() {
    auto sound = vega.read_audio();
    auto window = create_window({ "Granular", 800, 600 });
    window->show();

    auto config = Granular::GranularConfig { .grain_size = 4800, .hop_size = 4800, .feature_key = "rms" };
    auto matrix = store(Granular::make_granular_matrix(ComputationContext::SPECTRAL, Granular::GranularOutput::STREAM));

    struct Playback {
        std::shared_ptr<Kakshya::DynamicSoundStream> stream;
        std::shared_ptr<Kriya::SamplingPipeline> sampler;
    };
    auto playback = std::make_shared<Playback>();

    Granular::process_to_stream_async(
        matrix, sound, AnalysisType::FEATURE,
        [playback](std::shared_ptr<Kakshya::DynamicSoundStream> stream) {
            playback->stream = std::move(stream);
        },
        config, "rms");

    sound | Audio;

    on_key_pressed(window, IO::Keys::Q, [playback] {
        if (!playback->stream) return;
        if (!playback->sampler) playback->sampler = create_sampler_from_stream(playback->stream);
        playback->sampler->play(0, playback->sampler->slice_from_stream());
    });
}

The original plays at once. The worker prepares the rearranged stream in the background. Press Q after it is ready to make a sampler and play it; before then, Q does nothing. The key callback keeps the sampler for later presses.

A fragment shader runs once for every pixel of the window. It receives fragTexCoord, the position of that pixel across the picture from 0 to 1, and it returns a colour. The line that matters is:

outColor = texture(texSampler, uv);

texture reads the picture at uv. If uv is fragTexCoord, you get the picture as it is. Add something to uv before the read and each pixel borrows its colour from a different place.

The file name in .fragment_shader is looked up when the program starts. A name ending in .frag is compiled from source then, with no extra build step, and a name ending in .spv is loaded as already compiled. The search looks beside where you run the program in shaders and data/shaders, and the same two names one folder up. A project made from the Weave template keeps its shaders in data/shaders. A name that is found nowhere is logged as an error and the shader is not built, which is how a blank window usually starts.

vega.read_image() returns a picture that is loaded but not yet on screen. | Graphics registers it, and setup_rendering says where and how to draw it. Both calls matter, in this order:

auto picture = vega.read_image() | Graphics;
picture->setup_rendering({ .target_window = window, .fragment_shader = "audio_rows.frag" });

target_window is the window to draw into. fragment_shader replaces the default shader, which draws the picture as it is. Leave it out and you have the first section again.

When your own shader reads extra data that you attach yourself, as the next expansion does, attach it between these two lines. setup_rendering lets the picture start drawing, so the first frame should already know everything the shader reads.

read_image() loads the whole image by default; pass an optional Kakshya::Region as its second argument when you want a bounded pixel rectangle instead.

A storage buffer is a list of numbers that the shader can read at any position. bind_audio_buffer makes one from a block of sound:

Buffers::ShaderConfig config { "audio_rows.frag" };
config.bindings["audio_data"] = Buffers::ShaderBinding(1, 0, Portal::Graphics::DescriptorRole::STORAGE);

auto reader = create_processor<Buffers::DescriptorBindingsProcessor>(picture, config);
reader->bind_audio_buffer("audio", sampler->get_buffer(), "audio_data", 1, Portal::Graphics::DescriptorRole::STORAGE);

Three names are in play, and they are easy to mix up:

• "audio_data" is the key in config.bindings. It declares which set and binding the shader reads, here set 1 and binding 0, and the shader says the same: layout(set = 1, binding = 0)
• "audio" is the name you give this binding, in case you want to remove it later
• Set 0 belongs to the picture itself. Your own data goes in set 1

Every time the picture is drawn, the latest block of the sound is converted from doubles to floats and copied to the GPU. The shader asks the buffer how long it is with samples.length(), and readonly tells it that it will not write. The block is the buffer size of the engine, 512 samples unless you changed it.

DescriptorRole::STORAGE suits a list that can be any length. DescriptorRole::UNIFORM is for a small fixed block of numbers.

bind_audio_buffer takes any audio buffer. The sampler’s buffer, sampler->get_buffer(), is the block it is playing right now. A microphone listener, create_input_listener_buffer(0), is the block that just arrived. The shader does not know the difference.

A network can be bound too, with bind_network. A resonator, a bell or a string that you have sent to the speakers with | Audio already produces a block of sound every cycle, and that block goes to the shader:

reader->bind_network("bell", bell, "audio_data", 1, Portal::Graphics::DescriptorRole::STORAGE);

A network that produces neither sound nor geometry cannot be bound, and bind_network says so in the log.

A push constant is a handful of numbers sent with every draw, small and fast. The shader declares them, and your program declares the same layout:

layout(push_constant) uniform Push {
    float level;
};

struct Push {
    float level;
};

renderer->set_push_constant_size<Push>();
renderer->feed("level", source, offsetof(Push, level), sizeof(float));

feed takes a function that is called once every frame. Whatever it returns is written at offsetof(Push, level). Return a double and it is stored as a float. Only four or eight bytes are written, so each field gets its own feed. If you add float tilt; to both the shader and C++ Push blocks, it gets a second feed:

renderer->feed("level", [block]() { return mean_level(block); }, offsetof(Push, level), sizeof(float));
renderer->feed("tilt", [state]() { return state->tilt; }, offsetof(Push, tilt), sizeof(float));

Anything that can be a number works as the source: the level of a block, a value a key press changed, the output of a node. Use push constants for a few numbers, and a storage buffer when the shader needs the whole shape.

The C++ is the same in every case, and the shader decides what the sound means. Three readings:

By position. The picture’s height is the block’s time. This is the first block:

int row = clamp(int(fragTexCoord.y * float(n)), 0, n - 1);
uv.x += samples[row] * 0.25;

A few points as strengths. Take eight evenly spaced samples and let each set the strength of one sine wave across the picture, a slow wave to a fast one. The sound’s shape becomes a bend with eight layers:

for (int i = 0; i < 8; ++i) {
    float freq = float(i + 1) * 3.14159;
    shift += samples[i * (n / 8)] * sin(uv.y * freq + float(i));
}

Cells with a mind of their own. Cut the picture into a grid. Give each cell its own offset from a hash of its position, and let the sound decide how far the offsets reach and how fine the grid is:

float grid_n = mix(4.0, 14.0, clamp(energy, 0.0, 1.0));
vec2 cell = floor(uv * grid_n);
vec2 cell_rand = hash22(cell) * 2.0 - 1.0;
uv += cell_rand * (0.02 + abs(cell_energy) * 0.12);

hash22 is a small function that turns a cell’s position into two numbers that look random. Here energy is the whole block’s level and cell_energy is a level chosen for that cell; a complete shader computes both. A fuller version also tears some cells off to random places when a strike arrives.

Everything above leaves the picture alone and changes where it is read. A compute shader can instead write a new picture. This one blends a stack of pictures into one, with a weight for each:

#include "MayaFlux/Yantra/Executors/TextureExecutionContext.hpp"

using namespace MayaFlux::Yantra;
using namespace MayaFlux::Portal::Graphics;

void compose() {
    auto window = create_window({ "Morph", 1280, 720 });
    window->show();

    auto square = Kakshya::Region::image_rect(0, 0, 511, 511);
    auto a = IO::ImageReader::load(std::string("path/to/a.png"), 4, square);
    auto b = IO::ImageReader::load(std::string("path/to/b.png"), 4, square);
    auto c = IO::ImageReader::load(std::string("path/to/c.png"), 4, square);

    const uint32_t width = a->width;
    const uint32_t height = a->height;
    const size_t layer_bytes = a->byte_size();

    std::vector<uint8_t> stack(layer_bytes * 3);
    std::memcpy(stack.data(), a->data(), layer_bytes);
    std::memcpy(stack.data() + layer_bytes, b->data(), layer_bytes);
    std::memcpy(stack.data() + layer_bytes * 2, c->data(), layer_bytes);

    auto& loom = Portal::Graphics::get_texture_manager();
    auto array = loom.create_2d_array(width, height, 3, ImageFormat::RGBA8, stack.data());

    auto layers = std::make_shared<Kakshya::TextureContainer>(width, height, ImageFormat::RGBA8, 3);
    layers->from_image_array(array);

    struct MorphPC {
        uint32_t layer_count;
        uint32_t normalise;
        uint32_t layer_offset;
        uint32_t pad;
    };

    const std::vector<GpuBufferBinding> weights_binding = {
        { .set = 0, .binding = 2, .direction = GpuBufferBinding::Direction::INPUT, .element_type = GpuBufferBinding::ElementType::FLOAT32 },
    };

    auto morph = std::make_shared<TextureExecutionContext>(
        GpuComputeConfig { "texture_array_morph.comp", { 16, 16, 1 }, sizeof(MorphPC) },
        ImageFormat::RGBA8, TextureExecutionContext::OutputMode::IMAGE, 1, weights_binding);

    morph->set_push_constants(MorphPC { .layer_count = 3, .normalise = 1, .layer_offset = 0, .pad = 0 });
    morph->set_binding_data(2, std::vector<float> { 1.0F, 0.2F, 0.6F });

    DataIO input;
    input.container = std::static_pointer_cast<Kakshya::SignalSourceContainer>(layers);
    morph->execute(input, {});

    auto result = vega.TextureBuffer(width, height, ImageFormat::RGBA8) | Graphics;
    result->setup_rendering({ .target_window = window });
    result->set_gpu_texture(morph->get_output_image(0));
}

Choose three pictures at least 512×512 pixels. The reader takes the same 512×512 square from each, so every layer has the same size. To use a larger area, increase the crop’s inclusive end coordinates to fit all three pictures; image_rect(0, 0, 1023, 767) takes 1024×768 pixels when each picture is at least that large. The shader reads one weight per layer from a list of floats and divides by their total, so the weights are proportions and not fixed amounts. Here the first picture is strongest, the third is at about half of it, and the second is faint. Change the numbers and run again to see a different blend.

The weights are just a list of numbers. They could as easily be the energy of a few bands of a sound as three constants.