MAX/MSP
Audio - visual interaction
2025
MAX/MSP
Dimensions Variable
In the visual module, the system collects live camera footage, processes it through brightness, contrast, and grayscale adjustments, and converts it into a single-channel dynamic matrix. Using pixel sampling (via jit.peek~), the system reads brightness values from different locations in the frame and maps them to sound-frequency parameters. As a result, the viewer’s face, movements, and changes in lighting immediately alter the pitch, rate, and rhythmic qualities of the sound, forming a structure in which image drives sound.
The sound component consists of three sine-wave voices driven by visual data, along with three additional switchable music tracks. All sounds are processed through reverb and mixed together, creating a spatial and slightly unpredictable sonic environment. Each movement from the viewer disturbs the sound field, giving the work a semi-random, semi-controlled quality.
Visually, Zhiyan Li didn’t want the piece to rely solely on the camera feed, so she added a 3D geometric interaction module. This section is built using Max’s OpenGL environment. The mouse position directly controls the spatial coordinates of three-dimensional forms—spheres, cubes, and toruses. Pressing the keys 1, 2, or 3 switches between shapes, while different key strikes trigger random color changes and scaling effects. The viewer’s position relative to the camera, combined with their hand movements, collectively shapes a dynamic and responsive 3D visual space.