Immersive Listening

Week 6: Spatial Sound Puzzle in Unreal

At the end of our last class, TK shared a very clever idea that I decided to run with a bit farther this week: he described attaching stems of Michael Jackson song to pickup cubes in the VR template. Since it’s also super fun to throw items in VR, I turned that idea into a game inside of giant cube pit of sorts.

Imagine entering a VR scene surrounded by walls of identical-looking cubes emanating sounds. Some of the sounds are similar, but not all. There are six special cubes each containing a stem of an oldie but a fun-to-dance-goodie if you can find and assemble them all. Toss the cubes away that don’t sound sense, but take care not to throw away a song stem. You’ll have really use your ears for this one. Here’s a hint: you’re listening for: vocals, drums, bass, keys, guitars, and another cube with some extra fill-in sounds. You’ll know when you find one: just bring it close and listen carefully.

This sketch picks up threads from my previous ones: in this minimal environment, sound is the key component for navigation as well as the primary motivator for engaging in the scene. But here the puzzle is to actively interact with the cacophony until you create a state of musical order.

A few notes on my process. I developed this project entirely with the Oculus Rift and touch controllers. Each pickup cube in the VR template is based off a blueprint. In order to embed a sound within a cube, I created a sound cue and attached that to the cube’s blueprint. Because I had many audio files, I created as many cube blueprints as I had sound cues. Once a cue was attached to a blueprint, I made multiple cubes with that sound. Adjusting any audio setting on the blueprint automatically applied to the related cubes. As a general practice, I checked to override attenuation in both the cue itself and on each cube’s blueprint. Though I set looping in the sound cues, I set the attenuation function in the blueprints to natural sound—I remember Michael (?) saying that with this the loop does not restart if you leave and walk back into the radius of hearing. The radii of the sounds spheres vary, but I made sure to increase those of the stems a bit to overlap on the player when the game starts—turns out that this is a hard puzzle to solve so it’s helpful to give a glimpse of the end goal.

Stems from here.

Additional credits from Freesound.org:
blip-plock-pop by onikage22
Crickets_04.wav by RSilveira_88
Cuckoo[1].mp3 by Navadaux
Ghostly Whispers by dimbark1
Mystery_chime.ogg by Unaxete
Pop!.wav by kwahmah_02
pulse echo_01.L-Joined-0001.aiff by martian

Week 5: Sound Cues in Unreal

This week introduced creating sounds with a synthesizer—a software version called VCVRack, and sound cues in Unreal. While I initially looked forward to the prospect of using a synth, I didn’t get anywhere exciting to my liking, and since we’ll be using Unreal Engine for one more week, I decided to focus on learning sound cues instead.

Sound cues are like blueprints for audio files. Whereas last week we learned how to adjust settings for individual sounds, sound cues make it extremely easy work with a group of sounds in a given location, especially if you want them to play in a sequential order, randomly, or with slight modifications. For example, you can rig a cue with multiple sounds of which only one will play at random. You can layer multiple sounds (with the mixer or concatenator palettes) for random playback when the player triggers the cue or enters into its radius. You can send sounds through a modulator to diversify pitch and volume during playback. So many options!

My guiding question for this week’s sketch was: how I can use sound to promote playful exploration of an environment? In my scene, jumping off each large stair (three giant staircases in total) triggers a goofy playback. And while it looks nothing like a fun house, that’s kinda the point: how can I synergistically pair action and sound to drive discovery despite the drab look?

Credits from Freesound.org:
blip-plock-pop by onikage22
Boing.wav by juskiddink
cartoon slide sound effect.mp3 by reasanka
Comedic Boing, A.wav by InspectorJ
Cute Question Mark by plasterbrain
jawharp_boing.wav by plingativator
Pop!.wav by kwahmah_02
Slide Whistle, Descending, B (H1).wav by InspectorJ
Wood slide whistle - cartoon fall.wav by casemundy

Week 4: Spatial Sounds in Unreal

We’re moving into sound design for virtual worlds created Unreal Engine. I spent the week exploring and acquainting myself with the interface and its blueprints. For my sketch I drew a scene with primitive objects that was originally inspired by ancient stone circles but ended up somewhere between a stone forest, a de Chirico painting, and Wolfenstein 3D sans Nazis.

My scene utilizes first person perspective and navigation takes the user through sound “hotspots”. Each sound was placed in specific positions with attenuation parameters, including the shape of the space it inhabits and dynamic volume levels that change as you move through it. A separate, non-localized background track plays consistently during play mode; I learned that is called head-locked audio. I reworked the sounds in Audacity before placing them into my landscape to create a my own sound palette, if you will. But you can hear the originals below.

The soft point lights mixed with the harsh shadows and direct light from skylight above are meant to disorient. I’m exploring how to use sound to map one’s location in the environment since the visual cues of the repeating columns are more-or-less similar. Do the sounds help you navigate to the same places or not? (headphones recommended)

Credits from Freesound.org:
Crickets_04.wav by RSilveira_88
Cuckoo[1].mp3 by Navadaux
Ghostly Whispers by dimbark1
Mystery_chime.ogg by Unaxete
pulse echo_01.L-Joined-0001.aiff by martian
Valece, komische Grillen, zirpen.WAV by VMan533