Physical Computing

Weeks 14 & 15: Heart Sync

Project Description
Visualize your heartbeats and try to sync them with others! Heart Sync pictures the heartbeats of two people and encourages them to match their heart rates. Participants may also adjust the color of their pulse information to co-create the shared color field between them. Depending on their moods or personalities, it's either a calming, meditative moment or a playful, competitive game for two. 

Materials, Tools, and Process
Frequent playtesting at ITP provided valuable feedback, and since Week 13 I replaced the knob potentiometers with sliders, programmed the ellipses to pulse, and added rings to emerge if the beats per minutes (BPMs) aligned. I also built an enclosure for all the components. Detailed descriptions of all follows.

The Physical Components

FritzingHeartSync.png

1 Laptop
1 Eizo ColorEdge CG222W Display (or another monitor that lies flat)
1 Arduino Uno
1 Half-size Breadboard
2x Pulse Sensors
2x 10K Ohm Slide Potentiometers
2x Slide Potentiometer Knobs
2x Phillips Rounded Head Screws, M3 x 0.5 mm Thread, 6 mm Long
Stranded Core Wire
Soldering Iron
Solder
Heat Shrink Tubing
Black Spike Tape 1/2" Thick

The physical guts of the project are really quite few. The pictures above show how the sensors attach to the Arduino (schematic created with Fritzing) and the hand panels. I soldered the slide potentiometers to stranded core wire long enough to stretch from the panels to the microcontroller which, when all assembled within the final enclosure, tucks into the most perfect opening underneath the monitor near the monitor's power and VGA ports.

01-171203IMG_8098.jpg

2x 9x12" Clear Acrylic 1/8" Thick Sheets
2x Round Stickers 5/16" Diameter
Blue Painter's Tape
Black Spray Paint Matte
White Spray Paint Glossy
Digital Caliper
Adobe Illustrator
Laser Cutter
Heartbeat Icon (by Deemak Daksina S from the Noun Project)

For the hand panels I started with clear acrylic sheets. For each one I measured the location of the pulse sensor and covered that area with a round sticker. Then, I sprayed the entire sheet with black spray paint. Once dried, I covered the panel with blue painter's tape and headed to the 75 Watt Epilog laser cutter. There I cut the mounting holes and an opening for the slide potentiometer (four passes at speed 15, power 100, frequency 100) and etched away the blue tape and black spray paint to reveal the hand print and heart pulse icon (four-five passes at speed 100, power 40). Next, I sprayed the panel with glossy white spray paint and once dry, removed the blue painter's tape. I also removed the round sticker, revealing a clear spot under which to attach the pulse sensor. 

30 x 40" Black Foam Board 3/16" Thick Sheet
30 x 40" Black Foam Board 1/2"Thick Sheet
Utility Knife (and many fresh blades)
12" and 24" Metal Rulers
Cutting Mat
Paper
Hot Glue Gun
Glue Sticks

After determining the length of the monitor and my hand panels I measured and first cut the four sides of the box from the 1/2" thick foam board. Then, as practice for the top and before using the thinner foam board, I measured and cut a piece of white paper. The paper template helped me to determined the spacing I wanted between the hand panels, the monitor, and the surrounding border edge. With that set, I measured and cut the top from the 3/16" thick foam board. Next, I cut supports for the hand panels to run parallel to the monitor; instead of securing the panels to the enclosure they will rest on top of these supports. (Later I filled the entirety of these cavities with foam board for extra support.) After cutting out a notch on one of the long sides to allow for the monitor and Arduino cables to exit, I finally glued everything together for a very snug fit around the monitor.

The Code (GitHub)
Arduino Sketch 
As noted in my previous project post, for I modified the original sketch, TwoPulseSensors_On_OneArduino, from the Pulse Sensor Playground. This sketch provided crucial pieces of information via asynchronous serial communication for my Processing sketch from each pulse sensor: the raw signal data, the heart rate or beats per minutes (BPMs), and the moment of each beat. The latter I added myself to increase the size (strokeWeight) of each "heart" visualization and to create essential timers within my Processing sketch. I also added code to capture changing values from two slide potentiometers; in Processing, I mapped these changing values to a sliding range of colors such that each participant might change the color of their pulse information. Finally, I commented out the code to light and and pulse LEDs on the breadboard. They were useful during the preliminary testing phases of the project, but no longer so.

Processing Sketch 
My Processing sketch also modifies an original sketch by Joel Murphy, PulseSensorAmped_2_Sensors, and visualized the incoming pulse sensor data from the Arduino in several ways, some of which I detailed in earlier posts. Here's a summary of everything.

From each sensor, the raw signal data is used to draw pulse waveforms across the screen. As well, it is mapped to the radii of ellipses for each participant's "heart." This is a change from Week 13, and the result is that the ellipses appear to "breathe" as they pulse (instead of turning on and off abruptly due to changes in strokeWeight only).

The moment of each beat is still used to increase the strokeWeight of the ellipses in each heart. In the original code, though, this moment was set by the BPM value, which seemed off. However, I learned from this portion of the sketch how to set timers for events elsewhere in my sketch. For example, I also use beat detection from each sensor to determine if the ellipses (hearts) should move across the screen. If the program senses BPM values from each user, then it compares the rates and moves the respective hearts closer or farther away from one another: the smaller the difference in rates, the closer hearts to the center of the screen. The moment of syncing actually occurs within a threshold BPM difference of 0-3. (At one point I also used the BPM timers to display BPM values on the screen while testing the above but since removed that data in favor of a cleaner and less game-like display.)

For the changing radii and x-coordinate locations of the ellipses, I continued to play with the lerp function to create smoother transitions within the animations.  

As I mentioned above, I mapped the incoming values from the slide potentiometers to changes in hue to allow each participant to alter the color of their pulse information. 

Finally, I wished for participants to received a stronger, more gratifying visual indicator at the moment of syncing. I envisioned rings emerging at that moment and learned that I could achieve this with a simple particle system. Thanks to Dan Shiffman for pointing me in that direction and for his resources available here, here, and here. Now when the ellipses align in the center, concentric rings ripple and fade out across the screen. These rings are actually ellipses themselves but with no fill and thick strokeWeights. Over time their radii increase as their strokeWeight and alpha values decrease, eventually disappearing altogether.

Conclusions
What did I learn? Wow, I learned SO MUCH this semester from how to work with sensors and output data, to how to program more extensively in P5 and Processing, to how to fabricate enclosures that communicate the functionality of my projects. Along the way I tried to note major progress updates on my blog. It's impossible to capture everything, but here are some highlights.

I learned to work with the pulse sensors by breaking down the code in both the existing Arduino and Processing sketches. It was difficult, and it took a long time. But I as stayed with it, I eventually saw that the data from both sensors was wrapped up in an array. As I worked out the visualizations in Processing, I uncoupled each sensor from this array because it was easier for me to visualize the variables as separate entities and consider how I wanted them to interact. Along the way, I learned more about syntax and how to use new functions and create my own. I tried to organize my code at the end (and where I could, re-wrap the sensor data back into an array), but it still feels quite messy to me. However, so was my learning throughout this process. I need to keep practicing to to communicate my ideas succinctly in this medium. (After all, it's only been 15 weeks since school started!)

More importantly I discovered that I really enjoy creating open-ended, collaborative, and playful experiences for people in which participant interaction is the content. I'm interested in the conversations and relationships that are initiated or supported by the technology. In this regard, Heart Sync is similar to my Social Drawing Sol LeWitt shared drawing canvas and to some extent, to my Multiples of Five Modular Sculpture, and I'm excited to explore these ideas further at ITP. 

The most fun aspect of realizing this project has been watching people come up with different strategies--completely unprompted by me--to match their heart rates, including: sitting still and quietly, breathing deeply in unison, running in place, holding hands, and listening to music. Several times I've heard the word, play. Can we play?! I want to play. I had so fun much playing! These moments are the most gratifying.

Thanks!
A huge thanks to ITP faculty Ayodamola Okunseinde, Dan Shiffman, and Ben Light, and ITP residents Lisa Jamhoury, Aarón Montoya-Moraga, and Davíd Lockard, for all of their advice and technical help. And of course to all my fabulous classmates who provided ongoing invaluable feedback on the floor. I can't wait to see what we all create in the next year!

Week 13: Final Project Update

More progress this week! Here are the highlights:

Better Base Code
As I shared updates with classmates on the floor, including my uncertainty with how deal with noisy sensors and erratic BPM returns, I got a tip that fellow peers, Jim and Camilla, were using the same pulse sensor. Jim had already written his own beat detection algorithm, but after I sent him the link to the Pulse Sensor Github repository, he did some digging and discovered some newer code that was much more organized in the Pulse Sensor Playground. After I downloaded added the library into Arduino IDE (instructions here), I tested the example TwoPulseSensors_On_OneArduino. (All examples here.) Unlike the previous version, this one bundled different tasks into separate classes, including the pulse detection and serial output, and it was by far significantly better! I noticed way less noise with uncovered sensors (even though I’ll still keep them covered when not in use), much more stable reporting of BPMs (even and including when both sensors where in use), and hardware interrupts disabled by default (the only option in the previous version; with them enabled, I found much more noise and erratic pulse detection). Jim, you're a lifesaver!

Pulse Ellipses to Individual Beats instead of BPMs
I noticed in the original Processing sketch that the "hearts" beat according to the calculated heart rates and not individual pulses. This didn't seem intuitive. 

I identified in the Arduino sketch where to add a new serial output event*, 

 if (pulseSensor.sawStartOfBeat(i)) {
    pulseSensor.outputBeat(i);
}

...but I wasn’t sure how to handle the data output of two sensors as an array. But with the help of ITP Resident, Aarón Montoya-Moraga, I saw it was much easier than I expected. After testing each one we separately, we stored each detected pulse in the string variable, eachBeat, along with the letter, x. Before updating the Processing sketch, and using our own pulses to verify, we checked the Arduino Serial Monitor to ensure that we saw a running list of x0s and x1s for each of our pulses.

 for (int i = 0; i < PULSE_SENSOR_COUNT; ++i) {
        if (pulseSensor.sawStartOfBeat(i)) {
          pulseSensor.outputBeat(i);
          String eachBeat = "x" + String(i);
          Serial.println(eachBeat);
        }
      }

Next, it was time to update the Processing sketch, which already parsed data for the raw serial data, the BPM, and IBI for each sensor. In it’s original form, a heart variable (which I changed to beat) is set for each input of BPM. Before all this incoming data, we added lines to look for the incoming x0s and x1s and the start the beat[i] timers. now commented out from the BPM data:

void serialEvent(Serial port) {
  try {
    String inData = port.readStringUntil('\n');
    inData = trim(inData);                 

    if (inData.charAt(0) == 'x') {
      if (inData.charAt(1) == '0') {            
        println("x0");
        beat[0] = 20;                           
      }
      if (inData.charAt(1) == '1') {            
        println("x1");
        beat[1] = 20;                            
      }
    } else {

      for (int i=0; i<numSensors; i++) {
        if (inData.charAt(0) == 'a'+i) {           
          inData = inData.substring(1);           
          Sensor[i] = int(inData); 
          //beat[i] = 20;
        }
        if (inData.charAt(0) == 'A'+i) {           
          inData = inData.substring(1);           
          BPM[i] = int(inData);                   
          heartRate[i] = 100;                            
        }
        //if (inData.charAt(0) == 'M'+i) {             
        //  inData = inData.substring(1);           
        //  IBI[i] = int(inData);                   
        //}
      }
    }
  }
  catch(Exception e) {
    // print("Serial Error: ");
    // println(e.toString());
  }
}

Reverse a Pulse WaveForm
(From here down, I discuss my Processing sketch exclusively.) Having decided to seat participants across from one another, I wanted to reverse one of the pulse waveforms such that each would appear to emanate from its nearest person. 

After determining that these lines dictated the that part of the sketch to draw the lines from left to right:

   for (int x = 1; x < width-1; x++) {
      vertex(x+10, ScaledPPG[i][x]);                    
    }

…we duplicated a second version to have one for each sensor, such that for i = 0 (for the person seated to the left), the line switched directions to also start from the left instead of the right.

 for (int x = 1; x < width-1; x++) {
      vertex(x+0, ScaledPPG[i][width - x]);                    /
    }

Only Display BPMs (Beats Per Minute) when Pulses are Detected
I set a timer value for every time BPM data was detected in the serialEvent for each sensor. Every time that information arrives in the serial port the timer, heartRate is set to 100, the same as the frame rate. In the Draw() loop, the function printBPMs is called, and when it is, that timer is decreases by one each time Draw() runs. If no there are no fingers on the sensors, then this is no pulse to detect, and hence no BPMs to calculate. Without those new calculations, heartRate will continue to decrease to less than zero unless new data arrives to reset the timer to 100. If heartRate drops below zero, then "0 BPM" will display for that sensor. I also color-coded the BPMs to match its sensor each sensor, and moved one closer to its matching pulse waveform.

Move Ellipses in Relation to their BPMs
This was a big one! My playtesting was lacking last week; I asked folks to imagine the ellipses moving closer or apart based the difference in their respective BPMs. I used the heartRate timer I just created to manage this as I only wanted the ellipses to move when BPM information was detected. For this I calculated the absolute value of the difference between the two BPMs, and for each ellipse, mapped that distance to range between its starting point and the center of the screen. The x-coordinate of the ellipse would change depending on the absolute value, causing the ellipse to be redrawn at the new location. Then, I figured out to use the lerp function to smooth out that animation. Lerp!

Change Color Mode to HSB
Processing's default color mode is RGB, and each color is determined by the amount of three values: red (R), green (G), and blue (B). I really want to provide participants with an opportunity to change the color of their heart data, but providing three dials for each person seems excessive. Plus, I don't have six extra analog pins on my Arduino (it's possible, but I'm running out of time to learn it right now). After reading up on the HSB color mode for hue, saturation, and brightness here and here, I realized that I could potentially provide each participant with a slide potentiometer to change their heart color via the hue value--there are 360 potential options! My Processing sketch looks exactly the same but now it's color is generated via HSB. Now to figure out how to incorporate two more inputs of analog data into my already busy serial communication...

Add Sliders to Alter Colors
...and I did with the great help of ITP Resident, Lisa Jamhoury. To start, we added this to the start of the loop function in my Arduino sketch, just before all of the calls to capture and send the pulse sensor data to serial. Pot readings are only sent if the values are new (otherwise the same values would be sent over and over).

  int pot1 = analogRead(A2);
  int pot2 = analogRead(A3);

  String tempPotReading1 = "pa" + String(pot1);
  if (tempPotReading1 != potReading1) {
    potReading1 = tempPotReading1;
    Serial.println(potReading1);  
  }
  
  String tempPotReading2 = "pb" + String(pot2);
  if (tempPotReading2 != potReading2) {
    potReading2 = tempPotReading2;
    Serial.println(potReading2);  
  }

And to handle the additional incoming information, the start of the serialEvent in Processing now looks like this:

void serialEvent(Serial port) {
  try {
    String inData = port.readStringUntil('\n');
    inData = trim(inData);                      

    if (inData.charAt(0) == 'p') {                 
      if (inData.charAt(1) == 'a') {              
        String potReading = inData.substring(2); 
        int potReadingInt = int(potReading);    
        heartNew0 = int(map(potReadingInt, 0, 1023, 0, 360)); 
      }
      if (inData.charAt(1) == 'b') {            
        String potReading = inData.substring(2); 
        int potReadingInt = int(potReading);       
        heartNew1 = int(map(potReadingInt, 0, 1023, 0, 360)); 
      }

Update Code with Programmatic Variables
I'm not sure if this is right lingo, but throughout this entire process, I've been replacing hard-coded numbers with variables for the sketch to run on any size screen at full screen. It had previously been written to run in a specific small-sized window.

Even More Playtesting
As always, my peers provided valuable feedback in our morning Physical Computing class, all of it concerning the visuals, on which I wholeheartedly agreed: consider the placement and alignment of BPM text for each participant, consider adding additional effects when heart rates sync and ellipses are layered on top of one another in the center of the screen, and consider playing with the saturation, brightness, and alpha values of the ellipses as they are moving closer or farther away from one another. Looking forward to more suggestions and questions in tomorrow's Computational Media class!

*I'll post the entirety of my sketches with the final documentation.

 

 

Week 12: Final Project Progress

This week I prototyped two partial-enclosure options and dissected the code for two pulse sensors on one Arduino, posted on Github by the World Famous Electronics company. 

Enclosure Practice
With the intention of running through various live play testing sessions this week on the floor, I created a separate pad for each sensor as to arrange them in different configurations. 

Learning the Code
With improvements to the sensor housing, it was easy to run multiple tests over, and over, and over, as I dove into the code. I had two sketches at my disposal: a sketch for the Arduino Uno and one for Processing. 

When powered, these sensors, also known as photoplethysmographs, emit a bright green light and record changes in the reflected light when blood moves through the skin. According to the company's website, "the heart pulse signal is an analog fluctuation in voltage....[The] Pulse Sensor Amped responds to relative changes in light intensity. If the amount of light incident on the sensor remains constant, the [raw] signal value will remain at (or close to) 512 (midpoint of ADC range). More light and the signal goes up. Less light, the opposite. Light from the green LED that is reflected back to the sensor changes during each pulse."

The Arduino sketch reads the incoming sensor data, and for each one, determines individual heart beats and the time between the beats, also known as Inter Beat Interval (IBI), and from that calculates the heart rate or beats per minute (BPM). The sketch sends the raw analog signals (0-1023), as well as the BMPs and IBIs, to the serial port for visualization with a Processing sketch.

Screen Shot 2017-11-19 at 4.36.52 PM.png

This Processing sketch draws the pulses' wave forms (in the large windows on the left) and the heart rates (smaller windows to the right), an animates the illustrated hearts with every beat. Because the code handles incoming and outgoing data for two sensors, all of the code in both the Arduino and Processing sketches is organized into in an array of two objects. (I used both hands, one on each sensor, to generate the equal readings above. The bottom one starts to go irregular as I moved my hand away to take the screenshot.)

prototype1e.gif

My first dive into code started by manipulated the existing Processing sketch. Once I removed the wave forms, I realized that they had provided some initial feedback upon placing my fingers over the sensors. Notice above that the hearts, which I changed to ellipses, do not immediately start pulsing. In fact, "the BPM is derived every beat from an average of the previous 10 IBI times."  This page walks through that process. I also observed that it took extra time for the heart beats to sync with one another. Interesting that the waveforms start drawing before the ellipses begin pulsing--I believe the former is drawing the raw signal data and the latter only once BPMs are calculated. Can I isolate the individual heart pulse variable and send that through serial communication to beat the ellipses sooner? 

Note: Be sure to incorporate initial visual feedback upon engagement with the sensors. What instructions or information might I provide so users except a short wait time before their heart rates are visible?

Next I familiarized myself with the Arduino sketch, and I noticed for the first time quite a bit of noisy data. When uncovered, both sensors sent out data continuously--usually around 150-200+ beats per minute, jumping around sporadically. This also happened when I connected just one sensor to the Arduino. Even when fingers are placed over the sensors (and stay still), there's still quite a bit of fluctuation in the readings.

Again, no hands on the sensors, but this time viewed through the provided Processing visualizer. Notice the noisy pulse waveforms and 200 BPMs.

I found a few other complaints of this online, though some of these posts were published before the current releases of the sketches that I'm using. My notes:

Here is where I learned about the concept of noise: "If you see values when there is no finger, then that is noise."

A comment in this post mentions adding "an if statement to check if BPM is within a certain range(40-100)."

This post from waaay back mentions the threshold variable, which is a global variable initialized at 530 in my sketch: "I did a bit more testing and found that I have better results with a default threshold value of about 475. But it is still pretty sensitive to placement. The code does not seem to handle variations in the signal level very well." In the Pulse Sensor Start Project sketch, threshold is defined as a value to "determine which Signal to "count as a beat", and which to ignore." I noticed in my Arduino sketch, under Interrupts, that threshold is set by the variable, thresh. According to the Arduino Code v1.2 Walkthrough, "The thresh variable is initialized at 512 (middle of analog range) and changes during run time to track a point at 50% of amplitude as we will see later." (Increasing or decreasing this value to 800 and 300 respectively did not change my results, if anything it made it harder or easier to register my heart beats. How can l check on making sure the BPMs are within a certain range as suggested above?)

Here I found, "There isn't very much noise reduction in the Pulse Sensor Amped code. So it is reacting to small noise fluctuations in the ambient light when it's not touching your finger. To fix this, you will need to add some noise cancelling code. There is a variable, called 'amp' in the interrupt code. That variable is updated every time the Pulse Sensor finds a beat. You can use a threshold on that value to limit the size of the 'pulse' signal that you want the sensor to react to." (Same as above. I think the point is that there will always be noise, even at the default settings, and I need to account for that.)

Here is a mention of,  "using a laptop and have the power cord connected...could be adding a bunch of noise to the USB power, which you could see in the signal at 5V, the 3.3V power is a bit cleaner, because the board is regulating it." (I followed the most recent directions to power my sensors at 5V. Switching to 3.3V returned a noisy pulse wave and no beat data whatsoever. Also I did not notice a difference with my laptop's power cord connected.)

I also saw mentions here and there of folks user software interrupts with timers to sample the pulse sensor at regular rates instead of the hardware interrupts. Maybe something to consider going forward?

Sooo...all the noise disappeared when I turned off my bright fluorescent desk lamp, shining directly down into the sensors. But I'm still getting visual outputs in my redesigned Processing sketch (above) that I don't understand, including a noisy readouts when only one sensor is covered and mismatched BPMs when I place both of my hands on the sensors. I also observed changing values depending on how firmly I pressed my fingers on the acrylic over the sensors.

More Playtesting

Curious to understand the impact of two different installation configurations, I checked out a monitor from the Equipment Room and set up in the corner of the lounge where I received a fair amount of feedback throughout the day. (Thanks to Ben Light for suggestion to keep playtesting!) First off, folks were drawn to the handprints. Their curiosities piqued, they immediately placed their hands on the sensor pads without any direction, and sat much longer than I expected watching the hypnotic pulsing ellipses and waveforms. (Some noted the matching medical vibe of the old school NEC monitor.) Originally, I anticipated installing some time of finger strap for proper finger placement, but I observed no apparent issues. I was thankful for that initial feedback of the pulse waveforms to keep their interest. Some testers were mesmerized by the visuals, while others wanted some instructions or a goal after a few moments. 

I tested the sensors across from one another, with the monitor laying flat in the middle, as well as next to one another with the monitor upright. It was generally agreed in my conversations with various participants that the two setups felt very different from one another. The monitor upright and forward felt very familiar, like watching television with someone. While it was fun to consider the visual representation of internal processes, the screen dominated the focus of the activity. With the monitor lying flat, the focal point of the experience became the person sitting directly across from you. Instead of making contact with the screen, we were making conversation with one another about what were seeing and how it was changing. The visuals were merely a starting point.

I also learned, rather by accident, that covering the sensors with white index cards completely stopped random data spikes (and pulses) when not in use. Those of us in the area decided it was environmental; these pulse sensors are light sensors after all.

When I started this project I had some vague ideas of wanting to explore interactions between people, as well as using normally unseen biometric data. After all the helpful comments and critical questions from the floor (thanks again to ITP Resident, Lisa Jamhoury), my head was spinning at the end of the day: What exactly is this project about?!  I responded well to the setup in which people sat across from one another, and words such as, "exploring", "creating", and "playing together". After all the input, I decided to focus the activity as a challenge to sync your heart rate with another person. Can two people even do this? This could be the seeds of a much more involved project about intimacy, but it's hard for me to flush the idea out beyond just a light and fun game at the moment. Perhaps I'll suggest some prompts to help folks who are game to try.