Week 13: Final Project Update

More progress this week! Here are the highlights:

Better Base Code
As I shared updates with classmates on the floor, including my uncertainty with how deal with noisy sensors and erratic BPM returns, I got a tip that fellow peers, Jim and Camilla, were using the same pulse sensor. Jim had already written his own beat detection algorithm, but after I sent him the link to the Pulse Sensor Github repository, he did some digging and discovered some newer code that was much more organized in the Pulse Sensor Playground. After I downloaded added the library into Arduino IDE (instructions here), I tested the example TwoPulseSensors_On_OneArduino. (All examples here.) Unlike the previous version, this one bundled different tasks into separate classes, including the pulse detection and serial output, and it was by far significantly better! I noticed way less noise with uncovered sensors (even though I’ll still keep them covered when not in use), much more stable reporting of BPMs (even and including when both sensors where in use), and hardware interrupts disabled by default (the only option in the previous version; with them enabled, I found much more noise and erratic pulse detection). Jim, you're a lifesaver!

Pulse Ellipses to Individual Beats instead of BPMs
I noticed in the original Processing sketch that the "hearts" beat according to the calculated heart rates and not individual pulses. This didn't seem intuitive. 

I identified in the Arduino sketch where to add a new serial output event*, 

 if (pulseSensor.sawStartOfBeat(i)) {
    pulseSensor.outputBeat(i);
}

...but I wasn’t sure how to handle the data output of two sensors as an array. But with the help of ITP Resident, Aarón Montoya-Moraga, I saw it was much easier than I expected. After testing each one we separately, we stored each detected pulse in the string variable, eachBeat, along with the letter, x. Before updating the Processing sketch, and using our own pulses to verify, we checked the Arduino Serial Monitor to ensure that we saw a running list of x0s and x1s for each of our pulses.

 for (int i = 0; i < PULSE_SENSOR_COUNT; ++i) {
        if (pulseSensor.sawStartOfBeat(i)) {
          pulseSensor.outputBeat(i);
          String eachBeat = "x" + String(i);
          Serial.println(eachBeat);
        }
      }

Next, it was time to update the Processing sketch, which already parsed data for the raw serial data, the BPM, and IBI for each sensor. In it’s original form, a heart variable (which I changed to beat) is set for each input of BPM. Before all this incoming data, we added lines to look for the incoming x0s and x1s and the start the beat[i] timers. now commented out from the BPM data:

void serialEvent(Serial port) {
  try {
    String inData = port.readStringUntil('\n');
    inData = trim(inData);                 

    if (inData.charAt(0) == 'x') {
      if (inData.charAt(1) == '0') {            
        println("x0");
        beat[0] = 20;                           
      }
      if (inData.charAt(1) == '1') {            
        println("x1");
        beat[1] = 20;                            
      }
    } else {

      for (int i=0; i<numSensors; i++) {
        if (inData.charAt(0) == 'a'+i) {           
          inData = inData.substring(1);           
          Sensor[i] = int(inData); 
          //beat[i] = 20;
        }
        if (inData.charAt(0) == 'A'+i) {           
          inData = inData.substring(1);           
          BPM[i] = int(inData);                   
          heartRate[i] = 100;                            
        }
        //if (inData.charAt(0) == 'M'+i) {             
        //  inData = inData.substring(1);           
        //  IBI[i] = int(inData);                   
        //}
      }
    }
  }
  catch(Exception e) {
    // print("Serial Error: ");
    // println(e.toString());
  }
}

Reverse a Pulse WaveForm
(From here down, I discuss my Processing sketch exclusively.) Having decided to seat participants across from one another, I wanted to reverse one of the pulse waveforms such that each would appear to emanate from its nearest person. 

After determining that these lines dictated the that part of the sketch to draw the lines from left to right:

   for (int x = 1; x < width-1; x++) {
      vertex(x+10, ScaledPPG[i][x]);                    
    }

…we duplicated a second version to have one for each sensor, such that for i = 0 (for the person seated to the left), the line switched directions to also start from the left instead of the right.

 for (int x = 1; x < width-1; x++) {
      vertex(x+0, ScaledPPG[i][width - x]);                    /
    }

Only Display BPMs (Beats Per Minute) when Pulses are Detected
I set a timer value for every time BPM data was detected in the serialEvent for each sensor. Every time that information arrives in the serial port the timer, heartRate is set to 100, the same as the frame rate. In the Draw() loop, the function printBPMs is called, and when it is, that timer is decreases by one each time Draw() runs. If no there are no fingers on the sensors, then this is no pulse to detect, and hence no BPMs to calculate. Without those new calculations, heartRate will continue to decrease to less than zero unless new data arrives to reset the timer to 100. If heartRate drops below zero, then "0 BPM" will display for that sensor. I also color-coded the BPMs to match its sensor each sensor, and moved one closer to its matching pulse waveform.

Move Ellipses in Relation to their BPMs
This was a big one! My playtesting was lacking last week; I asked folks to imagine the ellipses moving closer or apart based the difference in their respective BPMs. I used the heartRate timer I just created to manage this as I only wanted the ellipses to move when BPM information was detected. For this I calculated the absolute value of the difference between the two BPMs, and for each ellipse, mapped that distance to range between its starting point and the center of the screen. The x-coordinate of the ellipse would change depending on the absolute value, causing the ellipse to be redrawn at the new location. Then, I figured out to use the lerp function to smooth out that animation. Lerp!

Change Color Mode to HSB
Processing's default color mode is RGB, and each color is determined by the amount of three values: red (R), green (G), and blue (B). I really want to provide participants with an opportunity to change the color of their heart data, but providing three dials for each person seems excessive. Plus, I don't have six extra analog pins on my Arduino (it's possible, but I'm running out of time to learn it right now). After reading up on the HSB color mode for hue, saturation, and brightness here and here, I realized that I could potentially provide each participant with a slide potentiometer to change their heart color via the hue value--there are 360 potential options! My Processing sketch looks exactly the same but now it's color is generated via HSB. Now to figure out how to incorporate two more inputs of analog data into my already busy serial communication...

Add Sliders to Alter Colors
...and I did with the great help of ITP Resident, Lisa Jamhoury. To start, we added this to the start of the loop function in my Arduino sketch, just before all of the calls to capture and send the pulse sensor data to serial. Pot readings are only sent if the values are new (otherwise the same values would be sent over and over).

  int pot1 = analogRead(A2);
  int pot2 = analogRead(A3);

  String tempPotReading1 = "pa" + String(pot1);
  if (tempPotReading1 != potReading1) {
    potReading1 = tempPotReading1;
    Serial.println(potReading1);  
  }
  
  String tempPotReading2 = "pb" + String(pot2);
  if (tempPotReading2 != potReading2) {
    potReading2 = tempPotReading2;
    Serial.println(potReading2);  
  }

And to handle the additional incoming information, the start of the serialEvent in Processing now looks like this:

void serialEvent(Serial port) {
  try {
    String inData = port.readStringUntil('\n');
    inData = trim(inData);                      

    if (inData.charAt(0) == 'p') {                 
      if (inData.charAt(1) == 'a') {              
        String potReading = inData.substring(2); 
        int potReadingInt = int(potReading);    
        heartNew0 = int(map(potReadingInt, 0, 1023, 0, 360)); 
      }
      if (inData.charAt(1) == 'b') {            
        String potReading = inData.substring(2); 
        int potReadingInt = int(potReading);       
        heartNew1 = int(map(potReadingInt, 0, 1023, 0, 360)); 
      }

Update Code with Programmatic Variables
I'm not sure if this is right lingo, but throughout this entire process, I've been replacing hard-coded numbers with variables for the sketch to run on any size screen at full screen. It had previously been written to run in a specific small-sized window.

Even More Playtesting
As always, my peers provided valuable feedback in our morning Physical Computing class, all of it concerning the visuals, on which I wholeheartedly agreed: consider the placement and alignment of BPM text for each participant, consider adding additional effects when heart rates sync and ellipses are layered on top of one another in the center of the screen, and consider playing with the saturation, brightness, and alpha values of the ellipses as they are moving closer or farther away from one another. Looking forward to more suggestions and questions in tomorrow's Computational Media class!

*I'll post the entirety of my sketches with the final documentation.