Week 12: Final Project Progress

This week I prototyped two partial-enclosure options and dissected the code for two pulse sensors on one Arduino, posted on Github by the World Famous Electronics company. 

Enclosure Practice
With the intention of running through various live play testing sessions this week on the floor, I created a separate pad for each sensor as to arrange them in different configurations. 

Learning the Code
With improvements to the sensor housing, it was easy to run multiple tests over, and over, and over, as I dove into the code. I had two sketches at my disposal: a sketch for the Arduino Uno and one for Processing. 

When powered, these sensors, also known as photoplethysmographs, emit a bright green light and record changes in the reflected light when blood moves through the skin. According to the company's website, "the heart pulse signal is an analog fluctuation in voltage....[The] Pulse Sensor Amped responds to relative changes in light intensity. If the amount of light incident on the sensor remains constant, the [raw] signal value will remain at (or close to) 512 (midpoint of ADC range). More light and the signal goes up. Less light, the opposite. Light from the green LED that is reflected back to the sensor changes during each pulse."

The Arduino sketch reads the incoming sensor data, and for each one, determines individual heart beats and the time between the beats, also known as Inter Beat Interval (IBI), and from that calculates the heart rate or beats per minute (BPM). The sketch sends the raw analog signals (0-1023), as well as the BMPs and IBIs, to the serial port for visualization with a Processing sketch.

Screen Shot 2017-11-19 at 4.36.52 PM.png

This Processing sketch draws the pulses' wave forms (in the large windows on the left) and the heart rates (smaller windows to the right), an animates the illustrated hearts with every beat. Because the code handles incoming and outgoing data for two sensors, all of the code in both the Arduino and Processing sketches is organized into in an array of two objects. (I used both hands, one on each sensor, to generate the equal readings above. The bottom one starts to go irregular as I moved my hand away to take the screenshot.)

prototype1e.gif

My first dive into code started by manipulated the existing Processing sketch. Once I removed the wave forms, I realized that they had provided some initial feedback upon placing my fingers over the sensors. Notice above that the hearts, which I changed to ellipses, do not immediately start pulsing. In fact, "the BPM is derived every beat from an average of the previous 10 IBI times."  This page walks through that process. I also observed that it took extra time for the heart beats to sync with one another. Interesting that the waveforms start drawing before the ellipses begin pulsing--I believe the former is drawing the raw signal data and the latter only once BPMs are calculated. Can I isolate the individual heart pulse variable and send that through serial communication to beat the ellipses sooner? 

Note: Be sure to incorporate initial visual feedback upon engagement with the sensors. What instructions or information might I provide so users except a short wait time before their heart rates are visible?

Next I familiarized myself with the Arduino sketch, and I noticed for the first time quite a bit of noisy data. When uncovered, both sensors sent out data continuously--usually around 150-200+ beats per minute, jumping around sporadically. This also happened when I connected just one sensor to the Arduino. Even when fingers are placed over the sensors (and stay still), there's still quite a bit of fluctuation in the readings.

Again, no hands on the sensors, but this time viewed through the provided Processing visualizer. Notice the noisy pulse waveforms and 200 BPMs.

I found a few other complaints of this online, though some of these posts were published before the current releases of the sketches that I'm using. My notes:

Here is where I learned about the concept of noise: "If you see values when there is no finger, then that is noise."

A comment in this post mentions adding "an if statement to check if BPM is within a certain range(40-100)."

This post from waaay back mentions the threshold variable, which is a global variable initialized at 530 in my sketch: "I did a bit more testing and found that I have better results with a default threshold value of about 475. But it is still pretty sensitive to placement. The code does not seem to handle variations in the signal level very well." In the Pulse Sensor Start Project sketch, threshold is defined as a value to "determine which Signal to "count as a beat", and which to ignore." I noticed in my Arduino sketch, under Interrupts, that threshold is set by the variable, thresh. According to the Arduino Code v1.2 Walkthrough, "The thresh variable is initialized at 512 (middle of analog range) and changes during run time to track a point at 50% of amplitude as we will see later." (Increasing or decreasing this value to 800 and 300 respectively did not change my results, if anything it made it harder or easier to register my heart beats. How can l check on making sure the BPMs are within a certain range as suggested above?)

Here I found, "There isn't very much noise reduction in the Pulse Sensor Amped code. So it is reacting to small noise fluctuations in the ambient light when it's not touching your finger. To fix this, you will need to add some noise cancelling code. There is a variable, called 'amp' in the interrupt code. That variable is updated every time the Pulse Sensor finds a beat. You can use a threshold on that value to limit the size of the 'pulse' signal that you want the sensor to react to." (Same as above. I think the point is that there will always be noise, even at the default settings, and I need to account for that.)

Here is a mention of,  "using a laptop and have the power cord connected...could be adding a bunch of noise to the USB power, which you could see in the signal at 5V, the 3.3V power is a bit cleaner, because the board is regulating it." (I followed the most recent directions to power my sensors at 5V. Switching to 3.3V returned a noisy pulse wave and no beat data whatsoever. Also I did not notice a difference with my laptop's power cord connected.)

I also saw mentions here and there of folks user software interrupts with timers to sample the pulse sensor at regular rates instead of the hardware interrupts. Maybe something to consider going forward?

Sooo...all the noise disappeared when I turned off my bright fluorescent desk lamp, shining directly down into the sensors. But I'm still getting visual outputs in my redesigned Processing sketch (above) that I don't understand, including a noisy readouts when only one sensor is covered and mismatched BPMs when I place both of my hands on the sensors. I also observed changing values depending on how firmly I pressed my fingers on the acrylic over the sensors.

More Playtesting

Curious to understand the impact of two different installation configurations, I checked out a monitor from the Equipment Room and set up in the corner of the lounge where I received a fair amount of feedback throughout the day. (Thanks to Ben Light for suggestion to keep playtesting!) First off, folks were drawn to the handprints. Their curiosities piqued, they immediately placed their hands on the sensor pads without any direction, and sat much longer than I expected watching the hypnotic pulsing ellipses and waveforms. (Some noted the matching medical vibe of the old school NEC monitor.) Originally, I anticipated installing some time of finger strap for proper finger placement, but I observed no apparent issues. I was thankful for that initial feedback of the pulse waveforms to keep their interest. Some testers were mesmerized by the visuals, while others wanted some instructions or a goal after a few moments. 

I tested the sensors across from one another, with the monitor laying flat in the middle, as well as next to one another with the monitor upright. It was generally agreed in my conversations with various participants that the two setups felt very different from one another. The monitor upright and forward felt very familiar, like watching television with someone. While it was fun to consider the visual representation of internal processes, the screen dominated the focus of the activity. With the monitor lying flat, the focal point of the experience became the person sitting directly across from you. Instead of making contact with the screen, we were making conversation with one another about what were seeing and how it was changing. The visuals were merely a starting point.

I also learned, rather by accident, that covering the sensors with white index cards completely stopped random data spikes (and pulses) when not in use. Those of us in the area decided it was environmental; these pulse sensors are light sensors after all.

When I started this project I had some vague ideas of wanting to explore interactions between people, as well as using normally unseen biometric data. After all the helpful comments and critical questions from the floor (thanks again to ITP Resident, Lisa Jamhoury), my head was spinning at the end of the day: What exactly is this project about?!  I responded well to the setup in which people sat across from one another, and words such as, "exploring", "creating", and "playing together". After all the input, I decided to focus the activity as a challenge to sync your heart rate with another person. Can two people even do this? This could be the seeds of a much more involved project about intimacy, but it's hard for me to flush the idea out beyond just a light and fun game at the moment. Perhaps I'll suggest some prompts to help folks who are game to try.