DWD Online Server

Week 6: Photo Booth Prototype

diptych_1520534798_2.png

Add yourself to the gallery here (while it lasts): Photo Booth Prototype
Code on GitHub

It all clicked into place, everything we've learned in these brief six weeks. Though not ready for an audience beyond my in-class demo, I made substantial progress on converting my photo booth into an online prototype, and I'm excited for my peer's critical feedback. In the progress, I incorporated nearly all of the web app features we practiced this semester and developed a solid understanding of how they integrate together in one environment. A Node server using the Express framework, my web app serves over HTTPS, handles GET and POST requests, uses an EJS template engine, and saves and retrieves data from a connected cloud database, making use of AJAX/JQuery methods to do so with JSON-formatted data. I also ventured out of the comfort of the P5 library to combine HTML, CSS, and JavaScript with my P5 Canvas (a HUGE step for me). To support this entire process, I learned how to use the console (of both my server and the browser) to troubleshoot code errors and print out data as I built out changes and additions. My increasing familiarity with reading and writing code in various syntaxes sharpened my online search inquiries, and I'm now good friends with Stack Overflow. In addition, I grew increasingly comfortable with git and GitHub to back up my project and deliver it to my server. 

For this most recent iteration of my project, I updated the user flow. Prior to this week, the user experience was a quick one: click the start button, see a random image, and then view the resulting diptych immediately afterwards. Curious about the code requirements to display more than one image, I modified the process and added two extra images.

Here's an outline of the current version including the behind-the-scenes tech:

Welcome
Now upon reaching the site, one is welcomed with a brief explanation that their portrait will be taken and displayed in a gallery shared with other participants. Should they choose to enter, they must press a button to position themselves in front of their computer's camera. Opening this initial page should trigger a browser message requesting access to the computer's camera, and once granted, turn it on immediately (notice the red dot in the Chrome browser tab) to prevent delays in subsequent stages of the app.

Screen Shot 2018-03-08 at 12.23.15 AM.png

Position Yourself for the Camera
A live video feed plays back to the user. Once situated within the frame they click the button to proceed. 

Screen Shot 2018-03-08 at 12.23.21 AM.png

View Images
Next, several images display in succession. Each image is titled as a number--0.jpg, 1.jpg, 2.jpg, making it easy to preload them into an image array when the app starts and call them up when needed here, in both instances with a For loop. Each image displays for 2 seconds, but at the 1.5-second mark behind the scenes a portrait is taken of the viewer (using an offscreen image buffer), that portrait is rendered into a diptych with the other image (again with an offscreen buffer), and the diptych file is saved into a directory (as described in last week's post). For this week's iteration, I figured out how to send the index number of the display image along with the diptych string to the server. Now the diptych's filename and the index of it's accompanying display image are saved in a JSON object in my connected Mongo database. This makes for easy queries and displays of files to the Gallery page. Speaking of which, back to client-side: when the user reaches the final image in the series, a button to the Gallery appears near the bottom of the screen. 

Visit the Gallery
Here users scroll through diptychs grouped by the images they just viewed. This was most technically-challenging for me to solve this week. First, I figured out how to query the database on the index/filename of the display image and draw to screen one related diptych. (Coding best practice: do one thing at a time. First get one image, then focus geting all of them.) For this I wrote a JQuery AJAX method to make a GET request with the display image's index value to a specific route on my server file which it used to query the database. The return response rendered the related dipytch to my Gallery page, an EJS template. Next, I learned how to display all of the related diptychs. I knew to use a For loop to repeat over the JSON objects, but at first I couldn't incorporate the JavaScript into my JQuery method without getting errors. It took a while, but I finally came across this Stack Overflow post which suggested generating all of the HTML I needed with the JavaScript For loop first, storing it in a variable, and then dropping that into my JQuery method. Awesome! Finally, Shawn helped me set the value of each Gallery sidebar image button to equal its index number and on click, pass that value into the call method at the right place. As for the minimal CSS styling, ITP Resident, Mathura, pointed me in the direction of Flexbox to keep all items aligned and scrolling smoothly. 

Feedback for Future Development
On the "position yourself" page: flip the video feed. It's not a mirror image, and people are not used to seeing non-mirrored images of themselves when they look into any kind of reflective surface, including one generated by a live camera feed.

Maybe give additional direction to make sure folks center their faces for their cameras?

Consider the length of viewing time before the user's portrait is taken. Maybe make it longer for folks to arrive at a conclusion or opinion as to what they are seeing. And maybe just show users one image a time, instead of several in row.

What if you captured short video clips or GIFs of participant reactions instead of still photographs? 

There was some confusion as to what people were seeing on the gallery page. Several thought that their portrait was paired with an animal as the result of a machine learning algorithm that matched similar-looking faces. Shawn suggested adding a some text, i.e. "Person looking at _____". 

Provide an option for users to delete their images from the gallery. But also think about how you might integrate the gallery feed with an Instagram or Twitter account.

For many, it all happened too fast to understand what was happening. Isa suggested considering ways to slow down the process. The typical photo booth is usually way too fast, but this is not your typical photo booth. 

Test this version on browsers beyond Chrome and on devices other than a Mac laptop--can you make it responsive? (For example, Amitabh mentioned that the gallery's sidebar pics appeared as hyperlinked text until he changed the resolution of his Windows laptop.)

You're currently saving both images at a time as diptychs, which are ~1.3 MB and a little slow to load from the Digital Ocean Droplet (at least when you're at home). Depending on the gallery design, perhaps just save the portrait of the user and redraw the original image alongside of that to halve the overall file size.

Also, during the in-class demo, diptychs were not necessarily categorized with the correct animal image on the gallery page. This means that the incorrect index number of the animal photo was saved into the database when the diptych was created. Why the misfiring? Is this from multiple uses of the app all at once?

I'm not convinced the project should live online, but I certainly learned a lot in the process of building it out for this platform.

As a project reference, see Kyle McDonald's People Staring at Computers (2011).

Thanks
A special thanks to both Shawn and Mathura for your support on this project, especially for the extra tips and encouragement along the way! And thanks so much to everyone in class for the constructive questions and suggestions after the demo.

Week 5: Serving Over HTTPS

This week we learned how to encrypt traffic between clients and our servers by implementing HTTPS with a certificate (from a certificate authority that has verified our DNS identity) and a private key. It was perfect timing for my photo booth app: I need HTTPS enabled in order for folks to access their webcams. As a quick refresher, my app snaps a photo of an individual as they view another picture. Afterwards a diptych of both images renders on the screen. After testing HTTPS locally and also on my Digital Ocean Droplet (my server), I set to work on the next phase of my project’s development. 

Though I completed most of my photo booth's general workflow in P5 last week, I needed to figure out how to save the resulting diptych. Before setting out to save it to my server, I focused on saving it to disk. For this I learned that I could store my canvas in a variable and from that copy the specific area of the diptych to then automatically download to my computer with P5's save(). Here's the relevant snippet (the full code is available on GitHub):

function setup() {
  canvas = createCanvas(windowWidth, windowHeight);
}

function displayDiptych() {
    let doubleFrameX = windowWidth / 6;
    let doubleFrameY = (windowHeight - 4 * windowWidth / 9) / 2; 
    let doubleWidth = windowWidth * 2 / 3;
    let doubleHeight = 4 * windowWidth / 9;
  
    diptych = createGraphics(doubleWidth, doubleHeight);
    diptych.copy(canvas, doubleFrameX, doubleFrameY, doubleWidth, doubleHeight, 0, 0, doubleWidth, doubleHeight);
    save(diptych, "diptych_" + time + ".jpg");
}

But for this project to live on my server, I needed a way send the image data there and direct it to the proper folder. This was not a simple task for me to figure out. Fortunately in office hours, Shawn suggested appending elt to my diptych Graphics object in order to use the toDataURL() method.

diptych = createGraphics(doubleWidth, doubleHeight);
diptych.copy(canvas, doubleFrameX, doubleFrameY, doubleWidth, doubleHeight, 0, 0, doubleWidth, doubleHeight);

let imageString = diptych.elt.toDataURL();
console.log(imageString);

This encodes the image into machine readable format known as Base64 produced perhaps the longest string of characters I have ever seen in my browser's JavaScript console—around 600 pages if you were to print it out. To ensure that this was indeed image data, we pasted the string into this online decoder where it returned my original diptych in picture form. Magic! It was truly like being the analog darkroom for the first time to see that incredibly long string turn into a photograph before my eyes and easily the highlight of my week. 

Buy how to send this data to my server? I could not get the P5 httpPost method to work, but fortunately, I found an AJAX/JQuery example on Stack Overflow that did:

let imageString = diptych.elt.toDataURL();
saveDiptych(imageString);

function saveDiptych(img){
  let url = "https://emn200.itp.io:443/save";

  $.ajax(
    {
        url: url,
        type: "POST",
        crossDomain: true,
        dataType: "json",
        data:
        {
            image: img
        },
        success: function(response)
        {
            console.log(response); 
        },
        error: function(response)
        {
            console.log(response);
        }
    });
}

But not immediately. Initially, I received the error message in my server's console, “request entity too large.” However, the fact that my server was still seeing the request gave me hope. A search lead me to this suggestion to increase the default limit of the body parser (a module I installed to parse incoming data server-side), and this did the trick.

var bodyParser = require('body-parser');
var urlencodedParser = bodyParser.urlencoded({ extended: true }); 

var express = require('express')
var app = express()

app.use(express.static('public'));

app.use(bodyParser.json({limit: '50mb'}));
app.use(bodyParser.urlencoded({limit: '50mb', extended: true}));

When I console-logged the data of the request on the server side and saw that long string of characters appear (aka my image data) it was easily the second highlight of my week. 

Screen Shot 2018-03-01 at 9.46.56 AM.png

Now that I could send the image string to my server, I needed to decode it back to binary format (jpeg or png and human-readable with the right viewer) and save the file in a directory. This was accomplished by these lines of code, again thanks to Shawn:

app.post('/save', function (req, res) {
  var data = req.body.image;

  var searchFor = 'data:image/jpeg;base64,';
  var strippedImage = data.slice(data.indexOf(searchFor) + searchFor.length);
  var binaryImage = new Buffer(strippedImage, 'base64');

  var timeStamp = Math.floor(Date.now() / 1000);

  fs.writeFileSync(__dirname + '/public/diptychs/diptych_' + timeStamp + '.jpg', binaryImage);
})

Client and server code on GitHub

I still have quite a bit to do for my app. How will I display all the diptychs? All at once in a gallery or individually by session? And the styling?! Though it's a minimal design, I still need to make decisions about how I want the final piece to look. And of course there remains the question of which batch of images to curate for the booth. Time to get started...

Week 4: Single Page Web App

Is it sunny on Mars?
New features, new languages, new syntax! It's another week in DWD, and this time around we learned how to build single page web applications using jQuery with AJAX and experimented with transmitting JSON-formatted data to and from an Express server. For practice, I built an app to display a photograph taken by NASA's Mar's Curiosity Rover for any given day since it landed on the planet in August 2012. In this week's project the browser does all the work from the code in my html file: it makes a call to the NASA API and displays an image on the page for whatever date is submitted without refreshing the page; no data is sent nor received from my own server. I'm still wrapping my head around how all of this works, but my novice understanding is it's possible for my client-side browser to communicate with NASA's server and load the data dynamically through the use of a type of JavaScript called AJAX, and using the JQuery library makes AJAX methods a lot easier. Unlike the JSON data feed from the OpenWeatherMap example in class, I realized that NASA's feed contains an array of nested objects. The rover takes multiple images a day (over 332,000 since it started in 2012), and in order to render just one of them from any given day in my app, I needed to specify its index number when declaring the value to pull during the API call. To run this I set up a basic HTTP server, the same as from the first week of class.

Curious? Check it out here (for now): marsscapes

Code on GitHub

Final Project Proposal: Photo Booth App
I want to build an online photo booth app, and I have two ideas: 1) build a booth with my own custom image processing algorithms or 2) photograph users' as they look at other images, displaying both shots in a diptych at the end (to see the viewer and the viewed at the time of the viewing). I'm interested to use this latter design with the first idea, too. The second idea is a online remake of a Processing project and would allow me to share it with more people. For either, I'll need to learn how to serve over HTTPS, a secure version of HTTP, in order to utilize laptops' webcams. As part of my preparation this past week, I drafted a version of the second idea in P5 here.