Week 1: Socket Game

Aaand I'm back from the summer and right into playing with sockets again! For the first assignment of Understanding Networks, I created a device to automatically connect to a Wifi network and then to a server using a TCP socket connection to play a multiplayer pong-like game. The game's objective is to keep a ball, which drops down the screen vertically, from hitting the ground. Once connected to the game server, my device controls it's own "paddle" on the screen, and the more times the ball bounces of my paddle, the more points I accrue. 

I decided to work with an Arduino MKR1000 board for its Wifi connectivity capability and multiple pins (in the event that I might need it for future projects). I thought it would be fun to learn how to work with an accelerometer and gyroscope and move my physical paddle in 3D space to control the digital one in the game. For that I settled on the fancy Adafruit 9-DOF Absolute Orientation IMU Fusion Breakout BNO055, which also provides many more features that might come in handy at a later time. It had been awhile since I worked with an Arduino, and because coding to connect to wireless networks was new to me, as well as working with the 9-DOF sensor, the process of creating my game device was completed over several stages. 

Part 1: Board Installation & Wifi Connectivity
Home: First I installed my new board in the Arduino IDE, followed by the Wifi101 library. Then I ran a couple of the provided example sketches to ensure that I could connect to my home router and access the Internet. The Wifi101 ScanNetworks sketch scans the area and displays 802.11b/g networks that it can see. It also prints my board's MAC address (Media Access Control address), a number that is unique to my board and necessary for using my device at school (see below). With my home router in sight, I adapted the Wifi101 WebClient sketch with my network SSID and password to connect to my router and make a GET request to Google. My MKR1000 connected successfully to the Internet.

School: In order to connect my MKR1000 to ITP's Sandbox network, I registered my board with NYU with the previously retrieved MAC address. With that completed, I ran the same example sketches to ensure that could connect to the Internet while on the floor.

Part 2: Testing Game Play with Buttons
Before working the 9-DOF sensor, I made a prototype with buttons to ensure that I understood the process of connecting to the game server and could successfully move my digital paddle with my physical device. I modified this sketch replacing the joystick with four buttons for right, left, up, and down movements, and added a LED to indicate that my board was connected to the router. Currently, the board automatically attempts to connect to the Wifi network when powered on, but as it takes a few moments, I wanted visual feedback to ensure a successful connection. I also included LEDs for each directional button press for additional feedback. The button to connect to the game server was also paired with an LED. Last but not least, I added a button to toggle the label on my game paddle between my device's IP address and my name.

To test it out the game my laptop, I first retrieved my computer's IP address (assigned by my router) and included that in my Arduino program. Next I opened the game server file, written in Processing, and hit the play button to launch the server and open a socket. Pressing the gray button on my device connects it to the server, and pressing the other buttons moves my digital paddle around the screen. At this stage the physical interaction was difficult because of all the wires and button placement, but it worked! 

Part 3: Working with the Adafruit BNO055 Absolute Orientation Sensor
After downloading and installing the Adafruit BNO055 library and the Adafruit Unified Sensor Driver, I ran some sample code mentioned in the Adafruit tutorial (Arduino Code > Wiring for Arduino > Adafruit Unified Sensor System) which is noted as "probably the easiest option if all you care about is absolute orientation data across three axis." (Note: this sketch derives from a much longer example included with the library once installed in the Arduino IDE: Examples > Adafruit BNO055 > sensorapi.) 

With this sketch I determined how to retrieve data from movement along the X axis (left to right) and the Y axis (up and down). Lifting the sensor up increases the Y values and downward decreases them. Likewise, moving the sensor right increases the X values and to the left decreases them. With that information, I wrote a simple sketch to print out the letters to the serial monitor for each direction I moved the sensor: "u" for up, "d" for down, "l" for left, and "r" for right. Curious, I bypassed Adafruit's sensor fusion algorithms to work with the data directly (Examples > Adafruit BNO055 > rawdata), and found similar but slightly smoother results with Euler angles. Using this data I could point my physical paddle at the screen like a TV remote to move the digital paddle.

Next I incorporated this work into my game client sketch. Early on I noticed that the BNO055 example sketches employed a delay at the end of every sensor reading, which I was loathe to do and freeze everything while it was connected to the server. Doing so produced jumpy results. Without a delay, the sensor data printed quite fast to the serial monitor during initial testing, but after various iterations I focused on two areas to slow it down: 1) the interval for which I retrieved new values of X and Y and 2) the sendInterval for which messages are sent to the server with the client connected. Adjusting these resulted in the screen paddle responding at a more manageable speed, although it could be refined in the future. (Also consider revisiting the threshold for setting previousX & previous Y values.)

Finally, I tidied up my breadboard. The current version includes a red LED to indicate power from the battery, a blue LED to indicate when its connected to WiFi, and a green LED (next to the green push button) to indicate when it's connected to the game server. The gray button toggles between my name and my device's IP address on the game screen. Take a look and watch me play below:

Final Materials
1 Arduino MKR1000
1 Adafruit 9-DOF Absolute Orientation IMU Fusion Breakout - BNO055
3 LEDs
2 Pushbuttons
2 10k Ohm Resistors
3 220 Ohm Resistors
1 3.7V 1200mAh Lipo Battery with JST Connector
1 Breadboard
22 & 24-Gauge Solid-Core Wires

Conclusions
This assignment was a lot of fun, but I need to spend more time learning how to work with the rest of the raw data on the BNO055 and refine how the movements in physical space translate to the screen. As of now, the digital game paddle seems to mirror my hand movements quite well when the client is connected, but it might be a bit too fast. Too slow, however, and it’s really annoying. It’s a fine line, but one that I wouldn’t have considered had I not challenged myself to work with this sensor. One problem that I encountered several times during testing at home and at school was the server freezing upon toggling between my name and my device’s IP address. It didn’t happen all the time, however, and I wasn’t able to repeat the problem in the same way. In class, game play was definitely more glitchy and delayed with all of us playing together.

Code on GitHub