19 Nov 2018

Introduction

This blog post is about what I did at Sac Hacks. The first intercollegiate hackathon in Sacremento with 400+ hackers. This year's sponsors were companies like EA, XYO, UC Davis, GitHub, SmartCar, Nvidia, Twillio, Google and more. If you'd like to find out more click here. My team was Siddhartha Datta, who I had met at the GM hackathon and did Cal Hacks with. Rea Savla, my lab partner for machine structures and her friend, Tushar Jain.

Going in, Siddhartha and I knew we wanted to do something with ML because that's where both of our interests lie but beyond that we had no biases. What I did know was that I wanted to take my mistakes from Cal Hacks and learn from them. My goal was to have a more finished and polished deliverable at the end which would require better communication and management across our team. Especially if we went for something bigger, which we did.

We had a pretty rough start in the beginning because we couldn't settle on a strong idea which took advantage of the various API's and companies presented to us. Each idea had either been done before, was too simple, too infeasible or just not practical enough. We had 24 hours to come up with an idea and execute it but it took us 6 hours to finally decide upon an idea. Definitely the most painful and difficult part of the process, but as we would later find, worth it.

Our Project

We had three main parts to our project, detecting crime automatically using cars, notifying the network and rewarding cars for being part of the network.

Identifying Crime

This part was mainly Siddhartha's role as only him and I have a lot of experience developing machine learning solutions and I was the only one who could do full stack to bring all of the pieces together. He used an inception v2 R-CNN trained on a weapons dataset. The idea was to place cameras in each car in the network, or for newer cars simply access their built in cameras, and use the model to run real time inference to detect suspicious activity. As we didn't have guns or a camera we used our webcam to demo this. You can see Siddhartha's wonderful acting here. And I put a picture of our model's structure below.

Notifying the network

We did this through SMS and a web application. Rea and Tushar were in charge of getting the car's geolocation, odometer, locking all surrounding cars and sending out an SMS when a threat was detected by the model. For this they used SmartCar's API which gave us access to the car's GPS, odometer and locking mechanisms. For the SMS they used Twillio and NoonLight which let them send texts to those in the network as well as the police.

I was in charge of the web application (you can see here) which needed to visualize the crimes, where the car was when it saw the crime and the rewards for each car in the network. I used Flask to build the web app, Google Cloud Platform's Map API and some JavaScript to visualize the crime data, Ethereum's API to access the ledger and Twitter's bootstrap for some of the CSS. When crime data was pushed to the database it would all be visualized here, when each dot is clicked you can view the frame which set the model off as well as the location and time. Each of the paw prints represents where the car saw the crime. Finally, the ledger shows the rewards each car got for detecting a crime.

Rewarding Cars

Since this is meant to be a community solution to a real problem we wanted to give the power to the masses by allowing anyone to put up rewards for detecting crimes in their neighborhood. So we turned to smart contracts with Ethereum, letting users and companies alike put up rewards.

Rea was the only one who had experience with blockchain before so she was in charge of making the smart contracts, though luckily she taught us how they worked along the way. The idea is that a community could put up a reward for finding crimes in their area and if a crime was found the user would get a payment as a function of how far they had driven in our network and the severity of the crime detected. The first would be gotten with SmartCar's API and the second with our crime detection model. Allowing users to place rewards was very important as we are all from Berkeley, though its a great school we have a lot of crime around here and we would love to be able to give power to student's.

The smart contract Rea wrote in Solidity

Putting it all together

My main goal this time was to have a polished and cohesive deliverable at the end. But using so many API's and having so many moving parts, this was definitely the hardest part of the project. Overall the APIs and frameworks we used were

So one thing I learned from my last hackathon was to make the input and output each person would be expecting for their parts very clear. Explaining that images were to be placed into a folder with a certain naming scheme to provide metadata, that the ledger should output JSON data, etc. Taking the time to work out the entire workflow and formats of each piece made each piece fit into the full product much easier. It also helped that I had such a great team, Siddhartha and I had worked together before and are good friends, Rea and I work together every week in machine structures and she has known Tushar for a long time. Having good communication was the only way we were able to put together an application with so many moving parts and have a great presentation at the end.

Awards

We were lucky and won four awards.

This was my third hackathon, Siddhartha's fourth and Rea and Tushar's first hackathon so we won best beginner hack.

They liked our name which Tushar and Rea spent two hours coming up with and our logo which Tushar spent an hour creating so we won best tech domain!

The SmartCar team was extremely kind, they helped us use their API and work it into our product, hopefully we will be able to work with them later on. They awarded us second place for using their API

The one we are most proud of, third place overall, we were all shocked that we won an MLH hackathon and Tushar loves the medal. The MLH guys were all really nice and helped us with our hardware needs as well as our domain issues.

A big thank you to the SacHacks team for putting on the event, the SmartCar team for helping us with their API and all their support through the event, the XYO blockchain team for inviting us to present at their seminar next month and helping us create and deploy our smart contract, the MLH team for helping with the event, our hardward needs and all other issues we had and of course Siddhartha, Rea and Tushar for all their hardwork!