[JS Kongress 2016] KittyCam.js - Raspberry Pi Camera w/ Cat Facial Detection

50 %
50 %
Information about [JS Kongress 2016] KittyCam.js - Raspberry Pi Camera w/ Cat Facial...

Published on November 29, 2016

Author: tomomi

Source: slideshare.net

1. girlie_mac@ KittyCam.js Smile, you’re on camera!Smile, you’re on camera! Tomomi Imura

2. girlie_mac@ Hello, I’m Tomomi & I’m a: ● San Francisco Dweller ● Front-End Engineer ● Open Web + Tech Advocate ● N00b Hardware Hacker ● Sr. Developer Advocate at Nexmo ● Cat lady of InterWeb

3. girlie_mac@

4. girlie_mac@ CC-BY-SA 3.0 https://commons.wikimedia.org/wiki/File:Basic_robot.jpg

5. girlie_mac@ Arduino ● MCU-based kit ● Open-Source hardware & software (IDE) ● The first developer-friendly boards

6. girlie_mac@ Sketch ● Language for Arduino ● Loosely based on C ● Wut, C? ● HALP!!1!!! #include <Servo.h> #include <Wire.h> #include <Firmata.h> #define I2C_WRITE B00000000 #define I2C_READ B00001000 #define I2C_READ_CONTINUOUSLY B00010000 #define I2C_STOP_READING B00011000 #define I2C_READ_WRITE_MODE_MASK B00011000 #define I2C_10BIT_ADDRESS_MODE_MASK B00100000 #define I2C_MAX_QUERIES 8 #define I2C_REGISTER_NOT_SPECIFIED -1 #define MINIMUM_SAMPLING_INTERVAL 1 int analogInputsToReport = 0; byte previousPINs[TOTAL_PORTS]; byte pinConfig[TOTAL_PINS]; byte portConfigInputs[TOTAL_PORTS]; int pinState[TOTAL_PINS]; unsigned long currentMillis;

7. girlie_mac@ Raspberry Pi ● $35 single-board computer ● Broadcom chip with ARM-compatible CPU & GPU ● Runs Linux ● More language choices: C, C++, Python...

8. girlie_mac@ But… I want to code in

9. girlie_mac@ Johnny-Five ● JavaScript robotics framework ● Works with Arduino-compatible Boards ● IO plugins for more platform supports ● http://johnny-five.io/

10. girlie_mac@ Awww, JavaScript all the way!

11. girlie_mac@ Hello world!

12. girlie_mac@ KittyCam Raspberry Pi camera with cat facial detection! ● Hardware: Raspberry Pi, camera, and PIR sensor ● Software: Node.js + J5 + More open-source goodies

13. girlie_mac@ Motion detected! snap! Mmm… donut

14. girlie_mac@ kittyCam.js 1. Detect motion 2. Take a photo 3. Cat facial detection 4. Store the photo in cloud 5. Real-time view on web 6. SMS the photo link HELL, YEAH!

15. girlie_mac@

16. girlie_mac@ Raspberry Pi 3 PIR Motion Sensor Camera Module F/F Jumper Wires (x3)

17. girlie_mac@

18. girlie_mac@

19. girlie_mac@ 1. Raspberry Pi 3 2. 5MP Camera Board Module 3. Pyroelectric Infrared (PIR) motion sensor 4. Female/Female wires

20. girlie_mac@ Programming Raspberry Pi Pre-installed on Raspbian OS: C / C++

21. girlie_mac@ Programming RPi with Node.js

22. girlie_mac@

23. girlie_mac@ Installing Node ARM $ wget https://nodejs.org/dist/v4.4.5/node-v4.4.5-l inux-armv7l.tar.xz $ tar -xvf node-v4.4.5-linux-armv7l.tar.xz $ cd node-v4.4.5-linux-armv7l $ sudo cp -R * /usr/local/

24. girlie_mac@ kittyCam.js 1. Detect motion 2. Take a photo 3. Cat facial detection 4. Store the photo in cloud 5. Real-time view on web 6. SMS the photo link HELL, YEAH!

25. girlie_mac@ kittyCam.js Tech Stack 1. Detect motion w/ Johnny-Five IR.Motion obj 2. Take a photo w/ Raspistill, command line tool 3. Cat facial detection w/ KittyDar 4. Store the photo in Cloudinary 5. Publish & subscribe the url to display on web via PubNub 6. Send a text message via Nexmo

26. girlie_mac@ $ Raspistill canvas catDetect.js app.js kittyDar Motion detected take a photo Store the photo if cats are detected display photos on web browsers real-time anywhere Johnny-Five w/ raspi-io child process Returns url Notify with SMS via

27. girlie_mac@ Johnny-Five w/ Raspi-io const five = require('johnny-five'); const raspi = require('raspi-io'); let board = new five.Board({io: new raspi()}); board.on('ready', () => { console.log('board is ready'); ... });

28. girlie_mac@ Motion const five = require('johnny-five'); const raspi = require('raspi-io'); const board = new five.Board({io: new raspi()}); board.on('ready', function() { // Create a new `motion` hardware instance const motion = new five.Motion('P1-7'); ... }); a PIR is wired on pin 7 (GPIO 4) VCC Ground Data

29. girlie_mac@ Raspistill Command Line Tool $ raspistill -o myPhoto.jpg

30. girlie_mac@ PIR Sensor > Run Camera const child_process = require('child_process'); board.on('ready', () => { const motion = new five.Motion('P1-7'); motion.on('motionstart', () => { // Motion detected let filename = 'photo/image_'+i+'.jpg'; let args = ['-w', '320', '-h', '240', '-o', filename, '-t', '1']; let spawn = child_process.spawn('raspistill', args); spawn.on('exit', function() { console.log('A photo is saved as '+filename); ... motion detected! Take a photo! Spawns a new process w/ a given shell command

31. girlie_mac@ Processing Image spawn.on('exit', () => { let imgPath = __dirname + '/' + filename; // Child process: read the file and detect cats with KittyDar let args = [imgPath]; let fork = child_process.fork(__dirname+'/detectCatsFromPhoto.js'); fork.send(args); // the child process is completed fork.on('message', (base64) => { if(base64) { uploadToCloud(base64); // Send to cloud storage } }); ... Create another worker by running a new instance of V8 engine.

32. girlie_mac@ detectCatsFromPhoto.js const kittydar = require('kittydar'); const Canvas = require('canvas'); process.on('message', (m) => { fs.readFile(m[0], (err, data) => { ... let canvas = new Canvas(w, h); let ctx = canvas.getContext('2d'); ctx.drawImage(img, 0, 0, w, h, 0, 0, w, h); let cats = kittydar.detectCats(canvas); console.log('There are', cats.length, 'cats in this photo'); if(cats.length > 0) {base64Img = canvas.toDataURL();} process.send(base64Img); process.exit(0); }); Running in an another process a cat detected! KittyDar

33. girlie_mac@ KittyDar ● Open-source JavaScript cat facial detection written by Heather Arthur ● Takes a canvas obj & calculate the locations of cats in the image let cats = kittydar.detectCats(canvas);

34. girlie_mac@ KittyDar: Behind the Scene 1. Chops up the image up into many “windows” 2. Extracts data by measuring a set of gradients, from light & dark in order to find edges 3. Compares the direction of these edges to the edges found in known cat images Neural network (JSON w/ vector data) is pre-trained w/ thousands pics of cats & non-cats

35. girlie_mac@ Cat Facial Detection http://research.microsoft.com/pubs/80582/ECCV_CAT_PROC.pdf

36. girlie_mac@ Cat Facial Detection http://research.microsoft.com/pubs/80582/ECCV_CAT_PROC.pdf

37. girlie_mac@ Cat Facial Detection The annotation data sequence: (Number of points, default is 9), (Left Eye), (Right Eye), (Mouth), (Left Ear-1), (Left Ear-2), (Left Ear-3), (Right Ear-1), (Right Ear-2), and (Right Ear-3)

38. girlie_mac@ Cat Facial Data 9 247 94 294 92 273 127 213 81 207 29 247 53 286 50 320 20 322 74 00000561_012.jpg.cat00000561_012.jpg

39. girlie_mac@ Cat Facial Data Collection

40. girlie_mac@ Neural Network Positive Negative JSON data hog-descriptor kittydar.jsCompare data

41. girlie_mac@ Send the Pic w/ Kitty to Cloud const cloudinary = require('cloudinary'); // the child process is completed fork.on('message', (base64) => { if(base64) { cloudinary.uploader.upload(base64, (result) => { // Done! - Get the URL and do more stuff }); } else deletePhoto(imgPath); });

42. girlie_mac@ View Photos Real-time via Socket const pubnub = require(pubnub); Publish Subscribe

43. girlie_mac@ http://www.girliemac.com/RPi-KittyCam/

44. girlie_mac@ Send SMS via Nexmo const Nexmo = require(nexmo); let nexmo = new Nexmo({//config with API keys}); nexmo.message.sendSms( FROM_NUMBER, TO_NUMBER, ' '+ url, options, (err, responseData) => { if (err) console.log(err); else console.dir(responseData); });

45. girlie_mac@

46. girlie_mac@ QA Team Lead QA: Jamie Ginger Basil Alice Yugi Venom @kittenVenom Ozzy

47. girlie_mac@ github.com/girliemac/RPi-KittyCam

48. girlie_mac@ Next Steps ● Upgrade Hardware ○ Raspberry Pi 3 ○ NoIR Night Vision Camera ● Upgrade Node (was 0.12) & all dependencies ● ES5 to ES6 ● More features (Maybe) ○ Cat Identification w/ RFID ○ Photo Booth w/ Filter effects & props

49. girlie_mac@ More Cat Projects w/ Node.js

50. girlie_mac@ https://http.cat Thanks Rogério Vicente for the .cat domain & the API! Previously: HTTP Status Cats

51. girlie_mac@ HTTP Status Cats Slack Bot Install it at: http://girliemac.com/ slack-httpstatuscats/ Tutorial on Medium.com: https://goo.gl/saOUl0

52. @girlie_mac Next Project? ● Selfie bot (à la Mannie the Selfie Cat) Mannie the Selfie Cat by @yoremahm on Instagram https://www.instagram.com/yoremahm/

53. girlie_mac@ Danke schön! @girlie_mac github.com/girliemac

54. girlie_mac@ Attribution: Emoji by Emoji-One (CC-BY 4.0)

Add a comment