Making computers see Part of an ongoing series of learning creative coding in pure javascript. See more here . And get the files here . There are a bunch of techniques and libraries out there for to detect and play with your camera’s motion data. But most often a simple solution is more than you need. And I’ve found that rolling my own motion detection code gives me better performance, flexibility and understanding. 95% of the time, this simple method is all you’re gonna need. So let’s get going… The technique we going to use is simple. We compare the previous frame‘s colours with the colours of the current frame. If they are above a certain threshold, we can assume there’s motion. Of course there’s no need to compare every pixel. This will be too taxing on our computer and often introduces unnecessary noise into the system. So we’ll use a similar technique to the previous tutorial . Creating a Pixelation Filter To get started we need to get your webcam working in the browser. And I’m not going to go into this. There’s plenty of tutorials floating around. And it’s honestly not that interesting (and kinda verbose). I’ve created a re-usable file, so I’ll never need to re-write the code or even bother about firing up your webcam… . Simply include it in your html, as you do with any javascript file, and it will automatically create a element with a default size of 320x240px that is hidden via css. Javascript here’s the code video (Actually could draw it at a smaller size, then scale up our calculations, and get a further performance boost, and I’ll show you this in a bit. But let’s keep things simple for now). To draw the video to the screen we just reference the video and use the canvas function: drawImage() ctx.drawImage(video, 0, 0, w, h); Now we want to loop through the pixels and get their colour values… The code is exactly the same as creating a using , except we’re using a live video feed: pixelation effect getImageData() var ctx = createCanvas("canvas1"); // setup canvas // sample the colour of every 50 pixelsvar sample_size = 50; function draw(){ ctx.drawImage(video, 0, 0, w, h); // draw video onto screen var data = ctx.getImageData(0, 0, w, h).data; // get the screen's pixels data // loop through rows and columns for (var y = 0; y < h; y+= sample_size) { **for (var x = 0; x < w; x+= sample\_size) {** _// the data array is a continuous array of red, blue, green // and alpha values, so each pixel takes up four values // in the array_ **var pos = (x + y \* w) \* 4;** _// get red, blue and green pixel value_ **var r = data\[pos\]; var g = data\[pos+1\]; var b = data\[pos+2\];** _// draw the pixels as blocks of colours_ **ctx.fillStyle = rgb(r, g, b); ctx.fillRect(x, y, sample\_size, sample\_size);** **} }** } Testing for motion: To test if there’s motion in the video all we now need to do is store in an array our previous frame’s pixel values and compare these to the current values… if the difference is above a certain threshold then there is motion. We could do a formula for the sum of the RGB values or look atthe brightness, but in most cases really need only to compare the red values… **var previous_frame = [];**// choose a brightness threshold, if the old pixel values differs enough then we know there's movement // sample the colour every 50 pixelsvar sample_size = 50; // make an array to hold our old pixel values var threshold = 50; function draw(){ ctx.drawImage(video, 0, 0, w, h);var data = ctx.getImageData(0, 0, w, h).data;ctx.background(0); for (var y = 0; y < h; y+= sample_size) { for (var x = 0; x < w; x+= sample\_size) { var pos = (x + y \* w) \* 4; var r = data\[pos\]; var g = data\[pos+1\]; var b = data\[pos+2\]; // first check if it's not the first frame, but // seeing of when the _previous\_frame_ array // is not we empty, and then only draw something if there's // a significant colour difference **if(previous\_frame\[pos\] && Math.abs(previous\_frame\[pos\] - r) > threshold) {** ctx.fillStyle = rgb(r, g, b); ctx.fillRect(x, y, sample\_size, sample\_size); ** }** _// store these colour values to compare to the next frame_ **previous\_frame\[pos\] = r;** } } } And there you have it. That’s pretty much the basics of motion detection. Pretty neat and simple. To my our code easier to read and deal with, let’s quickly put the motion detection code into a function. There’s not much we need to change, except add a motion array to store all the motion points and their colour values, and then return those: function motionDetection(){ // create an array to store our motion data var motion = []; ctx.drawImage(video, 0, 0, w, h);var data = ctx.getImageData(0, 0, w, h).data;ctx.background(0); for (var y = 0; y < h; y+= sample_size) { for (var x = 0; x < w; x+= sample\_size) { var pos = (x + y \* w) \* 4; var r = data\[pos\]; var g = data\[pos+1\]; var b = data\[pos+2\]; // first check if it's not the first frame, but // seeing of when the _previous\_frame_ array // is not we empty, and then only draw something if there's // a significant colour difference ctx.drawImage(video, 0, 0, w, h); var data = ctx.getImageData(0, 0, w, h).data; ctx.background(0); for (var y = 0; y < h; y+= sample\_size) { for (var x = 0; x < w; x+= sample\_size) { var pos = (x + y \* w) \* 4; var r = data\[pos\]; var g = data\[pos+1\]; var b = data\[pos+2\]; // first check if it's not the first frame, but // seeing of when the _previous\_frame_ array // is not we empty, and then only draw something if // a significant colour difference there's if(previous\_frame\[pos\] && Math.abs(previous\_frame\[pos\] - r) > threshold) { // push the x, y and rgb values into the motion array **motion.push({x: x, y: y, r: r, g: g, b: b});** } _// store these colour values to compare to the next frame_ previous\_frame\[pos\] = r; } } return motion; } And then in our draw loop we can just loop through the motion array and do something with the resulting values: function draw(){ ctx.background(250);**var motion = motionDetection();for (i = 0; i < motion.length; i++) { var m = motion\[i\]; ctx.fillStyle = rgb(m.r, m.g, m.b); ctx.fillEllipse(m.x, m.y, sample\_size, sample\_size);** **}** } Now let’s work on that performance. As is, it’s pretty fast. But we can make it faster. If you think about it, drawing the video to the whole screen and then reading every 10 or 20 pixels is a bit silly, when we could just make a smaller video input and read those pixels… So let’s do that… Firstly let’s make our video even smaller… We’ll use a setup() function which will ensure the video is loaded before we manipulate it… and then we just set the video to a scale of our screen size… var scalefactor = 40; function setup(){video.width = w/scalefactor;video.height = h/scalefactor;} We’d then just need to loop over out video instead of the whole screen, and seeing our video is tiny, we’ll get a nice bump in performance. And because we’re looking over every pixel of the video now, there’s also a tiny reduction in noise. We’d also need to multiply the x and y positions back up by the scalefactor… function motionDetection(){ var motion = []; ctx.drawImage(video, 0, 0, );var data = ctx.getImageData(0, 0, , ).data; // draw the video and get its pixels video.width, video.height video.width video.height for (var y = 0; y < ; y++) {for (var x = 0; x < ; x++) { // we can now loop over all the pixels of the video video.height video.width var pos = (x + y \* **video.width**) \* 4; var r = data\[pos\]; var g = data\[pos+1\]; var b = data\[pos+2\]; if(old\[pos\] && Math.abs(old\[pos\].red - r) > threshold) { ctx.fillStyle = rgb(r, g, b); ctx.fillRect(x \* scalefactor, y \* scalefactor, scalefactor, scalefactor); _// push the x, y and rgb values into the motion array // but multiply the x and y values bck up by scalefactor // to get their actual screen position_ motion.push({**x: x \* scalefactor, y: y \* scalefactor**, r: r, g: g, b: b}); } old\[pos\] = { red: r, green: g, blue: b}; }} return motion; } And there you have it. A simple chunk of re-usable motion reactive code. Once you have your motion array, try attaching particles. Watch out for ambient light, which could add noise and adjust your threshold up and down accordingly. And easy way for us to make tweaking easier is to add some sliders to adjust our … threshold I like to use ’s quicksettings: bit101 https://github.com/bit101/quicksettings (Also check out his lab and Coding Math video series for some great creative coding examples and tips). Once you’ve included the library, you instantiate it like so: var settings = QuickSettings.create(); And then add Range sliders for and : threshold sample_size // addRange(label, min, max, start, increment, callback) settings.addRange("threshold", 5, 100, 50, 1, function(value) {threshold = value;}); Another technique that’s often used, for installations especially, is . The code is exactly the same. But instead of comparing the previous frame’s pixels, you take a snapshot on say a keypress, of your empty room, and then compare that image to your current pixels. Try get it going, and hit me up in the comments or if you like if you’re struggling and I’d be happy to help. background subtraction on twitter And that’s pretty much all you need for motion detection. Happy coding. As usual the full code is available on my github: https://github.com/GeorgeGally/creative_coding You can see previous all my previous creative coding tutorials . here And follow me here if you so desire: https://www.instagram.com/radarboy3000/ https://twitter.com/radarboy_japan _Radarboy. 145 likes. Art, design visualisation, hacks_www.facebook.com Radarboy
