Python/Flask Data Visualization & Interactive Maps

Have you ever wanted to create an interactive data visualization map? In my most recent side project, I created a pretty cool visualization for how a virus might spread across the United States. If you want to check out the finished site, you can click here: http://ethanszombies.herokuapp.com Now, I’ll walk through my thought process for creating this, as well as step by step instructions on how to create your own. I know there are probably better/more efficient ways to do this, so if you are reading this, and have a better idea, please feel free to share. Here’s a breakdown of how this process will work for me. I’m really dividing this project up into two parts. The first part is gathering the data. The second part is the user portion, so users can use the data we’ve processed in the first part. Here’s a flow chart of how this workflow might look: 1 — What do we want to create? — — — — — — — — — — — — — — — — — - For this, while I do want to make a map that simulates a virus outbreak, the virus I’m going to focus on is the zombie virus. Based on hours and hours of very important research, I found a few key points about how the zombie virus usually spreads: 1. The Zombie virus spreads more quickly in densely populated areas like inner cities. 2. Zombies are clumsy and uncoordinated, making unfavorable terrain difficult. Thus, areas that have a high elevation would make the zombie virus spread more slowly. 3. While the standard spread of the zombie virus is limited to a zombie walking around and biting people, occasionally the virus can spread more quickly if an infected individual tries to flee in a car or other vehicle, making roads and interstate highways favorable for quickly spreading the virus. There is certainly ample data to consider, like the effect weather, climate or the military would play in the advance of the virus, however, before things get too crazy, we’ll start with just tracking these three data points, 1. Population Density, 2. Elevation, 3. interstate highways. 2. — What considerations do we need to think about? — — — — — — — — — — — — — — — — — - Let’s say that we have a map of the United States, and each area in the map is colored white. We could represent an infected person by changing the color of that pixel, or div to a different color. a. Time Intervals First, we would need to decide on an interval of time. This interval could be 1 second, or even less. Each interval of time would represent an actual measurement of real time, like an hour or a day since the infection. b. Decision at each interval We could then simulate the actions of that div. At each interval of time, we would need to decide where the infected block moves, how fast it moves there, and whether or not it infects anyone. We could use the data about each area to make those decisions. Infection Population Density would determine if and how many other people an infected person infects at each interval. If an infected block is in a highly populated area, it would infect many people. If not, it may not infect anyone. Speed Elevation would be a factor in speed. At each interval, an infected block would move a certain distance, but if it is a higher elevation, that distance would be less. Direction Interstates would affect mainly direction. We could set the standard movement of each infected block to a random pattern, but if the zombie stumbles on or near a highway, then it’s direction would remain on the highway for the some amount of time. 3. — How would we get the data we want onto a map? — — — — — — — — — — — — — — — — — - The first step would be to encode the map with the data we need. For me, it seems like one way to do this would be to create a map of a certain dimension. For my purposes, I think 1150px X 768px should work nicely. Next, we would need to determine how much space on the map one zombie would occupy. After some testing, I found that an area approximately 5px X 5px was adequate. Any more than that, and it seemed clunky. Any less, that there was so much data to process, that it constantly ate up resources. After deciding on a proper map size, my next move is to create an overlay on top of the map. This overlay would contain buttons the size I need (5px * 5px each). For population density, to start with, I’m going to create a max population density of 10, and a minimum of 0. These numbers are arbitrary, and are just meant to represent variation in density, but don’t necessarily correspond to actual population numbers. Although the above examples aren’t to scale, you can see the idea here at least. By creating a grid to overlay on top of the map, we can encode data into each block of the grid, that would change the way the outbreak reacts depending on which blocks the outbreak reaches. My thought is that you could click each button, and on each click, the number (density) would increase by 1. If we have 40,000 buttons, clicking each one seems unreasonable, so instead, I may choose to bind that event to a mouseover to make the entry of data faster. It might also be a good idea to change the color on the mouseover event, so it’s easier to keep track of what divs contain what values. Now that we’ve thought about the pseudo code, it’s a good time to actually start setting up our file and writing some actual code, since we have a more clear idea of what we want to create, and how we’ll have to create it. In a lot of projects, I spend a lot of time drawing things out before I write any code, which tends to make the code more organized in the end. File Setup — — — — — — — — — — — — — — — — — - The file structure that we’ll eventually have will probably look something like this: To start off, we can create most of this using the command line. Some of the files, like the density.txt and densitydata.json will be created in the first portion of the project. on the command line, do the following: mkdir cd Now, we’ll create all the initial files on the command line: touch app.py .gitignore README.md requirements.txt runtime.txt mkdir templates mkdir static cd templates touch base.html cd .. cd static touch main.css Next, if we want to create a git repository and deploy on heroku, this is a good place to do that. git init Then, if we set up a heroku account, we can create a new heroku project by running: heroku create Before we start installing dependencies and libraries, we definitely want to set up a virtual environment in Python. To do this, we can run the following command: pyvenv-3.6 env (substitute your python version here) next, we’ll activate that environment as follows: source env/bin/activate In the command line you’ll now see the (env) before your project folder to let you know your running a virtual environment. If you ever get an error from here on out that python can’t find a module you’ve installed, check to see if you are still using your virtual environment. If not, then run the last command again. Next, since flask is the main dependency we’ll need, we can install that. pip install flask If we decide to run this on heroku, we’ll need to tell it which version of python we’re running and we can add that to our runtime.txt file: echo "python-3.6.4" >> runtime.tx (or whatever your version is) We’ll also want to install gunicorn if we’re using heroku: pip install gunicorn And then add the following text to our Procfile: echo "web: gunicorn app:app" >> Procfile Finally, we’ll run pip freeze in order to update our requirements.txt file: pip freeze > requirements.txt Cool, now that that’s all out of the way, most of the files in our above file tree should be there, and we’re ready to rock and roll. Going back to our flowchart from earlier, here’s the area we’ll be working on first: CSS — — — — — — — — — — — — — — — — — - Based on the workflow we talked about earlier, the first thing we’ll need is an image of a map that we can overlay the grid of buttons on top of. To do that, we’ll just save a map that suits us, and use it as the background image for div or element. We’ll also set it to the appropriate dimensions. This will go in the main.css file we created earlier: #body { background-image: url('/static/states.png'); background-size: contain; background-repeat: no-repeat; background-size: 1150px 768px; margin: 0px; padding: 0px; display: flex; flex-flow: row wrap; justify-content: flex-start; align-items: flex-start; width:1150px; height:768px; border:black solid 2px; } We’re also going to create two button classes here. When a user clicks, or an active zombie block infects another block, we’ll change the block from one css class to another. Here are the classes I created. ‘.btn’ class is the default button class, and ‘.active’ is the activated class: .btn { width: 4.5px; height:4.8px; background-color: transparent; margin: 0px; padding: 0px; font-size: 1px; } .active { width: 4.5px; height:4.8px; /* Permalink - use to edit and share this gradient: http://colorzilla.com/gradient-editor/#136325+0,305937+100&1+15,0+88 */ background: -moz-radial-gradient(center, ellipse cover, rgba(19,99,37,1) 0%, rgba(23,97,40,1) 15%, rgba(44,90,53,0) 88%, rgba(48,89,55,0) 100%); /* FF3.6-15 */ background: -webkit-radial-gradient(center, ellipse cover, rgba(19,99,37,1) 0%,rgba(23,97,40,1) 15%,rgba(44,90,53,0) 88%,rgba(48,89,55,0) 100%); /* Chrome10-25,Safari5.1-6 */ background: radial-gradient(ellipse at center, rgba(19,99,37,1) 0%,rgba(23,97,40,1) 15%,rgba(44,90,53,0) 88%,rgba(48,89,55,0) 100%); /* W3C, IE10+, FF16+, Chrome26+, Opera12+, Safari7+ */ filter: progid:DXImageTransform.Microsoft.gradient( startColorstr='#136325', endColorstr='#00305937',GradientType=1 ); /* IE6-9 fallback on horizontal gradient */ margin: 0px; padding: 0px; font-size: 1px; border-radius: 25%; } .btn:hover { cursor: pointer; } HTML — — — — — — — — — — — — — — — — — - Then, we can go ahead and create the actual HTML that is indicated in our CSS. This will go in the base.html file we created earlier: SUBMIT JavaScript — — — — — — — — — — — — — — — — — - Next, we’ll add some JavaScript. We’ll use a loop to create buttons for the length we need. If you used the same dimensions I did, then 40,000 buttons seems about right. Normally, you might have a separate JavaScript file. While that is probably best practice, I find that if I’m writing less than 300 lines of code, then including the JavaScript in a script tag is usually easier. If it looks like we’re going to have more than that, we can always change it later. $(document).ready(function(){ $(document).ready(function(){ for (var i = 0; i 0 `; $('#body').append(inputBtn); } }); Next, I’ll create the mouseover event for all of the tags we just created: var color1 = '#fff600'; var color2 = '#ffc32b'; var color3 = '#e8a600'; var color4 = '#e88300'; var color5 = '#ce7502'; var color6 = '#ce3b01'; var color7 = '#ad3201'; var color8 = '#7a2402'; var color9 = '#561a02'; var color10 = '#1e0800'; $('body').on('mouseover', '.btn', function () { var currentVal = $(this).html(); console.log(currentVal); var newVal = parseInt(currentVal) + 1; if (newVal > 10){ newVal = 10 } $(this).html(newVal); if ($(this).html() == 1){ $(this).css('background-color', color1); } else if ($(this).html() == 2){ $(this).css('background-color', color2); } else if ($(this).html() == 3){ $(this).css('background-color', color3); } else if ($(this).html() == 4){ $(this).css('background-color', color4); } else if ($(this).html() == 5){ $(this).css('background-color', color5); } else if ($(this).html() == 6){ $(this).css('background-color', color6); } else if ($(this).html() == 7){ $(this).css('background-color', color7); } else if ($(this).html() == 8){ $(this).css('background-color', color8); } else if ($(this).html() == 9){ $(this).css('background-color', color9); } else if ($(this).html() == 10){ $(this).css('background-color', color10); } }); In this previous example, each block or array element will have a value between 0 and 10. If we wanted to go back and edit this form and data, so that we could store data about the population and elevation on each block, we may want to increment the value differently, or store a true/false value instead. We can always change the input here, but have it write to a new file on the back end, depending on what data we want to store. We’ll talk more about this idea later, but just something to consider. And lastly, I’ll create a form, which allows me to send this data from my flask front end to python on the back end. I can then manipulate the data and create the hash. var inputArr = '' $('#submitform').click(function(){ $(".btn").each(function(){ var input = $(this).html(); inputArr += ("," + input); }); var form2 = ` `; console.log(inputArr); $('#formGoesHere').append(form2); $('#myForm').submit(); }); }); Even though we’ve created the form and everything on the front end, it’s not actually connected to anything yet. In order to connect this data to python, we’ll need to modify our app.py file, and use Flask requests to receive the data from our base.html file. At this point, we’re now here in our flowchart: Python — — — — — — — — — — — — — — — — — - In app.py, let’s go ahead and import the things we’ll need to make this work: from flask import Flask, request, render_template next, we’ll define the app: app = Flask(__name__) Finally, we’ll define the route which will be associated with our base.html file, which will just be the main route, (“/”): @app.route('/', methods=['GET', 'POST']) def searchTopic(): return render_template('base.html') This route will render the page, but we’ll also need to make a route for the form. If you go back to the html we wrote, the action for that form was “form1”, and that will be the route we’ll use for our form data. @app.route('/form1', methods=['GET', 'POST']) def getFormData(): data = request.form['t1'] f = open('file.txt', 'w') f.write(data) f.close() return render_template('base.html') Here, our route in app.py, “/form1” is associated with the action of the form in our html. And the input name in that form, “t1” is associated with the request.form[‘t1’] in app.py. Last, we’ll add the following line to the bottom of our app.py file: if __name__ == '__main__': app.run() Now, if we submit the form, that data will be saved as the variable ‘data’. Then, we’ll open a file called file.txt, and write our ‘data’ variable to that file. Now, we can iterate over the data in the file, and convert it to JSON, along with the other data we may collect later. Back to HTML/JavaScript — — — — — — — — — — — — — — — — — - If you set yours up like this, you can see that on the mousever, we’re changing the value of the html in each div, as well as the color. The color makes it easy to see what has been changed and what hasn’t. When I submit the form, I’m able to send the value of the divs as an array to python. The input process might look like this: The console log there, is logging the value of each div, and you can see the area map as we create it. Once we have all the map data that we want, we can comment that code out of our JavaScript file, since we won’t need it to actually run the simulation. But it is handy to have if we want to make a new map of any new kind of data. In my case, I submitted one map which contained only the population data…another map which contained the elevation data, and so on…and then combined each list into a hash in python. The data we’ve input via the front end, is being interpreted as an array, or comma separated values. The map data would basically be an array of 40000 elements ranging from 0 to 10, like so: [0,0,0,0,0,0,1,1,2,3,4,0,0,0,0,0,0,0,5,6,6,7,7,7,4,1,0……..] 4 — Assembling the raw data into JSON — — — — — — — — — — — — — — — — — - In python, each list will have the same number of indices, meaning, in list 1, the array at index 552 will correspond to the 552nd div in our grid. So I’ll loop through one list, but check the other list at the same time, without having to loop through both lists. In the example below, I have one list which contains the population data, and a second list which contains data about whether the current block is water or not. import json with open('population.txt') as f: file1 = [line.rstrip() for line in f] with open('file.txt') as f: file2 = [line.rstrip() for line in f] array = [] for idx, i in enumerate(file1): arr = i.split(",") waterArr = file2[idx].split(",") x = 0 for index, a in enumerate(arr): myObj = { "population": 0, "water": 0, } if int(a) > 9: myObj['population'] = 8 elif int(a) is 9: myObj['population'] = 7 elif int(a) is 8: myObj['population'] = 6 elif int(a) is 7: myObj['population'] = 5 elif int(a) is 6: myObj['population'] = 4 elif int(a) is 5: myObj['population'] = 3 elif int(a) is 4: myObj['population'] = 2 elif int(a) is 3: myObj['population'] = 1 elif int(a) < 3: myObj['population'] = 0 if waterArr[index] == '1': myObj['water'] = 1 array.append(myObj) with open('density2.json', 'w') as outfile: json.dump(array, outfile) Then I convert the data into json, and save it as a json file. The revised list of json data looks like this: ...{"water": 1, "population": 0}, {"water": 0, "population": 0}, {"water": 1, "population": 0}, {"water": 0, "population": 0}, {"water": 0, "population": 0}, {"water": 0, "population": 0}, {"water": 0, "population": 0}, {"water": 0, "population": 0}, {"water": 0, "population": 1}, {"water": 0, "population": 1}, {"water": 0, "population": 0}, {"water": 0, "population": 0}, {"water": 0, "population": 1}, {"water": 0, "population": 1}, {"water": 0, "population": 5}, {"water": 0, "population": 0}, {"water": 0, "population": 0}, {"water": 0, "population": 0},... Each element in the array contains data about both the population of that block, and whether or not the block is water (because we don’t want the zombies to wander into the ocean do we?) END OF PORTION 1 — — — — — — — — — — — — — — — — — - At this point, we’ve at least collected some data. We can always go back, using the file we’ve created, and enter new data about elevation, or population age, or whatever we think my be relevant. We’ll just need to give the new file a new name, so that we don’t overwrite a previous file. Then we can add that to our loop in python, and add the additional data to the JSON data. One thing to consider before moving on, is that, since this is a fairly small project, I’m not going to make additional files for different functions. I’m simply going to comment out the sections I don’t need. Since we’re moving from the first portion to the second portion of this project, we can comment out all of the code that relates to gathering, organizing or collecting the data. This includes the previous lines where we turn the text files into a hash, as well as the JavaScript where we create and submit the form data. However, if it’s easier for you to keep track of to have separate files, then you can do that. Just keep in mind that the file structure you have will look different than mine. PORTION 2 — — — — — — — — — — — — — — — — — - Now that we have this data, we’ll want to send it to the front end as json. We can do that through flask forms. In app.py, on the main route, we’ll just include a function that grabs that data from python. Like I mentioned earlier, I feel like it’s easier to just comment out the previous code we had in this file, which assembles our text file into JSON, and include these lines below the commented out section. But if you’d prefer to use a new file, that’s fine too : ------- IN TEST.PY ------------------- with open('density2.json') as f: popDensity = json.load(f) def getData(): return popDensity When our main route (‘/’) is called, we’ll call the getData function from the test.py module, and return that as output to be used in our JavaScript file. We’ll also modify the app.py route we created earlier, to accept and send this data to the front end. We’ll also include an import statement at the top, to import the module test.py that contains the function we want to use: from test import getData And we’ll add that function to the main route to get the data: @app.route('/', methods=['GET', 'POST']) def searchTopic(): output = getData() return render_template('base.html', result=output) 5 — Using the JSON data on the front end to affect the simulation. — — — — — — — — — — — — — — — — — - In our JavaScript file, we can retrieve the data by making use of flask: $(document).ready(function(){ var data = {{ result|tojson }}; Our variable data now contains the result that we passed in in our ‘/’ route function. My thought here is to encode each div with the JSON data in the form of element attributes. I’ll start with the attributes of “water”, “population” and “Direction”. Water and Population are attributes being passed in from python, and Direction will be something controlled completely from JavaScript. We’ll start out by creating those attributes so that we can use them in our elements: var population = document.createAttribute("population"); var direction = document.createAttribute("direction"); var isWater = document.createAttribute("isWater"); Our elements might look something like this: Now we don’t want to hard code each div, so we’ll do it in a loop instead. var directionArray = ['n', 'ne', 'e', 'se', 's', 'sw', 'w', 'nw']; for (var i = 0; i `; $('#body').append(inputBtn); } Again, hopefully we’ve commented out the JavaScript we used to create the data map, but this code will look pretty similar to what we did previously. Initially, when we were in the first portion of this project, and we use JavaScript to create our HTML divs, each div was blank. Now, in our for loop, we also assign attributes to each element before appending it to the #body. because zombies often behave erratically anyway, I’m going to randomly set their initial direction of travel at random. Here’s how we’ll determine the direction of the zombie later on: Implementing the Intervals — — — — — — — — — — — — — — — — — — — - If a particular block becomes infected, as shown above, and that block is div # 456, then there are 8 possible directions it can travel from it’s current position: n, ne, e, se, s, sw, w, nw. If we think of these divs as an array, then the direction of east would be the current position + 1. West would be the current position -1. north and south depends on the number of elements in each row. In my case, that’s -255 for north, and + 255 for south, and so on. If, at any given interval, the current infected block is not in a densely populated area, then it will move to a new block. Now, if we did nothing else at this point, the zombie direction is already set for each block. At each interval, we could check the direction on the given block, and then move it to the new corresponding space. However, let’s imagine we did that, and the current block has a direction of ‘ne’. Here’s what that would look like at each interval: Obviously, we probably want to randomize this a bit. Going back to our Interval that we talked about earlier, we’ll need to make the following decisions at each interval: What direction it moves, what speed it moves, and if it infects anyone. Our logic might look something like this: So now that we have some pseudo code, here’s some actual code. We’ll start with the onClick event, which will be the first event to trigger all the others: $('body').on('click', '.btn', function () { let color9 = colorArr[Math.floor(Math.random() * colorArr.length)]; $(this).attr('isActive', 1); $(this).removeClass('btn'); $(this).addClass('active'); $('.btn').click(false); }); On the click, we remove the class of btn, disable any other button clicks, and add the class of active. Now, when we get our setInterval function, it will pick up this block. Next, I’m going to make an array of two elements, a 1 and a 0: var changeArr = [0,1]; In our set interval function, each time it finds a block, we’ll pick one of these two numbers at random, like a coin toss. If we get a 1, we’ll change the direction attribute of the current block. If we get a 0, we’ll keep the same direction. setInterval(function(){ var buttons = $( ".active" ).toArray(); Inside the setInterval function, the first thing we’ll do is get an array of all the active buttons. At first, there will only be one. Next, we’ll get all the adjacent buttons, and put them in an array. These buttons will be our directional buttons, north, south, east, west etc… var currentID = $(buttons[i]).attr('id'); var id1 = parseInt(currentID)+1; var id2 = parseInt(currentID)+254; var id3 = parseInt(currentID)+255; var id4 = parseInt(currentID)+256; var id5 = currentID-1; var id6 = currentID-254; var id7 = currentID-255; var id8 = currentID-256; var adjacentbuttons = [$('#'+id1+''),$('#'+id5+''),$('#'+id7+''),$('#'+id8+''),$('#'+id6+''),$('#'+id3+''),$('#'+id4+''),$('#'+id2+'')]; Next, we’ll do something if the population density is over 10: if($(buttons[i]).attr('population') >= 10){ $(adjacentbuttons[2]).attr('isActive', 1); $(adjacentbuttons[2]).removeClass('btn'); $(adjacentbuttons[2]).addClass('active'); $(adjacentbuttons[4]).attr('isActive', 1); $(adjacentbuttons[4]).removeClass('btn'); $(adjacentbuttons[4]).addClass('active'); $(adjacentbuttons[0]).attr('isActive', 1); $(adjacentbuttons[0]).removeClass('btn'); $(adjacentbuttons[0]).addClass('active'); $(adjacentbuttons[6]).attr('isActive', 1); $(adjacentbuttons[6]).removeClass('btn'); $(adjacentbuttons[6]).addClass('active'); $(adjacentbuttons[5]).attr('isActive', 1); $(adjacentbuttons[5]).removeClass('btn'); $(adjacentbuttons[5]).addClass('active'); $(adjacentbuttons[1]).attr('isActive', 1); $(adjacentbuttons[1]).removeClass('btn'); $(adjacentbuttons[1]).addClass('active'); $(adjacentbuttons[3]).attr('isActive', 1); $(adjacentbuttons[3]).removeClass('btn'); $(adjacentbuttons[3]).addClass('active'); } We’re basically selecting several from the array, and turning them all to active. Essentially, it will infect a lot of other adjacent blocks if it stumbles into a highly populated area. We can add logic for other densities as well. For example: else if($(buttons[i]).attr('population') == 3){ $(adjacentbuttons[7]).attr('isActive', 1); $(adjacentbuttons[7]).removeClass('btn'); $(adjacentbuttons[7]).addClass('active'); } If the density is only 3, we only infect one other adjacent block. Direction — — — — — — — — — — — — — — Here’s the logic i’m using for each direction: if($(buttons[i]).attr('direction') == 'n'){ let change = changeArr[Math.floor(Math.random() * changeArr.length)]; if (change == 0){ $(buttons[i]).attr('isActive', 0); $(adjacentbuttons[2]).attr('isActive', 1); $(adjacentbuttons[2]).attr('direction', 'n'); $(adjacentbuttons[2]).removeClass('btn'); $(adjacentbuttons[2]).addClass('active'); $(buttons[i]).removeClass('active'); $(buttons[i]).addClass('btn'); } else if (change == 1 && $(buttons[i]).attr('isWater') != 1){ let newDirection = directionArray[Math.floor(Math.random() * directionArray.length)]; $(buttons[i]).attr('isActive', 0); $(adjacentbuttons[2]).attr('isActive', 1); $(adjacentbuttons[2]).attr('direction', newDirection); $(adjacentbuttons[2]).removeClass('btn'); $(adjacentbuttons[2]).addClass('active'); $(buttons[i]).removeClass('active'); $(buttons[i]).addClass('btn'); } else if ($(buttons[i]).attr('isWater') == 1){ $(buttons[i]).removeClass('active'); $(buttons[i]).addClass('btn'); } } Basically here, I’m looking for three things at each interval. I pick a random selection from the choice array. 2 — So does the direction change? If not, I send it on it’s merry way. 3 — If the direction does change, I select a new direction at random from our direction array, and use that as the new direction. 4 — Unless it’s water…then I set the button back to ‘btn’ instead of ‘active’ In conclusion: In all honesty, this is fairly bare bones, and is pretty basic. However, there’s a lot more that you could do with this. Even in this example, we could include data about elevation, which we haven’t yet done. We could include all other kinds of data. If you’re looking for a fun way to create a quick, interactive map, I hope this helps. My biggest concern is that the setInterval is a really resource heavy way to tackle this, and I’m sure there’s a much better solution. If you have any ideas or other feedback, feel free to share. Thanks!