How to Process Pandemics Data from WHO.int

This article and the repo referenced here is intended to help Data Scientists and data processing professionals to handle and manage data related to the COVID-19 disease outbreak. I hope this information helps researchers to find new ways of analysis to defeat the COVID-10 disease outbreak, as well as empower them to solve more complex math calc using less time. We all know how necessary is to have a trusted analysis of the situation. Today the most accurate data in terms of global information is coming from WHO.int. Unfortunately the use of the API explained here is not licensed on any form so you must use it at your own risk. The repo exposed here, the author and the tools used here are not related to WHO.int. DISCLAIMER: QCObjects Collab is licensed under LGPL v3 and the terms explained in this LICENSE :::notes Keep in mind these are not only numbers, they are lives, they are people that we wished to save but we couldn't! They are people that we want to save using the resulting data of the fault. As a researcher you must keep focus always in data but as a person you also have feelings and a soul. Please remember that when you show your analysis to the public. ::: Connecting to the API of WHO.int The API of WHO.int used on this repo has public access and it is not licensed in any form. You have to use it at your own risk, with no warranty of uptime. Try not to abuse of the requests made to this API, it is recommended to only do a request once a day and save the response to a temporary file to not overuse and/or cause overload. There is not any individual data or private data of people present here. Please use this information only for research purpose. Connecting to the API and extract data To connect to the API using cURL, create a shell file named and put inside the following code: ./get-covid-19-data.sh curl 'https://dashboards-dev.sprinklr.com/data/9043/global-covid19-who-gis.json' \ -H 'authority: dashboards-dev.sprinklr.com' \ -H 'accept: application/json, text/plain, */*' \ -H 'user-agent: QCObjects Collab for Data Science' \ -H 'origin: https://covid19.who.int' \ -H 'sec-fetch-site: cross-site' \ -H 'sec-fetch-mode: cors' \ -H 'sec-fetch-dest: empty' \ -H 'accept-language: es-ES,es;q=0.9,en;q=0.8' \ -H 'if-none-match: "23ffd2d307702788a8553819a1431d24"' \ -H 'if-modified-since: Mon, 18 May 2020 01:15:19 GMT' \ --compressed # file: ./get-covid-19-data.sh download the file ./get-covid-19-data.sh from here https://github.com/jeanmachuca/qcobjects-covid-19/blob/master/get-covid-19-data.sh Installing QCObjects Collab for Data Science To work with the data, you can use that is a Open Source Engine that helps Data Scientists to automate data processing and collaborative analysis tasks. QCObjects Collab for Data Science QCObjects Collab uses a collaborative shell to do advanced math calc using ssh or unix-socket. Installing QCObjects Collab from Docker To use QCObjects Collab from a Docker, you can pull the image from (this is the most stable running solution) quickcorp/qcobjects-collab docker pull -a quickcorp/qcobjects-collab && docker run -it --name qcobjects-collab --rm -it -p 10300:10300 quickcorp/qcobjects-collab Connecting to QCObjects Collab from external systems You can use a TCP socket to connect yourself to the engine: ssh nc user@ 0.0 .0 .0 0.0 .0 .0 10300 You can also use a Unix Socket to connect yourself to the engine: ssh user@0.0.0.0 nc -U /tmp/qcobjects-unix-socket (change "user" for whatever your username is!) If you are trying to access the server from the same machine you can use the IP 0.0.0.0 or if you're trying to access it from an external network terminal you must use the corresponding IP and default port is 10300. Start QCObjects Collab in your local machine using NPM/NPX To start using QCObjects Collab from your local machine without the need of starting a remote/cloud server, you need to have Node.js installed. Download and install Node.js from . here Once you have installed Node.js, you can use to call QCObjects Collab straight away in your shell console. npx > npx qcobjects-collab Start QCObjects Collab in your local machine using QCObjects If you either own a copy or have installed or you can run QCObjects Collab directly from a command line. QCObjects Community Edition QCObjects Enterprise Edition To install QCObjects just follow the steps here To start QCObjects Collab from a command line, after installing QCObjects, just do this: > qcobjects-collab Processing COVID-19 Data using QCObjects Collab Once you have done the above steps, everything turns easy and handy. Inside the QCObjects Collab session, you can follow these instructions. Run .help to view the available commands and shortcuts > qcobjects-collab QCObjects Collab> .help Run .loadcmd_json command to extract the collected data into a variable You can run the following command to extract the collected data from the response of executing the shell command into a variable stored in memory inside the QCObjects Collab session. ./get-covid-19-data.sh Enter to a QCObjects Collab session and do the following command: QCObjects Collab> .loadcmd_json covid = ./get-covid-19-data.sh The above command will save the response of the API into a global object called . Then you can use this object to process the data and make calc to reduce the result. covid Using examples of data processing The file inside this repo, called contains some quick start examples of processing the data. Here are some explanations of this examples: data-covid-processor.js Helper functions As QCObjects Collab is been written on JavaScript, you can enrich your runtime code using it, by example: creating runtime helper functions attached to the common Array prototype. Two useful functions to process data are (to collect only once for every single value element using ) and (to show a table representing the current list using ) unique objectList.unique() table objectList.table() .prototype.unique = { .filter( self.indexOf(value) === index)} .prototype.table = { .table( ) } //helper functions Array ( ) function return this ( ) => value,index,self Array ( ) function console this Formatting the columns The most efficient way of process a list in JavaScript is using the map function. You can use it into QCObjects Collab to process , and objects (ArrayList and ArrayCollection objects are a part of QCObjects core framework and QCObjects SDK, that are automatically loaded into the QCObjects Collab session). Array ArrayList ArrayCollection The API returns a tree object that contains three separate collections ( , and ) dimensions metrics rows |-- covid |-- dimensions |-- metrics |-- rows To view the content of the variable just enter its name into QCObjects Collab and press . covid Enter QCObjects Collab> covid QCObjects Collab> covid.dimensions QCObjects Collab> covid.metrics QCObjects Collab> covid.rows.slice( , ) 0 10 // only get 10 rows If you concat the elements of and in the same order as they are collected, you can have a column name dictionary. dimensions metrics To create a variable that stores the name of the columns of the COVID-19 WHO API data, do the following: columns columns = [ ].concat(covid.dimensions.map( dimension.name).concat( covid.metrics.map( metric.name))) // format columns "" => dimension => metric Changing the structure of the data using the columns collection. You can map and store it into another variable called which will contain the new structure based on . covid.rows data columns data = covid.rows.map( [{}].concat(row).reduce( { .assign(accumulator, { [columns[index]]: colData }) })) // format data using columns => row ( ) => accumulator, colData, index return Object Get a dictionary of country names from the API collected data To build a dictionary of country names you have to do the following: countries = data.map( d[ ]).sort().unique() // Obtain only unique country names => d "Country" The above code will create a collection with data object mapped with a lambda function (d => d["Country"]) that only returns the property of every element. Then, it will sort and filter the list to return only the unique elements. countries Country Calc the total amount of deaths in the world caused by COVID-19 disease The following code maps the data to only return the Deaths property of every element, then reduces the data to to a signle value containing the total amount of deaths accumulated in the world caused by COVID-19 disease. cumulativeDeaths = data.map( d[ ]).reduce( cumulative + deaths) // reduce the data to get total sum of deaths worldwide => d "Deaths" ( ) => cumulative, deaths Re-usable function to calc deaths by country The following code defines a function called that accepts one parameter with the country value to filter the list, then map the filtered result to only get the property on every element and reduce the list to a single value with the sum. getCumulativeDeathsByCountry Deaths getCumulativeDeathsByCountry = data.filter( d[ ]===country).map( d[ ]).reduce( cumulative + deaths) // a helper function to calculate deaths by country ( ) => country => d "Country" => d "Deaths" ( ) => cumulative, deaths To use this function you can do the following: cumulativeDeathsOfUSA = getCumulativeDeathsByCountry( ) "US" Get the summary of deaths by country The following code will create a single object with the summary of deaths by every country on the data cumulativeDeathsByCountryObject = countries.map( { {[country]:getCumulativeDeathsByCountry(country)}}).reduce ( .assign(prev,current)) // calculate cumulative deaths by every country => country return ( ) => prev,current Object To summarise the data into a list of objects with the pair country -> amount , do the following: cumulativeDeathsByCountry = countries.map( { { :country, :getCumulativeDeathsByCountry(country)}}) // calculate cumulative deaths by every country => country return "Country" "Deaths" orderedDeathsByCountry = cumulativeDeathsByCountry.sort( current.Deaths > prev.Deaths ? : ) // order the list of deaths by country by descendent (most deaths first) ( ) => prev,current 1 -1 You can find a complete example of the data processing in this file: https://github.com/jeanmachuca/qcobjects-covid-19/blob/master/data-covid-processor.js The complete example repo for this tutorial is here: https://github.com/jeanmachuca/qcobjects-covid-19 Wash your hands, keep social distancing... Be safe! Previously published at https://github.com/jeanmachuca/qcobjects-covid-19/blob/master/README.md