Accessing World Bank Data from Julia

While looking for ways to access the World Bank data bank programmatically, I found this very useful library to access it with Julia (along with many others for python and R). The library is WorldBankData.jl, by 4gh. Although the repo doesn’t have much activity, the library works great. Here are some examples of what can be done with the World Bank data, and the powerful Julia dataframes. I will also use Plots.jl for visualizations;

using WorldBankData
using Plots
using DataFrames

The World Bank offers two kind of data sources: Countries and Indicators. Indicators are the variables being measured in every country. Every year 13,398 indicators are measured in 258 countries. I have selected some countries to work with, since not all data from the world bank is complete. This filtering will facilitate our task:

countries = ["AW","AF","AO","AL","AD","AE","AR","AM","AS","AG","AU","AT","AZ","BI","BE","BJ","BF","BD","BG","BH","BS","BA","BY","BZ","BM","BO","BR","BB","BN","BT","BW","CF","CA","CH","CL","CN","CI","CM","CD","CG","CO","KM","CV","CR","CU","KY","CY","CZ","DE","DJ","DM","DK","DO","DZ","EC","EG","ER","ES","EE","ET","FI","FJ","FR","FO","FM","GA","GB","GE","GH","GN","GM","GW","GQ","GR","GD","GL","GT","GU","GY","HK","HN","HR","HT","HU","ID","IM","IN","IE","IR","IQ","IS","IL","IT","JM","JO","JP","KZ","KE","KG","KH","KI","KN","KR","XK","KW","LA","LB","LR","LY","LC","LI","LK","LS","LT","LU","LV","MO","MA","MC","MD","MG","MV","MX","MH","MK","ML","MT","MM","ME","MN","MP","MZ","MR","MU","MW","MY","NA","NC","NE","NG","NI","NL","NO","NP","NZ","OM","PK","PA","PE","PH","PW","PG","PL","PR","KP","PT","PY","PF","QA","RO","RU","RW","SA","SD","SN","SG","SB","SL","SV","SM","SO","RS","SS","ST","SR","SK","SI","SE","SZ","SC","SY","TC","TD","TG","TH","TJ","TM","TL","TO","TT","TN","TR","TV","TZ","UG","UA","UY","US","UZ","VC","VE","VI","VN","VU","WS","YE","ZA","ZM","ZW"];

A list of countries has to be given to the wdi() function, along with an indicator. This function queries the WBDB and returns a dataframe with all data available. If a year is added to the query, only the data for that year will be returned. Let’s work with total country population.

df=wdi("SP.POP.TOTL", countries, 2015)

Lets visualize the first 50 countries with a barplot.

bar(df[:iso2c][1:50], df[:SP_POP_TOTL][1:50], size=(900,400), label="Total Population")

The result is somehow expected. China is within the first 50 countries in the list, and it scales the graph for itself. Let’s find out exactly how much is big.

maximum(df[:SP_POP_TOTL])

1.37122e9

More than one billion people live only in china. This is almost 20% of the world’s population:

maximum(df[:SP_POP_TOTL])/sum(df[:SP_POP_TOTL])

0.18750300206223874

Right now we have the population for 208 countries. Almost all the countries measured by the WB.

size(df[:SP_POP_TOTL])

(208,)

In julia to find the index of the maximum value, the indmax() function can be used. This is useful if you want to know more about the element with the maximum value in a column.

indmax(df[:SP_POP_TOTL])

36

In fact, element number 36 in our dataframe is China:

df[:country][36]

"China"

Now let’s try something else. We want to see the most populated countries in the world. The first thought to come to mind is to sort the dataframe, right?

sort(df)

Well, not really. This function will sort only by the first column. To sort by other column we can add the cols argument to the function.

sort(df, cols=[:SP_POP_TOTL])

Just one more little detail. This last query has given us a dataframe with the smallest cities in the world. We want to revert the order of the sorting. This time I’ll use the sort!() function, to save the ordered dataframe into the df variable.

sort!(df, cols=[:SP_POP_TOTL], rev=true)

Ready! now our df variable is in order. Let’s try to visualize it again.

bar(df[:iso2c][1:50], df[:SP_POP_TOTL][1:50],
    size=(1000,400),
    label="Population Total")

There it is! Clean and tidy. In fact, let’s only use these 50 cities.

biggest = df[1:50,:]

Next we want to add another indicator: Female population percentage. This indicator has the code: ‘SP.POP.TOTL.FE.ZS’, and it represents the percentage of the total population that are registered as female.

df1 = wdi("SP.POP.TOTL.FE.ZS", countries, 2015)
println(size(df1))
println(size(df))

(191,4)
(208,4)

As you can see, the size of the two data frames we have right now are not the same. This is because some information is missing from the female percentage indicator. Not all indicators can be accurately measured in all countries. therefore, once in a while we can find missing values in them.

A workaround for this is to create a new dataframe with the countries we DO have the information for.

df = DataFrame(iso2c=[],country=[],SP_POP_TOTL=[],SP_POP_TOTL_FE_ZS=[])
for country in biggest[:iso2c]
    push!(df, @data([country,biggest[biggest[:iso2c].==country,:][:country][1],
                    biggest[biggest[:iso2c].==country,:][:SP_POP_TOTL][1],
                    df1[df1[:iso2c].==country,:][:SP_POP_TOTL_FE_ZS][1]]))
end
df[1:5,:]

See, now we have a dataframe with no missing values. Still, percentage of female population is actually a variable that depends on the total population. Let’s create a new dataframe with absolute values.

population = DataFrame(iso2c=df[:,1],
    country=df[:,2],
    total=df[:,3],
    females=df[:,3].*(df[:,4]./100),
    males=df[:,3].*(df[:,4].+(-100))./-100)
population[1:5,:]

Of course, this is now an approximation, but it will work for our visualization. Lastly, to plot these three variables we need to set the bounds to plot. This is done with the minimum() and maximum() functions.

minimum(population[:total])

2.5155317e7

Lastly, let’s create a visualization to describe all information we have acquired. Since there are three variables we have the option of plotting 3D points in space, but It will look clearer if we use the total population as color or size of a scatterplot. I like to use gr backend for my plots, but this code will work with almost all of Plots.jl backends.

gr()                                                                   # GR backend
plt = scatter(population[:,end],population[:,end-1],                   # Ploting male vs female
xlim=(minimum(population[:total])/2,maximum(population[:total])/1.07), # Limits for the X axis
ylim=(minimum(population[:total])/2,maximum(population[:total])/1.25), # Limits for the Y axis
size=(1000,1000), left_margin=[50px 0px],label="",                     # Window Size, lateral margins and label
ylabel="Female population",xlabel="Male Papulation",                   # X and Y labels
xscale=:log10, yscale=:log10, bottom_margin=50px,                      # Logarithmic scale and bottom margin
markersize=population[:,end-2]./minimum(population[:total]),           # The size of the marker is given by the total population
zcolor=map(log,convert(Array,population[:,end-2]./minimum(population[:total]))), # The color is given by the logarithm of the total population
grid=false)                                                            # NoGrid Science is art ;)

# Last a line is added to mark weather a coutry hase more male or female population
plot!(plt,x->x,minimum(population[:total])/2,maximum(population[:total])/2,label="", color=:green)