Book - 5.6) Python - Visualizing Big Data

A large software library, like matplotlib, may have dozens or hundreds of reusable components - called functions - that are available in building new programs. In many programs only some of these functions are needed. To make the distribution and reuse easier the functions are usually grouped together, each group consisting of functions that are more likely to be needed together. In Python these groups of functions are called modules. We have already seen some examples of using the import facility in Python to make the functions in a module available for (re)use. Several modules may themselves be further grouped together in a bigger collection. In Python this bigger collection is called a package.

Matplotlib is a package containing a number of modules. One such module is the pyplot module. The pyplot module has many functions related to constructing 2D graphs and charts. Some of the functions in the pyplot module are the plot and show. This organization is shown in the following figure. Some other functions we will see in the pyplot module are the  title, xlabel, and ylabel functions. We will see these functions used later.

Organizing Packages, Modules, and Functions

The scheme used in Python to name a function in a package or module is called a “dotted notation”. In this notation, the full name for the plot function is:

matplotlib.pyplot.show

However, Python provides a way to introduce a “nickname” for a module. This form of import is useful if there are several functions in a model that will be used. The nickname is given in the import statement as shown in this example:

import matplotlib.pyplot as plt 

plt.plot(...)    # plot some data 
plt.title(...)   # add a title 
plot.show(...)   # show the data in a window

where plt is the “nickname” defined in this program for the matplotlib.pyplot module.

If only a single (or a few) function(s) from a module are to be used then only that (those) function(s) can be imported. For example, we will see a package named mpl_toolkits that contains code for generating geographic plots. This package has a module named basemap, one of whose functions is named Basemap (note the difference in capitalization between the module name and the function name). In the code example to be seen, only the Basemap function is needed.

from mpl_toolkits.basemap import Basemap

# use the Basemap function as

map = Basemap(...)   #generate a geographic map to plot data on

It is important to remember that the from...import... only provides access to the Basemap function and not any other functions in the mpl_toolkits.basemap module.

Matplotlib examples

Line Graph¶

A common data exploration question is whether there is any relationship of the data with time. For earthquakes we might want to see if there is any pattern, for example, where the magnitudes of the earthquakes rises over time and then decreases. To answer this question a ling graph can be created where the horizontal axis (x-axis) is time and the vertical axis (y-axis) is the magnitude of the earthquake. A related question is how the data relates to the average value of the data.

The following code uses matplotlib to produce a line graph of the magnitudes of the earthquakes. For visibility successive data points are connected by a line. A line representing the average value of the magnitudes is also drawn.

Demo-Line-Graph.py Download Demo-Line-Graph.py

# Line Graph Example

import matplotlib.pyplot as plt

magnitudes = [
              1.98, 1.8, 1.8, 1.7, 3.1, 2.0, 1.8, 0.6, 2.8, 0.5,
              2.51, 3.51, 1.4, 2.6, 2.5, 2.4, 2.9, 2.6, 2.8, 2.5,
              ...]

# Compute average magnitude
avMagnitude = sum(magnitudes)/len(magnitudes)

# Plot magnitudes as a line graph
plt.plot(magnitudes)

# Add line for average magnitude: a line from (0,avMagnitude)
#                                       to (len(magnitudes), avMagnitude)
plt.plot([0, len(magnitudes)] , [avMagnitude, avMagnitude])

# Label Axes and figure
plt.xlabel('Time')
plt.ylabel('Magnitude')
plt.title('History of Magnitudes')

# Display histogram
plt.show()

# Clear before the next graph
plt.clf()

In general, the pyplot functions are used to build up a visualization. The visualization is then displayed in a separate window. The Python program waits until this window is closed before proceeding so that a number of visualizations can be constructed and viewed in sequence.

In the above code, the import statement is used to give a nickname for the matplotlib.pyplot module. The builtin sum and len functions are used to compute the average magnitude. The visualization is constructed by the following steps:

• the plot function is used to produce a line plot using the list of magnitudes.
• the plot function is used again using a list with two data points defining the endpoints of a straight line at the average value on the y-axis.
• the xlable and ylable functions add explanatory text labels to each axis.
• the display as a whole is given a title.

After the visualization has been constructed it can be displayed in a separate window using the show function. The program pauses at this point until the separate visualization window is closed. Finally, the clf function clears the internal structures that matplotlib uses to represent the visualization.

The visualization produced by the line graph code is shown below.

The Line Graph Visualization

 

This visualization shows three things:

• there is no apparent pattern in the magnitudes over time
• there is considerable variability of magnitudes from the average
• earthquakes of the highest magnitudes are rare

This shows the value of even simple visualizations.

The window displayed by matplotlib has a number of controls in the lower left-hand corner. Holding the cursor over any of the controls gives a brief description of that control. For the simple visualization we will be using three control are useful. These are:

• right-most control saves the visualization using any one of a variety of standard formats.
• the second control from the right is a zoom control. Pushing this control changes the cursor in the visualization window to a “+” symbol using which a rectangle can be selected. The display will zoom in on this region.
• the third control from the right is a pan-zoom control. Pushing this control changes the cursor in the visualization window to a symbol with two crossed double arrow heads. With the left mouse button down the display can be moved around. With the right mouse button down the display can be compressed and stretch horizontally or vertically.
• the left-most control returns the visualization to its original state.

You should run the line graph code and experiment with these controls.

Exercise. Add to the Line Chart program so that the visualization shows not only a line for the average but two additional lines - one a standard deviation above the average and one a standard deviation below the average. Note: the numpy module has a function std that computes the standard deviation of a list of numbers. Give the code and provide a .png image of the result.

Histogram¶

The line chart shows that there is a significant variation in the magnitudes. However, it is difficult to see from the line chart the distribution of the magnitudes. A histogram is a simple visualization of the distribution of data. In the code below the matplotlib function hist is used visualize the distribution of the earthquake magnitudes.

The code for this example can be downloaded from here:

Demo-Histogram.py Download Demo-Histogram.py

 # Histogram Example

 import matplotlib.pyplot as plt


 # Sampled earthquake magnitudes from real data stream

 magnitudes = [
       1.98, 1.8, 1.8, 1.7, 3.1, 2.0, 1.8, 0.6, 2.8, 0.5,
       2.51, 3.51, 1.4, 2.6, 2.5, 2.4, 2.9, 2.6, 2.8, 2.5,
       ...]


# Plot histogram of magnitudes
plt.hist(magnitudes, bins=[0,1,2,3,4,5,6,7])

# Label Axis
plt.xlabel('Magnitudes')
plt.ylabel('Occurrences')
plt.title('Histogram of Magnitudes')

# Display histogram
plt.show()

# Clear before the next set of graphs
plt.clf()

Much of the code in the histogram was described in the earlier example. What is new in this example is the hist function. The hist function call the first parameter is the list of magnitudes and the second parameter is for a parameter names bins. The bins parameter allows the caller to specify the categories for the distribution. Earthquake magnitudes are measured on the Richter scale. We can see from the line graph that the highest value in our data set is between 6 and 7. The bins value given in the histogram call specifies that the magnitude data should be grouped into the categories 0-1, 1-2, ..., 6-7.

The histogram produced by the example program is shown next.

The Histogram Visualization 

This visualization shows:

• the distribution is highly skewed with the vast majority of the magnitudes at or below magnitude 2
• there are no earthquakes (at least in this data) in the 5-6 category
• the most serious earthquakes (those in category 6) are rare

We now know something about the nature of earthquake magnitudes. But we would also like to know something about the characteristics of where earthquakes occur. This is what will be seen next.

Scatter Plot¶

A question for many data sets is whether two items are related to each other in some way, that is, are they correlated. In our case, we might be interested in knowing whether there is any relationship between the latitude and longitude of earthquakes. In other words, do earthquakes occur “all over the place” or are there certain locations that have a higher incidence of earthquakes.

To examine questions of correlation between two sets of data a scatter plot can be used. One set of data is treated as coordinates on the horizontal axis (x-axis) and the other set of data is treated as coordinates on the vertical axis (y-axis). A point is plotted in the scatter plot at (x[i], y[i]) where x[i] is an element of one data set and y[i] is the corresponding element of the other data set.

A random arrangements of points on the scatter plot means that there is no correlation. Clusters of points or patterns in the arrangements of point are visual evidence of a possible correlation. Statistical methods must be used to determine whether the apparent correlation is significant more merely random chance.

A scatter plot of the longitude and latitude of earthquakes is shown in the following code sample.

The code for this example can be downloaded from here:

Demo-Scatter-Plot.py Download Demo-Scatter-Plot.py

# Scatter Plot

import matplotlib.pyplot as plt


# Sampled earthquake latitude and longitudes from real data stream
# Earthquake i occurred at the location (latitude[i], longitude[i])


latitudes = [
              39.6835 ,  60.5828 ,  47.4227 ,  38.8175 ,  19.1345 ,
              19.4377 ,  60.0217 ,  38.7978 ,  19.1364 ,  38.8148 ,
            ... ]

longitudes = [
              -119.8202 ,  -151.0144 ,  -120.1962 ,  -122.8042 ,  -66.4852 ,
              -155.2545 ,  -152.0147 ,  -122.741 ,  -66.476 ,  -122.8202 ,
             ...]


# Generate scatter plot of locations of earthquakes on a 2D grid
# At each earthquake location (longitude[i], latitude[i]) put a red '+' sign

plt.scatter(longitudes, latitudes, c='r', marker='+')


# Label Axes
plt.xlabel('Longitude')
plt.ylabel('Latitude')
plt.title('Earthquake Occurrences')
plt.show()

# Clear before the next set of graphs
plt.clf()

Most of the code in the scatter plot example should be familiar from the earlier examples. The new element is the use of the pyplot.scatter function to generate the scatter plot itself. The two required parameters are the two lists of data which are, in our case, the longitude and latitude of each earthquake. There are two optional parameters used. The parameter named c allows the caller to specify what color to use for the points that are plotted. A red color is represented by the letter ‘r’. The second optional parameter, named marker allows the caller to specify what symbol to use for a plotted point. In this case a ‘+’ symbol is used.

The scatter plot visualization produced by the example program is shown below. It is clear from this visualization that there are definite clusters in the data. This means that the earthquakes described in our sample data appear to have a tendency to occur in some places more than other.

The Scatter Plot Visualization 

If we use the pan-zoom controls on the matplotlib display window we can see how localized the clustering of earthquakes is. The following figure shows a close-up view of the cluster of earthquakes in the upper left hand corner.

Enlarged View of the Scatter Plot Area

We can see in the enlarged view of the earthquake cluster that a sizable number of earthquakes occur in the region between -160 and -140 degrees of longitude and between 60 and 65 degree of latitude. The other clusters of earthquakes could be explored in a similar way.

This scatter plot does not tell us, however, what relationship there might be between these clusters and the magnitudes of the earthquakes. Do the clusters contain earthquakes that are frequent but have small magnitudes and the large magnitude earthquakes happen in other locations? or Do the larger magnitude earthquakes also occur in these clusters? Additional analysis of the data would be needed to answer this question.

The scatter plot above also does not show us very clearly where on the globe the earthquakes are occurring. This is especially interesting for the clusters that we have seen. The map plots below answer this question.

Bar Chart¶

A bar chart is used to compare different categories of data. It is called a bar chart (or bar graph) because the graph consists of a set of bars, each bar denoting a different category. The characteristic of each category is represented by some quantity which determines the height of the bar for that category. The width of the bar is arbitrary and can changed to improve the visual appearance.

The example code shown below generates a bar chart related to the earthquake data. A computational analysis (not shown) of the earthquake data shows that most of the earthquakes in the United States occur in either California or Alaska. The question to be answer is "How do the average magnitudes of the earthquakes in California and Alaska compare to each other and to the average magnitude of all earthquakes?" The bar chart depicts three categories: earthquakes in California, earthquakes in Alaska, and all earthquakes. The quantitative characteristic of each category is the category's average magnitude. 

The code for this example can be downloaded from here:

 Demo-Bar-Chart.py Download Demo-Bar-Chart.py

# Bar Chart Example

import matplotlib.pyplot as plt

# Analysis of the earthquake data set determined the following
# average magnitudes of earthquakes in California and Alaska as
# well as the average magnitude of all earthquakes.

average_magnitudes = [ 0.97, 1.63, 1.44]
labels = ["California", "Alaska", "All"]
x_positions = [0,1,2]

# Generate a bar graph comparing the average magnitudes
# Each bar and its label is centered at the x_positions locations
# on the horizontal axis

plt.bar(x_positions, average_magnitudes, align='center')
plt.xticks(x_positions, labels)

# Label the bar chart

plt.xlabel("Locations")
plt.ylabel("Average Magnitude")
plt.title("Comparison of Average Magnitudes")

# display the bar chart
plt.show()
plt.clf()

In this code, the bar function is used to generate the bar chart. The x_positions list determines where on the horizontal axis (the x-axis) each bar will appear. These positions are arbitrary and can be changed to alter the physical appearance of the visualization. Each element of the average_magnitudes list gives the average magnitude of the earthquakes in each of the three categories. Each value determines the height of the bar corresponding to that category. The parameter align='center' indicates that each bar and its label should be centered at the horizontal location specified for that bar. The xticks function is used to specify the label information for each bar.  The labels list contains the label for each bar. The x_positions list is the same as used in the bar function. Note, that each label will be centered at its specified location because the alignment specified in the bar function also applies to the labels. Finally, the labeling of the axis and the visualization itself is done and then the complete visualization is shown.

The bar chart produced by this example code is shown below.

A Bar Chart of Earthquake Magnitudes

Map Plotting¶

A visualization often needs to depict data on some form of geographic map. In our case we are plotting earthquake data which can be plotted on a globe or country map to show where in the world or country the earthquakes occurred. There are many other cases where geographic mapping is useful. Visualizations with geographic maps is a good way to display voter data, epidemiology data, weather data, population data, natural resource data, and others.

Latitude and longitude coordinates identify a point on the spherical globe. Making a 2D visualization of these points is slightly tricky. As an analogy, imagine drawing a dots with a magic marker on an orange. Making a 2D map is like peeling the rind from the orange and laying out the rind pieces flat. How do do this so that points on the surface of the orange are easily recognizable. Sometimes its easier to see the pattern in the points if we “stretch” the rind to form a more uniform surface. But this stretching also distorts the distance between points. Cartographers, people who make maps, have devised a number of different ways to “project” the surface of a 3D spherical globe onto a flat 2D plane. Here is more information on map projections . The example developed below uses a Hammer projection, named for Ernst Hammer who developed it in 1892. The Hammer projection looks like this:

The Hammer projection.

 

The example code below does map projections using the Basemap function. This function is part of the mpl_toolkits package*. This package contains a number of modules the are built “on top of” matplotlib and extend the functionality of matplotlib in a variety of specific application areas. The basemap module contains the function Basemap (note the difference in capitalization between the module name and the function name). The Basemap function implements a variety of map projections, like the Hammer projection. The set of available projections is described in the Basemap documentation .

The key steps in generating a globe-based visualization of data are:

1. use Basemap to define a map projection,
2. use the map projection to convert latitude and longitude coordinates into the corresponding coordinates in that map projection, and
3. use the matplotlib.scatter function to plot the transformed coordinates on the map

These there steps are marked by comments in the code below.

The complete code can be downloaded here.

Demo-Map-Plot-Hammer.py Download Demo-Map-Plot-Hammer.py

 # Map plot using the Hammer projection

 import matplotlib.pyplot as plt
 from mpl_toolkits.basemap import Basemap


# Sampled earthquake latitude and longitudes from real data stream
# Earthquake i occurred at the location (latitude[i], longitude[i])


latitudes = [
              39.6835 ,  60.5828 ,  47.4227 ,  38.8175 ,  19.1345 ,
              19.4377 ,  60.0217 ,  38.7978 ,  19.1364 ,  38.8148 ,
            ... ]

longitudes = [
              -119.8202 ,  -151.0144 ,  -120.1962 ,  -122.8042 ,  -66.4852 ,
              -155.2545 ,  -152.0147 ,  -122.741 ,  -66.476 ,  -122.8202 ,
             ...]

# Create world map with a Hammer (eliptical, equal-area) projection
# centered at 180 degrees longitude (i.e. longitude zero of the
# visualization is longitude 180 on the globe)
map = Basemap(projection='hammer',lon_0=180)   # Step 1

# Fill in the outlines of continents
map.drawcoastlines()

# Convert latitudes and longitudes to coordinates on the world map
x, y = map(longitudes,latitudes)              # Step 2

# Map the locations onto the world map
map.scatter(x, y)                             # Step 3

plt.title('Locations of earthquakes')
plt.show()

plt.clf()

Notice that the Basemap function returns a map that encodes the details of how to perform the projection (Step 1). This map is used (Step 2) to project its two parameters (the latitude and longitude data) and returns two values (the projected x and y coordinates). These projected coordinates are used as input to the scatter plot function (Step 3). To make the visualization easier to understand the drawcoastlines functions is used to show the continents. Here is the resulting visualization.

Earthquake data plotted on a globe map (Hammer projection)

 

The globe visualization shows that the two clusters of earthquakes seen earlier are located on the coasts of California and Alaska. In general, it can be seen that the majority of the earthquakes are in the Pacific rim. Much more extensive data would be needed to study in more detail the distribution of earthquakes.