Book - 5.5) Python - Layers of Abstraction

Mapping Layers of Abstraction using the Variable Explorer¶

Real-world data, like the data sets we are using for the project, can be quite complicated, often involving multiple combinations of dictionaries and lists. It is easy to get lost in the details of how the data is organized and how it can be accessed. This is often why the data is first filtered to produce a simpler data set to be used for further computation or visualization.

An aid to understanding the organization of a complex data set is a graphical depiction - a data map. The data map can be thought of an an x-ray of the data set, showing the skeletal structure of the data without the detail of the actual data and the syntax of the data structure; only the structure of the daa set is shown. Concretely, the map gives guidance on how to access the parts of the data of interest.

Before dealing with a project-scale degree of complexity a simpler example will be used to illustrate the basic techniques of constructing a data map. The example is the following combination of lists and dictionaries that represent a multi-city forecast of temperatures for the next three days. Following this simple(r) example a data map will be constructed for a more complex data set.

Multi_City_Forecast =  [ 
    { 'place': {'city' : "Blacksburg, VA", 'zip code' : 24061} , 'forecast' : [76, 72, 64] } , 
    { 'place': {'city' : "Seattle, WA",    'zip code' : 98101} , 'forecast' : [63, 63, 69] } , 
    { 'place': {'city' : "Miami, FL",      'zip code' : 33101} , 'forecast' : [86, 87, 88] } , 
    { 'place': {'city' : "San Jose, CA",   'zip code' : 95103} , 'forecast' : [79, 82, 83] } , 
    { 'place': {'city' : "New York, NY",   'zip code' : 10036} , 'forecast' : [85, 88, 92] }
]

Spyder's Variable explorer will be used to help construct a data map of the multi-city forecast data set. The figure below shows a portion of the Spyder window. The edit pane on the left contains the Python definition of the multi-city forecast data set. After the Run control is pressed the the Variable explorer window on the right contains a single entry.

Constructing a Data Map - Step 1

The entry in the Variable explorer window has four fields. The general meaning of these four fields is:

Name: the name of the Python property described by this entry
Type: the Python type
Size: the number of elements in the property
Value: the property's value

In this example these fields contain:

Name: Multi_City_Forecast (the name given to the data set in the Python program)
Type: list (because the first/top level of the data set is a list)
Size: 5 (because the data set has five elements in the list)
Value: the entire data set

The first element in the data map is used to represent this initial information about the data set.

This part of the data map shows the Name of the data set and the Type - in this case a list. Notice that all other detail about the list is ignored: neither the Size of the list is nor its Value is shown in the data map.

To build the next part of the data map we need to know the structure of each of the elements of the list. The Variable explorer can be used to see the list element. Clicking on the entry in the Variable explorer window produces a new Variable explorer window that shows its Value. This is shown in the following figure.

Constructing a Data Map - Step 2

This new Variable explorer window shows that each element of the list is a dictionary (note the Type field is "dict"). Clicking on one of the entries in this Variable explorer window will show another Variable explorer window with the details of each dictionary as shown in the next figure.

Constructing a Data Map - Step 3

From this Variable explorer window we can see that each dictionary in the list has two keys: forecast and place. We can now use this information to build the next part of the data map.

This data map should be read as follows: the Multi_City_Forecast data set is a list where each element is a dictionary with two keys: forecast and place. Note that the Python symbols denoting a dictionary - the "{" and "}" - are used in the data map to show a dictionary structure.

We can continue using the Variable explorer to show the structure of the value associated with the two keys forecast and place. Opening these two new Variable explorer windows by clicking on each entry is shown in the next figure.

Constructing a Data Map - Step 4

The left window (opened in response to clicking on the forecast element in the previous Variable explorer window) shows that the value associated with the forecast key is a list whose elements are of of type int (i.e., integers or whole numbers). The right window (opened in response to clicking on the place element in the previous Variable explorer window) shows that the value associated with the place key is a dictionary with two keys: city and zip code. Furthermore we can see in this window that the value associated with the city key is of type str (i.e., a string of characters) while the value associated with the key zip code is an int(eger).

All of this information can be collected into our final data map as shown in this diagram.

Two comments about the data maps. First, there is nothing special about the way these data maps have been drawn. A perfectly suitable data map might be drawn using other kinds of diagrams. The important thing is that we have some kind of representation to act as a guide when dealing with complex data structures. Second, the data map can be "read" to show exactly how to access a part of the data. For example, suppose that we wanted to output all of the zip codes in the data. The basic iteration accesses each element of the Multi_City_Forecast list one a time. Each such element is a complex entity, but reading the data map from top to bottom - following the arrows - yields this code.

for forecast in Multi_City_Forecast: 
    zip = forecast['place']['zip code'] 
    print(zip)

An example of constructing a data map for a more complex data set is shown in Chapter 7 (see Book - 7.1) Project - Project).