About Sajjad Anwar

Sajjad Anwar is a hacktivist and programmer based in Bangalore. He works in the research and design of data analytics and infographics. He hearts maps and often makes one. He is found working with other technologists, social activists and researchers in education, human rights and policy making. Find him on twitter.

Every project I have been involved thus far, I have helped people to ask the question – ‘Are maps really the right tool for us to tell this story?’ And I must say, there are not many people who are convinced. Maps are cool, they look nice, you can make them interactive, they may go viral (for good or bad), and yes, people like maps. Agree and that’s one of the many reasons why I love making maps and telling stories through them. If you do not ask the question, several things can go wrong.

I put together a repository to start gathering few examples of situations when maps go wrong. And spoke at an event in Bangalore and it was exciting. We will see some of those in this blog post. I am not intending to provide solutions to most of these, that will make a better blog post later. Broadly, there are six lists –

Misrepresentation of data

Careless handling of images and data can cause terrible mistakes, like the one below from the CNN a few weeks back.

CNN - Hong Kong is now in Brazil

Continue reading

Our tutorials so far have been focused on several aspects of cartography, from data structures to their analysis and representation. Not surprisingly, most of them are aligned to web technologies, and client browsers expect the applications to consume relatively low resources. Spatial data have comparatively higher memory footprints owing to the structure and the amount of information they hold. For instance, the taluk boundary level data for India is 46.3 MB in GeoJSON format. This means that it cannot be used directly in a web project; it needs to be optimised first.

Optimising spatial data essentially translates to simplifying the geometries in the file. Since, in a web context, it is not using it for analysis, a slight difference in the area or shape of corners will not make a huge difference. Users may not even realise that the shapes are simplified, if it is done in just the right way.

To give you an idea of the process, have a look at the following maps of Florida. The first row showcases the original data from the Florida Geographic Data Library, converted to GeoJSON (8.2MB). The second set of images shows the simplification of the geometry (note the sharp edges) in the GeoJSON (now 427KB). This really hasn’t changed the way the map looks on the whole, which is exactly what we need for web representation.

florida_combined florida_optimised_combined

In this article I will quickly look at a few easy methods to simplify geometries.

TopoJSON

TopoJSON, developed by Mike Bostock, is an extension of GeoJSON with encoded Topology.

Rather than representing geometries discretely, geometries in TopoJSON files are stitched together from shared line segments called arcs.

This simplifies the structure of the data by identifying the relationships and storing them in the same file, thus eliminating redundancy. TopoJSON works seamlessly with D3.js and can be integrated with pretty much any other web application.

Simplify using QGIS

The QGIS vector processing suite comes with a tool for simplifying geometries. It employs the popular Ramer–Douglas–Peucker algorithm which reduces the number of points in a curve. You have to select the layer that you want to simplify and pick a tolerance level. The higher the tolerance, the lesser the number of points and the lower the size of the file.

qgis_simplify

PostGIS ST_Simplify

In case you are serving spatial data from a PostgreSQL database through an API to the client-side, PostGIS implements the previously mentioned Ramer-Douglas-Peucker algorithm through the procedure called ST_Simplify. For example, to apply ST_Simplify on a geometry called ‘state’ of id 1, with a tolerance of 0.002 from a table called ‘country, the PostGIS command would be:

SELECT ST_Simplify(state, 0.002) from country where id=1;

These techniques are very essential when you deal with large amounts of spatial data that are required to be rendered in the browser. If you have more ideas or questions, let us know in the comments!

 

 

As a run-up to the Do-Din event in Hyderabad, geohackers.in is co-hosting an event called DataLore about putting data to good use, and how statistics and visualisations sometimes twist data to tell lies, this Wednesday, November 20th at 7 High Street Cooke Town, Bangalore.

People who want to make the world a better place look towards data in an effort to make that change. This very data then needs to be channeled into maps, statistics, and visualizations before it can be useful — and people are doing this everywhere. Stories of politics, corruption, oppression, and war are being told around the world using such tools. Unfortunately, a lot of what is being made fails at its task.  Maps that miss the point, visualizations that fail to engage, and statistics that mislead, all undermine action. On Wednesday evening, as a run-up to Do-Din, DataLore will attack this problem on two fronts:

You can’t just throw a map at a problem

Sajjad Anwar

When all you have is a hammer, everything starts to look like a nail. There are maps being made for every reason but some of them lack the point, they misrepresent information, they lie or they fail to engage the audience. We would like to discuss how people come up with these maps, what disasters they cause and how, as storytellers, we can improve the situation.

Nothing is what it seems — especially not statistics

The Ballot

 As they say, there’s lies, damned lies, and then there’s statistics. It’s easy to mislead or be misled by statistics and visualizations. Preconceptions and agendas can leak into them, and colour them with bias. Sometimes, a lack of knowledge about statistics leads to false conclusions, which is rather disastrous. We’ll use some examples to show you how this can happen, and how to both interpret and represent data properly.

We often find ourselves choosing between various data formats while dealing with spatial data. Consider this (not-so) hypothetical example: your data collection department passed on a bunch of KML files but your analysts insist on SHP files and your web team is very particular about their GeoJSON. If this sounds familiar, you’re reading the right post; we will quickly run through some of the popular vector and raster data formats you should care about and discuss some of the ways to convert data between these formats.

Vector

Shapefiles

The shapefile is perhaps the most popular spatial data format, introduced by Esri.

It is developed and regulated by Esri as a (mostly) open specification for data interoperability among Esri and other GIS software products. – Wikipedia

Esri still has the right to change the format when and if they choose to do so, it is otherwise open and is highly interoperable. Shapefiles can store all the commonly used spatial geometries (points, lines, polygons) along with the attributes to describe these features. Unlike other vector formats, a shapefile comes as a set of three or more files – the mandatory .shp, .shx, .dbf and the optional .prj file The .shp file holds the actual geometries, the .shx is an index which allows you to ‘seek’ the features in the shapefile, the .dbf file stores the attributes and the .prj file specifies the projection the geometries are stored in.
Continue reading

This tutorial is a proof of concept to use an HTML5 slider to control the opacity of a Leaflet map layer. If you want more information about setting up Leaflet and adding different layers, read the documentation.

We will start by adding two layers:

map.addLayer(stamen);
map.addLayer(mapquest);

And a slider:

<input id="slide" type="range" min="0" max="1" step="0.1" value="0.5" onchange="updateOpacity(this.value)">

The slider will invoke the function updateOpacity  when it is moved, which sets the opacity of the layer:

function updateOpacity(value) {
    mapquest.setOpacity(value);
}


If we want to change the opacity of the stamen layer, that’s possible too:

function updateOpacity(value) {
    stamen.setOpacity(value);
}

The code for the above example is here.

Web and mobile applications account for most of the maps usage today. We recently read that about 54% of smartphone users have Google Maps running on their phone making it the most popular application in the market. In the recent years, the technology behind web maps have improved considerably, owing to the incredibly fast and intuitive experience that we enjoy today. What we see, drag, scroll, touch, pinch and poke today is a set of map tiles.

The OpenStreetMap Wiki defines map tiles this way –

square bitmap graphics displayed in a grid arrangement to show a map

The fine folks at MapBox defined it this way –

tiles are typically 256×256 pixels and are placed side-by-side in order to create the illusion of a very large seamless image.

This technique of preparing and serving maps changed the way they are consumed drastically. Earlier, loading the map in a browser would take up so much memory that it was practically impossible to browse the map easily. Tiles make sure that only the required (usually the area which is currently viewed) have to be displayed on the browser, reducing the memory footprint. Even though Google got the usage of tiling right, they did not invent it. Web Map Service which came out in 1999 as an OGC standard set the web mapping revolution to a new level. WMS was slow for a lot of neogeographers. This frustration lead Anselm Hook to explore the idea of tiling the map for better performance.

The core of the application is a lightweight javascript application that runs in both Internet Explorer and Firefox. The approach is similar to the one taken at SVG Tile Engine which I wrote last summer. The difference is that this one talks to conventional WMS compliant mapping sources rather than a pre-tiled blue marble database and relies only on Javascript – not on SVG. This javascript engine is actually just a straight port of a java client based tile mapping engine which is visible at Java Tile Engine . The problem with the java applet approach however is that it cannot do cross domain image loading due to flaws in the security policy of java.

WMS-C was introduced following this idea to cache the map images which was super-ceded by the Tile Map Service by OSGeo.

Let’s take another step forward and see how the tiles work and how they are generated.

tilesWhen we view a map on the browser, there’s an immensely powerful feature – zoom. The world map at the least zoom level (level 0) is usually four square images which forms a grid of tiles. Every location on the earth is represented by a tuple with two elements – [latitude, longitude]. This, on your screen, translates to [x, y] which is the pixel coordinates. Zoom levels are incorporated to this data structure by adding one more element to the tuple – [latitude, longitude, zoom]. For instance, [12.9719, 77.5938, 12] is Bangalore at zoom level 12 and [12.9719, 77.5938, 15] is Bangalore at zoom level 15.

In your browser, the map is a collection of HTML image tags. This is achieved by using one of the various JavaScript map libraries like Leaflet.js, OpenLayers.js, or MapBox.js.

The geographic data in databases or shapefiles are rendered into the tiles through a process which involves several stages. We will quickly run through the most important and commonly used pipeline using a stack of open source softwares.

Mapnik is the de facto open source rendering library written in C++ that is used by large geographic data projects like OpenStreetMap to tiny map studios. Mapnik accepts a wide variety of input data – PostgreSQL databases, Shapefiles, GeoTIFF, and renders the data into set of map tiles depending on the style that you have developed. The styles are XML files which explain what Mapnik should do for each of the geographic feature (read tags) that it finds in the data source.

	<Style name="highways">
		<Rule>
			<Filter>[highway] &lt;&gt; ''</Filter>
			<LineSymbolizer>
				<CssParameter name="stroke">#808080</CssParameter>
				<CssParameter name="stroke-width">2</CssParameter>
				<CssParameter name="stroke-linejoin">round</CssParameter>
				<CssParameter name="stroke-linecap">round</CssParameter>
			</LineSymbolizer>
		</Rule>
		<Rule>
			<Filter>[highway] &lt;&gt; ''</Filter>
			<TextSymbolizer name="name" fontset_name="book-fonts"
				size="9" fill="#000" halo_radius="1" placement="line" />
		</Rule>
	</Style>

The above XML is one of the many style tags used by OpenStreetMap to render the tiles using Mapnik. This style tag refers to the highways that you see on the map. A style tag comprises of several Rules. A common technique is to apply CSS to the features that satisfy a rule and Mapnik will pick it up.

The tiles rendered by Mapnik are then served from what is called a Tile Server. The commonly used tools for a server is Apache with the mod_tile extension. I like the Python based server called TileStache. It’s fast and easy to setup. When the browser requests for a map tile, the server checks if the tile has been already rendered, if yes it is send to the browser. Otherwise, it is send to Mapnik for rendering.

We will discuss more about the configuration and best practices of setting up a rendering stack eventually in another blog post.

It has been a while since we started writing in a consistent pace. But somehow, I see that happening now. Today, we will see how to organize and align your data so that you can make a map or two out of it.

We often deal with data in CSV formats, which potentially can be visualized as a map. Let’s start with a sample file.

code district boys_appeared girls_appeared total_appeared boys_passed girls_passed total_passed pass_% rank
GA UDUPI 8013 8058 16071 6852 7537 14389 89.53 1
PA SIRSI 4582 4633 9215 3955 4183 8138 88.31 2
LL HASSAN 11783 11968 23751 9722 10685 20407 85.92 3
DD TUMKUR 12312 11085 23397 10305 9780 20085 85.84 4

The table above shows the first few rows from a CSV file containing SSLC results in Karanataka for the year 2012. You can download the complete file here. The contents of the file and what each row means is very evident from the column headers.

The column of interest for you right now should be ‘district’. We will now use this column to make a map from this data. The process of converting an address or part of an address to a geographic coordinate is called geocoding. We will geocode this data to find the latitude and longitude of the districts.

There are several ways of geocoding data – from free and easy APIs to comprehensive as well as expensive ones. Two of our favourites are: Batch Geocode and the MapBox Google Docs Geo plugin. We will use the second one for this exercise.

 

Continue reading

I recently rewrote the maps portal for the Karnataka Learning Partnership. The map is an important part of our project, action and process because it serves as the pivot point of navigation. I will quickly talk about the data and tools before we discuss the design aspects.

We have a fairly large dataset of schools in Karnataka. The name of the school, location, number of girls and boys etc. in a database. Fortunately, the data was clean and properly stored in a PostgreSQL database with PostGIS extensions. Most of my task was to modify the API to throw GeoJSON to the client using the ST_AsGeoJSON function and export the data.

We used the amazing Leaflet.js library and a wide range of plugins. Most of the UI elements are from Twitter’s Bootstrap. I cannot say that Leaflet and Bootstrap works well all the time, but in case you want to add something on the map, make sure that you use extend leaflet’s control layer. For instance, see how we added the Stop Drawing control.

We made several design decisions mostly inspired by the series of blog posts by Brian Timoney.

popup

Continue reading