Connecting data points through time (or not)

Answering the question “who was where when?” is central for investigations into allegations of human rights abuse(s). Because of this perhaps one of the most defining, and complicaticating, features of the Security Force Monitor’s data is that almost everything we research is connected to time including:

  • Existence of units
  • Parent relationships between units
  • Location of units
  • Areas of operation for units
  • Membership/participation of units of in multi-unit operations
  • Positions held by people

While attaching time to data points aids our mission to support human rights investigations and advocacy, it raises methodological challenges. The Security Force Monitor has developed a methodology to address the issue of time which this blog will lay out in detail including:

  • Why the Monitor would (or would not) connect two bits of data through time
  • How the Monitor handles gaps in the public record
  • Questions analysts run through while reviewing time based information

As always – questions, comments and any other feedback is welcome!

Fragmentary nature of time in sources

In an ideal world the Monitor would have a source from every day of the year stating where a unit was located or conducting operations. Barring that having multiple sources regularly making statements like “since X date this unit has been based in this city” would be tremendously helpful. Unfortunately, neither scenario currently occurs, or is likely to occur in the near future, making it necessary to develop a robust way of thinking through time.

Broadly speaking the Security Force Monitor uses agreement among sources to build up details on security force units and individuals. Most of the Monitor’s sources, like government press releases and newspaper articles, can be used to link a value, such as the location of a unit, to a specific date (usually the date of publication). As we collect more sources we need to determine what agreement among sources means for time based values, like the location of a unit.

Connecting through time or not

Example: the Monitor comes across Source A published on 1 July 2012 stating that the 1 Battalion is based in Lagos. If Source B published on 3 August 2012 also states that the 1 Battalion is based in Lagos we have a decision point about what claim we should make.

Utilizing sources A and B we have two options which can be expressed in text:

  1. Separate claims: “As of 1 July 2012 the 1 Battalion was based in Lagos and as of 3 August 2012 the 1 Battalion was based in Lagos, the Monitor does not know where the battalion was based between those two points in time.”
  2. Contiguity claim: “From at least 1 July 2012 to at least 3 August 2012 the 1 Battalion was based in Lagos.”

Thus, whenever the Monitor gets a new source of information we have to decide whether to make a “separate” or “contiguity” claim. Based on the example of the 1 Battalion above the Monitor would run through a series of questions to determine which claim (if any) to make:

  • In general, how do other battalions operate, are they sedentary, or highly mobile?
  • How has the 1 Battalion acted in the past, has it been sedentary or highly mobile?
  • Are there other sources disputing these claims (i.e. 1 Battalion being based solely in another city)?
  • Are there any sources indicating the 1 Battalion was in Lagos in July and/or August as part of a “special”, “emergency” or otherwise temporary posting?
  • Are there sources that indicate the 1 Battalion moved in between these two points of time and thus these should be treated as separate deployments to Lagos?
  • Is there anything related to the 1 Battalion’s parent or child units that may impact where it was based?
  • Are there any other mitigating sources (i.e. major restructuring of the military, constitutional changes, etc.) which may impact the basing of the unit?
  • Is more research needed before the Monitor can make any claim?

An argument could be the Monitor should always make “separate claims” as that would be more faithful to the sources. However, the result likely mean an almost incomprehensible amount of detail in the records of people and units, which would obscure when changes really did occur, for instance when a person changed positions or a unit ends operations in an area.

Perhaps the most important point is that it even though data points, like where a unit is based, can be continuous through time, it should never be assumed that those types of features remain consistent between two or more sources. Time is a constant challenge, but given that is a key element in identifying perpetrators of human rights abuses it is necessary to get it right.

Learning from our users – first feedback on our prototype

In mid-April we publicly released the first version of our web application for feedback. We sought out advice from human rights researchers, international criminal litigators, investigative journalists, and policy advocates – spending over an hour with more than 45 users. Our whole team was struck by the willingness of our colleagues to dedicate time to giving us feedback. To everyone that took the time to talk with us – thank you!

This post will cover what we’ve learned from our users, how their feedback helps our mission, and why we’ll be reaching out again shortly.

Interviewees really liked… Interviewees had questions about …
  • The content itself
  • Charts showing command structures
  • Being able to access all the sources
  • Ways of going forward and backwards in time
  • Background information about security forces in a country
  • Showing analysis rather than just data
  • Visual clutter and data density
  • How to find data quickly
  • Downloading our information
  • Not knowing how data was selected for inclusion
  • Completeness of the data
  • Slow application responsiveness

We are answering the right questions, but not always in the right way

Users strongly validated the premise of our work – they very much wanted (and generally found it hard to find) information on the organizational structure, command personnel, location and areas of operation of security forces tracked through time. Interviewees could see the value of the dataset – in part or as a whole – to their own work. They were also intrigued by the visuals on offer, liking our ambition.

Our visualization of the command chart was a huge success, users loved being able to see that information in a visual way. Users had extremely positive views about actual data. They got glimpses its value, in particular where we included the analysis we can produce using the data. For example, for each alleged human rights incident we included a list of nearby units, which numerous interviewees found clarified the overall purpose of our research. They also liked the ability to see sources for each datapoint, which would help them appraise the data for inclusion in their own work. While users did not find the timeline functionality intuitive to use, when we explained how they could “time travel” through our data to see the command tree, commanders and other data at a particular point in time they were thrilled.

The biggest issue for users was that they were often overwhelmed with information, particularly in the initial map view of country. This made it difficult to grasp what they were looking at, or where to go next. There were many questions related to our terminology (what do we mean by area of operations, affiliated person, etc). Users often had difficulty navigating around the application from page to page, and our tools to sort through the visuals (like the filters on the map page) were not intuitive.

Users Want More of This Users Want Less of This
moreofthis lessofthis

What we will do next

The 45 people we interviewed told us straight what they liked and did not about our work so far They have given us good guidance on the direction we need to take.

  • Make it far, far simpler to use – less of a stand-alone application and more of a webpage
  • Simplify the presentation of key information, using visualization more sparingly, offering contextual guidance where needed.
  • Make it ‘search first’, giving lots of ways of find, sort and filter throughout the application.
  • Speed it up, and give lots of cues about how user things change things when the user does something.
  • Let people take the data home.

We have taken the feedback users have given us and will launching a radically new and improved application in the coming months – so stay tuned for more updates!

Why I started the Security Force Monitor

by Tony Wilson.

Today, I’m sharing something with you that I’m proud of.  It’s a new research tool we’ve created – an early version that shows our research, and embodies our reason for being.  It brings to light something we have never seen so clearly in one place before: the structure and operations of security forces, surfaced and arranged from thousands of publicly-available sources.

This slideshow requires JavaScript.

You can visit our prototype online here. Currently it covers security forces in Mexico and Nigeria – over 1,000 discrete organizations and nearly 900 affiliated persons.

How could all this information not already exist?

I started the Security Force Monitor in order to address a simple problem – the lack of detailed information about the police, military and other security forces of a country. At the time, I was trying to help advocates raise human rights concerns about U.S. security assistance to Bahrain. Since the protests that began in February 2011, rampant human rights abuses had been documented, but it was incredibly difficult for human rights researchers to identify specific perpetrators because the security forces were not transparent. This was extremely frustrating since it made advocacy supporting human rights conditionality on security assistance even harder. So, the task was clear: find detailed information about the security forces of Bahrain.

After several weeks of pilot research together with a colleague we had found data from hundreds of sources, and compiled them into an ever-lengthening Word document. This initial research demonstrated the information gap could be filled, but just as quickly a second problem arose – making sense of large amounts of quite detailed data. Using the limited skills I had, I created a rough Google Map and a rudimentary organizational chart, in an effort to make sense of all that data.

This slideshow requires JavaScript.

Even with these basic tools, I could see compelling connections between alleged human rights abuses and specific units and commanders. But the limitations were also evident. It was clear to me that in order to be a sustained effort the Security Force Monitor could not just work off of a text document or even a spreadsheet. It would need professionally-developed tools that a team could use to make accurate data easy to create, and a way of publishing it that would aid others in their own investigations.

Armed with some sketches of what a potential platform could look like, I interviewed almost 90 journalists, advocates, human rights researchers and others engaged in public interest efforts and asked them what would be useful to them and their work.

A capture from initial sketches for an application, showing a command tree, and a time slider

Their feedback guided the genesis of Security Force Monitor.  I have been fortunate to gain the support of the Open Society Foundations and the Oak Foundation, win the Knight News Challenge on Data, and pull together a great team – Tom Longley and Michel E. Manzur. The Security Force Monitor also found a welcoming institutional home at Columbia Law School Human Rights Institute and has begun to build an exceptional Advisory Council for the project. To move from concept to tool we have worked with the creative civic technologists at DataMade, FFunction and OpenNorth. Together we worked to create the datasets and produce our first attempt at a product, the prototype that we are releasing today.

The picture it shows of security forces is rich and detailed, and changes with time. The prototype platform shows the changes that occurred over time as units were created, moved or disbanded; and as commanders were promoted, retired or fired. Finally, all our work is transparent: every data point is sourced with citations back to where we got our data.

This prototype is a big first step. When we were getting our idea off the ground we talked with almost 90 journalists, activists, researchers and policy experts. Now, with this prototype in hand, we will be conducting even more interviews over the next several weeks on what does and does not work in order to develop an even better version.

You may be getting an email from us very soon. In fact, if you have thoughts and feedback, email me directly – tony [at]

Our mission is simple but bold. The Security Force Monitor will organize every piece of public information about security forces. We’ll produce research of the highest quality that will help make security forces more transparent. I believe that producing this research will aid journalists, civil society, human rights researchers, oversight efforts and others in making security forces more accountable.

You can help us do this. Take the first step by using our prototype and telling us what you think. Working together we can make the Security Force Monitor an indispensable digital service for transparency, accountability and human rights.