We’ve been working hard on a permissions overhaul that touches every page of the DevResults app. Now we’re ready to release project-specific permissions, providing you with a huge increase in the flexibility and specificity of user permissions.

Since the dawn of (DevResults) time, we’ve lauded the benefits of broad data access to our customers. When colleagues can look at each others’ work, even if it’s not strictly speaking “their” project, they can answer their questions faster and get the data they need easier. Some people worry about the risk that someone may inadvertently enter data or delete documents for the wrong project, but in practice those mistakes are rare and are almost always caught immediately after the fact or in data review.

Even so, there are lots of valid reasons to be more particular about permissions across projects:

• Some projects may contain sensitive data or information about vulnerable people or partners.
• Some projects may work in locations that shouldn’t be known to anyone who isn’t directly involved.
• Some projects contain information about partners or other third parties that shouldn’t be broadly shared.
• Some organizations have such scale or hierarchy that one-size-fits-all permissions leads to an overwhelming amount of stuff.

So how can we offer a solution that’s simple to use but allows you to specify different permissions for different projects for each user? Lucky for us, we already have a system for assigning individual users to projects for which they are responsible — the Staff Roles table!

The same table that already gives partner users access to a project can also be used to identify when people need particular permissions for a project they work on versus one they don’t.

We decided to use this concept of “assigned projects” and “unassigned projects” as a handy way to differentiate between “broad permissions” and “limited permissions” within user groups. Since the concept of a ‘project’ is central to our data model, it’s easy for DevResults to translate project assignments to relevant indicators, results frameworks, and partner organizations without a lot of clicking and decision-making.

And it looks a little something like this:

Note that any users in this example group can do just about everything for projects they’re assigned to, but can still see most information and content on all the other projects. They can still participate fully in project discussions (to ask questions, share insight, etc.) but can’t see documents or financial information, which is restricted to assigned users only.

After much experimentation, this seemed the simplest, most straightforward way to give our customers greater control over what their users can see and do in DevResults. Combined with unlimited, custom user groups and unlimited user accounts, site owners have more flexibility than ever in striking the right balance between complexity and security.

We’ve also included some helpful tools to make implied assignments a little more visible. Site owners will be able to see a new Applied Permissions table on each user’s profile page to show which projects, data tables, and indicators they have access to:

In addition, every user will see a tiny ‘A’ for assigned or ‘U’ for unassigned in the top right next to the bookmark star whenever looking at a project, data table, or indicator so they know unambiguously whether or not they’re assigned to something in particular:

But what if you like your permissions the way they are — one set for each user group, without having to worry about individual project assignments? Well, you can keep doing what you’re doing! There's no need to upgrade if you are happy with how things are configured currently.

You can read more about project-specific permissions in our Knowledge Base. If you want to upgrade to project-specific permissions, get in touch at help@devresults.com!

*You may have noticed we’ve started saying ‘project’ instead of ‘activity’. That’s right, our customers have voted with their pseudonyms — ‘project’ has beaten out ‘activity’ as the preferred term to describe individual units of work. We’re following suit — and we’re currently updating our Knowledge Base with this change.

Today, an organization can go from zero to a fully-configured DevResults site in less than a week. Sometimes organizations need more time to gather program information internally, but once that info is ready, a few simple steps create a DevResults platform that’s ready for folks to report data and analyze results. How did we get here?

The beginning through 2013

DevResults began with a fairly tedious setup process, for both you and us. On our end, we pored over M&E documents and wrote custom code to pull information into the system. This took weeks or even months. On your end, there were no features for doing things in bulk online. For example, assigning 30 indicators to 10 projects took at least 300 clicks. We’ve come a long way since then.

The original DevResults setup template

In late 2013, we introduced a more systematized process that served folks well for a decade; we created a single Excel template that defined and captured all the program information needed to create a DevResults site. It had a dozen tabs that captured projects, organizations, indicators, disaggregations, locations, users, and reporting periods, plus all the ways these things were linked together. We typically collaborated with organizations for weeks to perfect this template, making sure that everything was internally consistent and linked up correctly. Then we used some standardized code on our end to import the template, creating the ready-to-use DevResults site.

This process offered huge benefits over the earlier bespoke process: The template clearly expressed what information was needed and highlighted any decisions that still needed to be made. But it came with drawbacks as well. First, it was a little complicated to represent all the content and relationships of an M&E program in a simple Excel file. Over the course of a decade, precious few ever nailed it on their first try without a fair number of reviews and revisions from our team. Second, the setup template was only used once to create a “final starting point” as we called it. Our counterparts doing the setup first had to learn how to use the template, and then they had to learn how to use DevResults itself.

Introducing our new User-Directed Approach

In the last couple years, we’ve added new setup features based on some simple guiding principles:

1. Users should have fast, friendly tools to define their M&E portfolios and enable reporting.
2. These tools should work the same way for getting started and for ongoing program changes down the road.
3. Users should have the option to work however they choose — online in the app, or offline with Excel templates.

At this point, we’ve covered just about everything, including:

• Projects (aka Activities, Awards, etc.)
• Users
• Indicators
• Locations
• Organizations
• Reporting Periods
• Result Frameworks
• Disaggregations
• User Assignments to Projects
• Indicator Targets
• Mechanisms
• Tags
• Project-Geography Mappings
• Project-Indicator Mappings
• Narrative Questions

Let’s have a look at indicators as an example. Our commitment to fast, friendly tools means that we support:

• Quick imports of new information
• Bulk actions for making the same edit to many things at once
• Easy editing for one-off changes
• Export and import options to get a full set of info, make edits offline, and re-import

Quick imports of new information

Our new bulk imports page collects all these templates in one place. Each key component of DevResults is listed with three buttons that let you visit the index for that component, get a blank template for making additions, and importing that template. A DevResults site can be created from scratch by filling out and importing each template in any order, and then it’s ready to go. Partners can then report data, managers can approve it and generate reports, and executives will have dashboards and visualizations at their fingertips. These tools can be used at any time to add a lot of information at once, like if you’re launching 10 new projects.

Bulk actions for making the same edit to many things at once

Each index page has a suite of relevant actions that you can do to some or all of the items in the index. For example, I’ve selected three indicators here and could bookmark them, assign them all to a selection of projects, assign tags, and so on.

Easy editing for one-off changes

It’s still easy to make individual changes to any part of DevResults by editing a field directly, if you have permission to do so. Changing an indicator’s code just takes two seconds on its definition page.

Export and import options to get a full set of info, make edits offline, and re-import

Sometimes you have a lot of edits to make, or even a systemic change to the whole index. For example, maybe you want to re-do your whole indicator coding scheme. No problem – you can export your whole indicator list (or just the relevant rows/columns), make edits, and re-import it just like attaching a file to an email.

These four tools are enabled for every key component of DevResults, from projects to users. We completed this shift about a year ago and have seen dramatic results:

• Organizations are now better empowered to update program information. We’ve eliminated the tedium of repeated changes and enabled users to make the kinds of bulk updates that were previously only realistic for our team to do on the backend. This reduces the friction between program changes and users having the reporting tools they need.

• We’ve increased the efficiency of the setup process. One of our main goals in making these changes was for 100% of the setup effort to double as useful training in how to administer a DevResults platform going forward. Previously, we’d collaborate in Excel, build the entire site, and then start training on the platform. Now the setup process itself gives our counterparts all the training they’ll need to manage changes in program information, meaning that trainings can focus on other tools and user groups. The result is better, faster transfer of capacity.

With this major project done and tested, we’ve turned our attention to enabling bespoke user permissions and, as always, anticipating organizations’ need for technology integrations across the international development sector. Stay tuned.

As always, we welcome your feedback, questions, and ideas at team@devresults.com.

What has twenty thumbs and received FedRAMP authorization in the last year? That’s right. It’s DevResults.

Our team has worked tirelessly over the last year to undergo rigorous security testing and receive authorization from the U.S. Federal Risk and Authorization Management Program. After many months of hard work — and with the support of our auditors, our agency colleagues, and the FedRAMP team — DevResults is officially FedRAMP authorized.

What did the process look like?

In short: a lot of paperwork.

We invested a huge portion of our time and resources. Our team spent a considerable part of that documenting our internal security processes and protocol, and when that internal process wasn’t robust enough, updating them to make sure they were in line with federal standards. Our engineering team focused on infrastructure updates and formalizing processes, as well as answering any number of questions from the auditors and our sponsoring agency as we moved through the process.

And then we did more paperwork based on those answers and updates.

So you chose to do all that paperwork? Why?

One of our main goals as a company is democratizing international development and humanitarian data. If you’ve worked with DevResults before, you’ve heard us talk about what this means, but you’ve probably heard about it in the context of our data structure. We’ve built our software to make sure that all indicator data is complete and comparable. But having complete and comparable data, especially in the contexts our clients work in, can be actively dangerous if it’s not also secure.

Investing our resources into making sure DevResults is as secure as possible — and then getting multiple external sources to verify that it is! — was a logical next step.

What does this mean for our clients?

For the most part: nothing is really changing for our users. We’ve always been transparent and tried to communicate clearly about everything we do, and we will continue to do so.

So what can you expect communication about?

• Every month, we hold an all-hands Security Day where everyone takes on security-related tasks, including fixing vulnerabilities found by our monthly external security scan, making sure all our software packages are up-to-date, manually spot-checking parts of the app for security issues that the scanner may have missed, and thinking through broader security questions that will help our users. We plan to host a webinar every month for anyone that’s interested in learning more about what we’ve done on that day.
• We’ll continue to update everyone on security fixes and upgrades we make via our release notes and blog. Here’s an example of a recent security-related fix.
• Once a year, we go through a rigorous penetration test (which assesses: whether DevResults can fall victim to known vulnerabilities, whether we have any internal misconfigurations based on common attack vectors, and whether we leak any sensitive information, amongst other things). If your team ever needs the results of this test, please email us at security@devresults.com.
• Once a year, we’ll be conducting a FedRAMP audit to make sure we’re continuously aware of and adhering to the FedRAMP standards.

FedRAMP authorization is just one of the many ways we’re striving to keep your data safe and secure. If you have any questions, or would like to know more about our security processes, please reach out to us at security@devresults.com. Please also keep an eye out for an invite to our webinars!

DevResults is pleased to announce support to USAID’s STOP Spillover to build the program’s digital monitoring system.

In October 2020, USAID funded a five-year program to enhance the capacity of priority countries in Africa and Asia to reduce the risk of viral spillover from animal hosts to humans. Strategies to Prevent Spillover, or STOP Spillover, assists target countries to enhance country capacity to prevent and/or mitigate spillover of zoonotic viruses from animals to humans, and to reduce the amplification and spread of these viruses once they have spilled over to humans. Working with host governments and relevant stakeholders, STOP Spillover will strengthen local capacity to identify, assess, and monitor risk associated with emerging zoonotic viruses, as well as develop and introduce proven and novel risk reduction measures. STOP Spillover focuses on a number of prioritized zoonotic viruses including Ebola, Lassa, Marburg, Nipah, animal-origin coronaviruses (including SARS, MERS-CoV, and SARS-CoV-2), and zoonotic influenza viruses.

STOP Spillover is working in countries that are at high-risk for the emergence and re-emergence of known zoonotic viruses with pandemic potential, starting with five countries in the project’s first year: Bangladesh, Cambodia, Liberia, Uganda, and Viet Nam.

The project includes three integrated and complementary objectives:

• Objective 1: Strengthen country capacity to monitor, analyze and characterize the risk of priority emerging zoonotic viruses
• Objective 2: Strengthen country capacity to develop, validate, and implement interventions to reduce risk of priority emerging zoonotic viruses spilling over from animals to people
• Objective 3: Strengthen country capacity to mitigate amplification and spread of priority zoonotic disease in human populations

STOP Spillover needed a digital platform to collect, store, organize and present the information necessary to manage its operations and performance. The project was looking for a company that would commit to modifying and maintaining the system as additional country programs and core activities are launched. And that's where DevResults came in.

DevResults — a technology company that supports organizations working in the international development and humanitarian assistance fields — was selected through a competitive process.

The STOP Spillover project mission and objectives include a heavy focus on collecting, analyzing, and monitoring the risk associated with zoonotic disease spillover, and DevResults has a proven track record that will help STOP Spillover meet its objectives. The system build is still currently in the setup and configuration phase and we look forward to formally launching the live DevResults STOP Spillover site in the coming months.

A series of blog posts will be periodically posted as important lessons are captured on the building, maintaining, and managing of the system. These blog posts will be published for open access and will detail different perspectives on the technical work, check in on the progress, and describe steps that are upcoming. So stay tuned for more!

Silly name, serious business. Today we’re releasing a new feature...that has existed for 10+ years: the DevResults Index, as you’ve never seen it before.

DevResults users have long used index pages across the app — they provide lots of ways to narrow down your search to exactly what you're looking for. We've now adapted other selection tools in the app to match, expanding functionality while streamlining the interface.

Here's an example...

This update puts one of our most robust and versatile selection tools in many new places, replacing other less-powerful selection tools, including:

• Selecting indicators for a MultiMatrix report
• Pushing or changing ownership of dashboards
• Adding indicators to one or many projects, and vice versa
• Adding users to one or many projects, and vice versa
• Adding reporting periods to a project

More importantly, we’d like to share how we got here.

The Backstory

Building a user interface (UI) requires creativity, invention, experimentation. We spend most of our time thinking up, drawing out, and talking through various ideas, only a fraction of which actually make it to our app. Sometimes we put a few different UI ideas out there to see which work best.

Case in point: how should users choose one thing to be added/linked/assigned/mapped to another thing?

Experimentation

Let’s say you’re adding indicators to a project. What’s the most intuitive way for someone to find a handful of indicators from a giant list? Well, most people tend to think about indicators as they relate to results frameworks (aka logframes). We see this all the time in spreadsheets that clients send us, with results and indicators tied together. Makes sense! Behold, the “indicator chooser” was born:

Elegant, intuitive, straightforward, no? With this pop-up, you could:

• “Select all” by ticking an entire framework, full of results and sub-results and linked indicators
• Pick and choose your results, and in the process, grab all nested indicators
• Expand your entire framework, and select individual indicators one by one.

That made sense for a long time. But over time, we noticed users asking questions that this UI pattern couldn’t answer:

• How do I select all indicators with a particular tag?
• How do I search for an indicator name if I don’t know the result it’s tied to?
• Why can’t I find an indicator that I know exists? (Spoiler alert: because it isn’t linked to a result, and therefore, will not appear in a result-based list like this.)

We had many other patterns for finding and linking other things.

Let's say you wanted to assign a user to a project. How about a dropdown menu? Better yet, a dropdown you can filter by typing a couple characters? That’s great...until you find yourself with 100s of users to assign to dozens of projects, one by one. Or you can't remember someone's first name, but you know which organization they're with.

Sometimes the simplest solutions are not always the best when dealing with complex linkages.

What if you wanted to push some dashboards to several users at once? Once again, we need to select a few key users from a much longer list.

How about a checklist with a search bar and a “select all” option? That's slightly better than just an alphabetized dropdown...except when you want to find just partners, or just users from a single organization, or, or, or,...

Once again, it just isn't possible to give users all the options they need to make complex queries with the standard UI tools that are out there like dropdowns and checklists. Worse, it's slightly frustrating to offer users a very capable query tool in one place, but not everywhere else.

The thing is...these are all solved problems. Despite all of our experimentation with different UI patterns and tools and ‘choosers’, the one that consistently beats them all is the one users are most familiar with: the humble Index.

Indexification

DevResults index pages are simple but powerful tools for filtering, searching, bulk editing, querying, and ultimately choosing things like projects, indicators, reporting periods, users, and more.

Instead of continually reinventing the wheel, we decided to use our most popular ‘chooser’ everywhere it could possibly go. Need to add an indicator to a project? Here’s the indicator index. How about a reporting period? Reporting period index. Users? User index. So on and so forth.

We hope — and our awesome beta testers have so far confirmed — that this will enable users to find things easier and faster. But the benefits don’t end there.

Consolidation

While experimentation can help you learn what is the best way to do something, the ultimate goal is to consolidate everything you can on that one best way.

This benefits users (🖱️) and developers (💻) alike:

🖱️ We’re busy people. We like consistency. When something is unexpected, it’s disruptive.

💻 We’re busy, too. We like to stay D.R.Y. — that means we’d prefer to provide the same value while writing and maintaining less code.

🖱️ Our colleagues are also busy. When there’s an existing template or pattern, it means we have to do less training.

💻 Hard same! When we onboard a new teammate, it’s easier for them if they only have to learn one way of doing something, not fifteen.

🖱️ We don’t like bugs. When something breaks, we can’t move forward.

💻 Ew, bugs are the absolute WORST. The fixes are not always obvious, and they can really derail your day. The less "surface area" (roughly, lines of code) our app has, the easier it is to exterminate those pesky little critters.

We hope you enjoy using indexes on all your choosers. If you have any feedback for us on this or anything else, don’t hesitate to get in touch at help@devresults.com!

Protection of personally identifiable information (PII) is one of the most important responsibilities in M&E and a solemn duty of technology providers.

As the situation deteriorated yesterday in Kabul, our team discussed what we could do to protect the people that our partners and software endeavor to serve. Our team reached out to every organization we work with that reports any data in Afghanistan and has applied the extra security measures deemed appropriate by these teams.

These options are available to any DevResults users (at any time for any reason) to add additional security:

1. Forced logout of all users: Forced logouts address the risk of someone gaining access to an unattended computer where someone is logged in or had logged in recently. A user must log in again with a password if they try to change pages or begin a new session.
2. Erasing passwords for some or all users: Erasing passwords addresses the risk that passwords have been compromised. It does not prevent someone from creating a new password; a user would need access to their work email in order to create a new password.
3. Temporarily removing access for some users: Owners of DevResults sites always have the option of temporarily changing the permission level of some users to “No Access”. This change takes effect when a user tries to change pages or log in.
4. Temporarily shutting down all access to the site: The DevResults team can temporarily deactivate any site.

Today our hearts are with the people of Afghanistan. For any development or humanitarian professionals in Afghanistan, we ask that you take a moment to think of what information you might have online or on a hard drive that could put individuals at risk, and to take steps to protect it in the case of breach of property or access. We recommend reaching out to your technology providers to assess your options. In particular, consider protecting or erasing PII of women and girls receiving training and education, and of anyone working with civil society organizations. Please don’t hesitate to reach out to us at help@devresults.com if we can help with anything.

Over the past few months, we’ve done some serious spring cleaning — by southern hemisphere standards — of our API endpoints and documentation, all in an effort to make our API more user-friendly and accessible to our clients. [I’m sure that most of you reading this are already bored. Don’t stop reading just yet! There’s something here for everyone, whether you know what an API is or not!]

DevResults has always had a public API, and we’ve always encouraged our clients to make use of it — to build integrations, to exchange data across systems, or to power external dashboards and websites. Occasionally we’ve built our own integrations, most recently with Dropbox, Power BI, and SurveyCTO.

To date, we’ve only had a few takers who used our public API to develop their own integration workflows. Even so, we believe in — and our company exists to — reduce data friction and solve the software puzzle for teams. When data can flow easily from user to user and system to system, busy people are more likely to consult that data to make decisions or judge their effectiveness. To that end, we committed to making our API easier to use, writing documentation that’s easier to follow, and providing more support to users who want to build their own integrations.

Throughout our sector, there has been a shift toward adopting technology solutions throughout the project lifecycle. From providing enumerators with data collection tools, to creating a single organizational source of truth, to providing visualizations that tell compelling stories from across the globe, the digital shift requires that technology tools communicate with one another to ensure the flow of data.

What even is an API?

An API is an Application Programming Interface. It’s become trite to say “APIs are how computer programs talk to other computer programs,” and such a simplification obscures both the power and the intricacies of what an API actually does. Unless you live in a science fiction movie, computers are actually pretty bad at talking to each other: they speak mutually incomprehensible dialects, they don’t know what to say unless they’re told, and they don’t know to listen unless they’re asked.

A slightly more useful metaphor to understand APIs might be the dinner table. Each person at your table wants to accomplish something similar (eat food), but each has different needs at different times. One person might be sitting in front of the potatoes, and doesn’t have to ask anyone else to help them get some on their plate. That said, they might offer the potatoes to someone on the other side of the table by saying, “Hey, would you like some potatoes?” while holding the bowl aloft. Someone else might say “I’d like seconds, would you pass them here?” and respond with “Thanks!” when they’ve been successfully handed over. There’s no need for a non-eating third-party server or waiter (i.e. a human user) to pass the bowls around; everyone can serve themselves and pass dishes back and forth as requested. (Can you see where I’m going with this?)

How does an API work?

An API works in the same way. Each platform has a list of requests that it understands — pass the potatoes, here are some potatoes, are there any potatoes left — helpfully written down in its API documentation. When a programmer knows what kinds of requests different platforms understand and what ‘dishes’ they have available, they can script out a conversation that can occur repeatedly (at a certain time each night) or as needed (when a user clicks on something).

In fact, this is exactly how every app that you use on a regular basis works: you push a button, and a flurry of requests are initiated to retrieve or write data in various places. It’s how the front-end user interface that you see interacts with the back-end database that you don’t see. (If you’d like to learn more, there’s a great video by technology communicator and YouTube celebrity Tom Scott that explores the promise and pitfalls of APIs, especially from 1:43 to 3:10.)

Is it really that simple?

Yes and no. It’s not wrong to say “APIs are how programs talk to programs,” it’s just not the full story. In order for one program to talk to another via their APIs, a programmer (read: human) has to write out the dialog. That script turns out to be pretty brittle; if one speaker suddenly starts speaking in a different language or even deviating ever so slightly from the words on paper, the other conversants won’t know how to respond. That’s when your API integration breaks. A programmer (a human, again) has to be alerted that something’s not working the way it should, and rewrite the dialog to accommodate the shift in language. Now the scripted dialogue may proceed.

This is exactly why we at DevResults have neither committed to build nor maintain client-specific integrations — it's a significant investment and risk to rely on a third-party’s API infrastructure that we have little control over. Even when we add “baked-in” integrations to our app, we are selective about who we integrate with — ideally, someone that has a well-documented API, that knows not to change that API unexpectedly, and is friendly and easy to work with if something unexpected happens. In return, we endeavor to also have a well-documented API that changes rarely with much notice, and failing that, to be a good neighbor when something breaks.

So how does an organization build integrations with DevResults?

Most users won’t ever read or even be aware of our API documentation, but we’ve invested time in documenting and maintaining it so that your IT teams, consultants, and programmers (if you’re lucky enough to have any) can tie your internal systems into our platform. However, given the complexities involved — some of which we’ve discussed, others we have not — building an integration with our API is probably best thought of as a last resort.

We recommend starting the integration process by asking a few key questions:

1. What problem needs to be solved? What task needs to be accomplished?
2. How often, and by how many people?
3. How tedious, complicated, or difficult is it to do manually?

Knowing the answer to #1 is important. People ask us all the time if we can integrate with another system. We ask, “what information do you want to move or update, in which system, from where, and based on what criteria?” Most people aren’t sure. If you’ve narrowed it down to some specific needs — for instance, “Program managers need to update their projects with information from the grant management system” — then you’re ready to proceed.

If the answer to #2 is “rarely by several people” or “regularly, but only by one person,” you might want to consider options that are easier and cheaper than building an API integration. Remember: the goal is not necessarily “integrate all the things!” The goal is to be good stewards of scarce resources, chiefly: time, money, skilled personnel, and high-quality data. It wouldn’t make sense to replace a 20-minute-per-quarter task with an automated API integration that could take months to build.

Here’s some alternatives to API integrations to try first, in order of the expense to your organization in terms of your staff’s time and effort ($-$$$):

• Shared key values or unique IDs ($) can join or link two or more records in different systems if and when data need to be combined. The joining/linking process is somewhat manual, but can still be done systematically with just a little bit of expertise (for instance, using the VLOOKUP function in Excel). There’s lots of ways to do this in DevResults: a few purpose-built fields are project short names or reference numbers, indicator codes, and organization short names or organization identifiers.
• Cross-referenced hyperlinks ($$) can ensure that users can jump directly from a record in one system to a related record in another system quickly and easily. It’s only slightly more “expensive” than shared key values because links can and do break from time to time, but luckily they’re easy to fix. Even with a team of developers in our company, we regularly use this integration method internally to link help tickets to development tasks and back again. That way, we never have to hunt for the right email thread to reply back to the user that we’ve finished a bug fix or a feature improvement. You can add links to most text fields in DevResults wherever they may be relevant, such as project or indicator description text fields — use Markdown formatting like this: [link text](www.website.com) — or in data table cells.
• Structured imports and exports ($$$) require a bit more coordination between different systems upfront, but then the export/import process itself is just a few clicks for each user. For instance, let’s say you want to take your grant management system records and use them to populate a batch of new projects in DevResults. If you can get your grant management system to generate a spreadsheet-based export of new grants similar to DevResults’ project upload template, a quarterly, monthly, or even weekly update process will be a cinch — less time than it takes to write an email to tell an intern to do it for you. If you’re going the other way — from DevResults to another system — you can either use one of our many built-in exports, or request a custom query that structures the data in the format that your other system is expecting.

If information needs to be exchanged frequently or by many people, or the process is very tedious, with lots of possibilities for human error, then it might be time to build a true, API-based integration. Actually, building it might be the easiest and cheapest part. The full cost of an API-based integration includes:

• Building it which typically requires hiring a developer or engaging a consultant.
• Testing it to make sure that it holds up under real world conditions with millions of records, not just a handful of test cases.
• Maintaining it over time (again, via an on-staff developer or third-party contractor) as systems change, security patches are required, or user needs evolve.
• Socializing it when unfamiliar users may prefer to manually transfer data themselves, rather than relying on an invisible robot to do it for them. (You’d be surprised how many people still prefer to import data from survey platforms manually, despite the fact that we’ve had multiple automated integrations over the years.)

If you’ve considered these elements and done the cost-benefit analysis, then we’re ready to help you use our API to its fullest extent! Start by familiarizing yourself with our API documentation, either at that link or at [your_site].devresults.com/api-help and get in touch if you have any questions. We’d like to hear how you’re using our API and how helpful you find the documentation so that we can publish new endpoints or add more detail where needed!

Let’s have some fun...

Want to earn some bragging rights? Let’s unlock your API achievement! All you need for these examples is your web browser. No coding required!

First, log into your DevResults site. This gives your browser the necessary “authentication token” to not only use the site, but also to use the API.

Then, enter the following GET request just like you would a website URL, being sure to replace the text in brackets with your site’s subdomain/name:

https://[your_site_name_here].devresults.com/api/awards

Next, hit enter. Congratulations - you’ve just made a GET request (your first?) from our projects endpoint! You told the API you wanted to get a list of projects (née awards in our database and API). Now let’s look at what you got back. It should look something like this:

[{"AwardID":38004,"Title":"Access to School Lunches","ShortTitle":"ASL","Code":"ED-08","ReferenceNumber":"01-519-HBD-00-5414","AwardingOrganizationID":27501,"StatusOptionID":2998,"MechanismID":8730,"PrimaryUrl":"awards/Award/38004","ShortTitleAndTitle":"Access to School Lunches (ASL)","PrimaryOrganization_OrganizationID":27481},{"AwardID":38013,"Title":"Addressing Corruption in Local Government","ShortTitle":"ACLG","Code":"DR-05","ReferenceNumber":"01-172-FEE-00-6674","AwardingOrganizationID":27501,"StatusOptionID":2998,"MechanismID":8727,"PrimaryUrl":"awards/Award/38013","ShortTitleAndTitle":"Addressing Corruption in Local Government (ACLG)","PrimaryOrganization_OrganizationID":27493},{"AwardID":38003,"Title":"AgriMAX","ShortTitle":"AgriMAX","Code":"AG-07","ReferenceNumber":"01-950-GDC-00-4751","AwardingOrganizationID":27501,"StatusOptionID":2998,"MechanismID":8729,"PrimaryUrl":"awards/Award/38003","ShortTitleAndTitle":"AgriMAX (AgriMAX)","PrimaryOrganization_OrganizationID":12345} …]

Note: if you forgot to log in to your site first, or if you don’t have permission to view your project index, you will instead see this:

{"Message":"Authorization has been denied for this request."}

Behold, this is what your project index looks like in JSON (JavaScript Object Notation, a common machine-readable language and data interchange format). You might see some familiar names and numbers, but on the whole this output is illegible and unusable… unless you’re a computer!

If you were a computer, you would be able to recognize, deconstruct, and parse this blob of text into useful things, like project reference numbers or partner organization IDs. Let’s see how! This time we’re going to make a more-specific GET request — not just a data dump — based on the first.

To get started, find your favorite project name in the text blob from earlier, then very carefully start reading from that name until you see "PrimaryOrganization_OrganizationID:" The number that follows is the ID that DevResults uses to uniquely identify which partner organization goes with which project.

Copy or write that number down and stick it in the following request (don’t forget to replace your site subdomain/name again too!):

https://[your_site_name_here].devresults.com/api/organizations/[that_ID_number]

What you should get back is another (much shorter) blob of text, this time with the name and short name of the primary partner organization for your favorite project. It should look something like:

{"OrganizationID":12345,"Title":"Caudill Associates","ShortTitle":"Caudill"}

These two GET requests show you existing information in your DevResults site. To make changes to DevResults via API, like creating a new user, editing a project, or writing an indicator result to the database, you would use a more sophisticated tool like Postman to make POST, PUT, and DELETE requests. A computer could run a complex series of requests on a schedule (or when a human clicked a button in another system) to do things like upload data to DevResults, export data from DevResults and import it somewhere else, update a project's budget, generate a report, add new users and assign them to projects, and pretty much anything else you can do in DevResults.

How can we help?

As a company, we provide an evolving platform that enables our partners to advance as quickly as technology changes. We commit our time and resources to create end-to-end continuity of data utilization. One of our key values as a SaaS company is focusing on “what’s next” in order to ensure that our platform evolves along with the sector. It’s this concept that has made us successful for 10 years running.

If you can think of a way that our API could help your organization connect and integrate your data systems more seamlessly, please reach out to our Helpdesk at help@devresults.com and we’ll guide you through the process!

Last year, DevResults instituted paid sabbaticals for employees with five years of service in order to expand our collective skills and reward long-term commitment. I was the third person in the company to take advantage of this personal and professional development opportunity and I’d like to share a bit of my experience with you.

Motivations

There’s always more than one reason to take a sabbatical. In my case, it was a mix of the following:

• Curiosity about the world of AI, something I had never had the time to pursue
• A need to switch gears and use different tools. When you work in a single product for over five years, the world changes around you. I felt like I was missing out.
• My body telling me it needed me to take better care of it

Timing

I started planning this before Covid-19 hit the world. Initially, I had picked a time of the year that interfered as little as possible with the company’s busiest months, my summer vacations (I’m in the southern hemisphere, so that’s January-February), and other commitments.

The pandemic made me and my colleagues reconsider. Ultimately, we acknowledged that unforeseen circumstances could have happened in any context, and it was still in our best interest to carry out the original plan.

I’m thankful for the company making this decision, as the stress resulting from the long and strict lockdown in my country would have been extremely detrimental to my health. Being able to take care of myself, my house, and my family (besides learning) while knowing my teammates weren’t depending on my input was extremely important.

Preparation

In order to get the best value out of my sabbatical time, I talked to the other people in the company that had had the experience. Besides planning specifics, like which online courses I was going to take, this helped me moderate my expectations. A great piece of advice I got: “Whenever you take on a project with lots of unknowns, you will accomplish a fraction of what you thought you would. Accept that and don’t let it discourage you.”

Initial direction

I started with an online course that covered a pretty good number of ML scenarios, from simple linear regressions to image recognition with deep learning. Around the middle of it, and once I had grasped the fundamentals of classification, I decided what my programming project was going to be: a personal expense tracker that would read transactions from a PDF and automatically classify them. This idea was born out of frustration with existing software, and I still think I can do better than the products I tried.

Issues

As I should have remembered from my experience, writing software usually entails 5% of the fun, shiny ideas and technologies that you want to use, and 95% of boilerplate, infrastructure, and many things you are not really interested in. When you work with a team, it’s easy to split the tasks in a way that everyone gets a reasonable share of cool and boring stuff. When you’re on your own, that’s not possible.

I did not abandon the project, but ultimately decided to do it as a longer-term, on-the-side thing.

New paths

At the beginning of August, a local network of schools which included my kids’ started a fundraiser for emergency scholarships, as many families were being hit very hard by the economic crisis due to covid-related restrictions.

When I contributed to this campaign, I found out a good amount of information about the donations was made public, and it was not hard to parse it from the website. I thought it would be fun and maybe useful to apply some data science to it.

A new interest

The reaction to the analysis results I shared was very positive, and encouraged me to pursue data science for fundraising as a central topic of my sabbatical.

For this next part of my journey, I mostly used a book. I enjoyed it very much. It didn’t only teach me how data science and machine learning can help for fundraising - it gave me a better understanding of how visualizations, particularly, can help tell a story.

Circling back to my job

Visualizations happen to be one of the things I was involved in the most at DevResults. I was a part of the teams that built our dashboard system, the indicator reports (graphs, maps, pie charts, matrix) and more.

I always liked creating tools that helped our users understand, analyze and report on the data in the easiest, fastest, most accurate way possible. However, as a more technically-focused person, I had not stopped to think about the stories our users are telling with these visualizations. A compelling story, which the appropriate images can tell better than thousands of words, can help get programs funded, priorities changed, and actions executed.

The future

As I write this in the last week of my sabbatical, I’m looking forward to working again in our product, and having a chance to engage with users about the stories they’re telling with data and to apply the things I learned to bring you better tools to tell your story and help your team improve the world.

The Pito tool has been serving DevResults users’ data-pivoting needs for nearly seven years. With the release of Matrix and MultiMatrix earlier this year, we migrated all existing Pito reports to the Matrix format and redirected the links in our menus and Knowledge Base. Now it is time to bid our old friend Pito adieu.

But before we say goodbye, we thought it a fitting tribute to reveal the origin story of the Pito tool, and the very real person who unwittingly lent the Pito tool its name.

Where did the name ‘Pito’ come from?

As with most of our best ideas, it started out partly in jest...

Back in 2013, mere days after joining DevResults, I flew to Zambia with Herb. We spent the first weekend at Victoria Falls, much of it working on laptops in an open-air hotel courtyard.

I was of little use at that time, being brand new to both international development and the software industry, so to give me something to do, Herb had me come up with a list of potential names for features. I thought naming features after pivotal figures in computer science would be clever.

Given that the report builder was essentially a pivot table, I thought that the name of the inventor (if there even was a single inventor) might work. A quick Google search confirmed that Pito Salas was indeed widely regarded as the “father of pivot tables.”

I'd recently learned about "managing up" so I amused myself by trying it on Herb: I presented the outcome I wanted alongside a few other truly terrible ideas. The result was, of course, the name 'Pito'.

How did clients respond to the name ‘Pito’?

I think the name ‘Pito’ has always said more about the people of DevResults than the tool itself...

Long before I joined the DevResults team, I became a client and user of DevResults in late 2013. My organization selected the platform because we were impressed with the demos and pilots we did at HQ. While I was fairly new to the M&E and international development world, I had to agree that it was a robust piece of software...with one exception: the query tool that allowed you to build reports and pivot data from across the platform left much to be desired, to put it mildly. "Don't worry," they said, "we're about to release a new upgraded report builder."

A few weeks later, I was traveling to Kyiv, Ukraine with a colleague (in January 2014, no less) to role out our new DevResults M&E platform with two of our program teams. We met Leslie for the first time during our layover in Frankfurt as we boarded our flight to Kyiv.

As the situation on the Maidan began to escalate, we sat just blocks away watching the snow fall gently outside as we learned the ins and outs of DevResults. At some point, Leslie unveiled the promised report builder upgrade. It was fantastic! It let us build exactly the tables and reports we were hoping to build.

"We call it 'Pito',” she said. I puzzled for a minute and responded, "Oh right, Pi-To, short for Pivot-Tool, right?" She responded, "No, it's named 'Pito' after Pito Salas, the 'father of pivot tables.'" I had to quickly arrest a dramatic eye roll — I knew the DevResults team was a bunch of nerds, but that was some deep trivia!

Why did MultiMatrix replace Pito?

While the Pito tool did many wonderful things, it had several shortcomings...

It didn’t let you turn off subtotals or totals columns, it didn’t let you display targets or disaggregations elegantly, it had somewhat limited configuration options, and above all it was computationally intensive, meaning that it took a long time to generate data for even simple queries.

For these reasons and more, we had been keeping a list of things we wanted a New Pito tool to do. Finally, with other priorities completed, we had the capacity on the engineering team and room on the calendar to begin an overhaul of Pito — and so began MultiMatrix.

Did anyone ever cross paths with the real Pito Salas?

Funnily enough, we did only just recently, and quite unintentionally...

About the time we finally released MultiMatrix, a few of us found ourselves on the island of Curaçao. (This job often takes you to amazing places!) While we were there, we decided to tour the historic Mikvé Israel-Emanuel Synagogue, the oldest surviving synagogue in the Americas. The complex includes a small museum dedicated to the history of the Jewish community in Curaçao, as well as a large sanctuary with sandy floors, intended to muffle the footsteps of worshippers and hide their numbers during the Inquisition.

The tiny museum was surprisingly well-curated. There were old Torahs, priceless antiques, photos of prominent families from the Jewish community, and decorative family trees. Of course we barely glanced at the names or the captions — it’s not like we would have known anyone from the Curaçaoan Jewish community...

...if only we had known then how close our paths were!

A few weeks later, we were discussing our plans to sunset the Pito tool and finalize the shift to MultiMatrix. It occurred to me that I had never fact-checked the validity of the whole “Pito Salas is the father of pivot tables” story.

So I googled it and discovered:

1. Pito Salas really did invent the pivot table, way back in 1991!
2. Pito Salas was from Curaçao!
3. Pito Salas and his family worshipped at that very synagogue!

I shared these revelations with the team in a couple different Slack threads:

We realized that none of us had ever spoken to or even tried to communicate with Pito Salas. We googled some more and found that he was a professor at Brandeis University in Massachusetts. Obviously we emailed him the entire story: how we named one of our core tools after him, how we frequently shared the story of his invention with our clients, and how we unknowingly stood in the very synagogue where (we later learned) he once celebrated his Bar Mitzvah.

I was hoping against hope that the real Pito Salas would read the email, laugh uproariously, share it with his friends and family, and join our weekly team meeting on Zoom. But I was secretly afraid that even if the email got past his spam filters, it would reach a stodgy old grump, too busy with his own research to trifle with long emails from adoring fans. “Never meet your heroes,” right?

How did Pito respond?

To our immense delight, Pito responded graciously a few days later:

Hi Reid and the rest,

What a great email! I really enjoyed reading it...I shared this story with all my family and they all found it really cool...I’d be glad to zoom up. Let me know what you think!

-Pito

A few weeks later, he did in fact join our weekly Zoom meeting. After we introduced him to DevResults and his namesake report builder, Pito regaled us with the story of the rise — and fall — of Lotus Improv, the forerunner of the pivot table that was hand-picked by Steve Jobs himself for his NeXT computer system.

So long, farewell, auf Wiedersehen, good night

There you have it. Having come full circle, we have now officially retired the Pito tool. Many thanks to our users for all the feedback that led to the creation of the MultiMatrix tool, and special thanks to Pito Salas for not serving us any cease-and-desist orders!

If there’s anything to take away from this meandering tale, perhaps it’s this: much like art, software can reveal the craft and often the personality of its creators. We take great pride in our craft here at DevResults...but we like to have a little fun too!

We hope that you’ve enjoyed learning about the history of the Pito tool and its eponymous originator. More than that, we hope that you've enjoyed using the Pito tool over the past seven years.

Pito (the tool) is dead...long live MultiMatrix!

DevResults has been providing engaging, hands-on trainings for seven years. With thoughtful adaptation and continuous innovation, we’ve crafted effective and fun trainings, whether in-person or virtual. Below are our four core pillars of a successful training and how we translate them to an online setting.

What are the hallmarks of an excellent training?

• Trainees learn by doing: Trainees should perform 100% of the skills they’ll need to acquire, first to learn a new process and later to reinforce that learning. Hands-on exercises not only help with internalizing key concepts, they also bolster internal capacity development. For example, we ask our trainees to demonstrate new steps to colleagues in preparation for them training others in their organization.
• The training provides opportunities to have necessary conversations: A training must provide dedicated time to discuss issues and resolve questions while everyone is gathered together. We’ve found that during the course of a training, stakeholders across M&E, IT, and program teams bring up issues that others might not have considered. The conversations that come out of our training sessions focus on everything from complicated indicators, to approval workflows, to naming and usage conventions, to how to train and build capacity throughout the organization.
• Attendees are fully engaged: Whether in-person or remote, the pull of distractions is strong. By engaging with trainees on every topic with exercises, mini quizzes, and colleague-to-colleague conversations, we help keep the focus on processing new skills and ideas.
• The training emphasizes problem solving: No one is expected to remember everything from a multi-day training. Effective trainings reinforce how to find information and solve problems weeks and months later. Throughout a training, we provide opportunities to use our knowledge base and help desk to answer questions as well as to familiarize users with all available resources.

What challenges do virtual trainings pose, and how does DevResults address them?

• While it’s reasonable to host an all-day training in person, virtual sessions must be much shorter. There’s plenty of research on how conference calls drain your energy. We’ve been a fully remote company for a decade, and even we get tired of Zoom.
  • Instead of a few days dedicated to training, we’re running 2-hour sessions for 5 days, spread out over two weeks. This way, the amount of time and energy dedicated to training sessions is more manageable for everyone.
  • We’re setting all trainees pre-work to get them ready and properly set up before a training begins. This way, trainees get familiar with the sites they’ll be working in, properly configure their hardware and software, and know how to reach out to trainers for assistance even before we start.
  • We assign asynchronous activities between and after sessions that encourage trainees to practice skills, discuss what they’ve learned with one another, and consolidate new processes with existing organizational workflows.
• Interacting with many people is more difficult remotely. It’s harder to gauge facial expressions, engagement, and attentiveness through a webcam. There are also technical hurdles in facilitating side conversations.
  • We ask our trainees to use temporary Slack groups for co-working and collaboration. This helps foster an atmosphere of ongoing discussion for both internal and app-related issues as they come up. It also means that trainees have a record of all conversation topics and can rely on them in the future.
  • We’re also using Slack to run several interactive polls and conversation threads to increase engagement throughout the training. The questions we ask are great for reinforcing key concepts and solidifying new skills.
  • We’ve expanded the number of interactive exercises we do, including our group scavenger hunt and troubleshooting exercises. We’re also making sure that every participant shares their screen and walks everyone through an exercise at least once every 2-hour session. If this isn’t practical, we can rely on polls, breakout rooms, and conversation threads to engage our attendees.

Virtues of Virtual

• Better for retention: With short online trainings spread out over a week or two, trainees can internalize more content than if they receive it all at once. Trainees also have more opportunities to leverage helpful resources (both ours and the input of their colleagues).
• Better for practice: Practice makes perfect and spacing out virtual trainings offers opportunity for more asynchronous exercises.
• More efficient: Virtual trainings take less time overall, ensuring minimal disruption to ongoing work.
• Cheaper: DevResults is able to provide virtual trainings at a fraction of the cost of in-person trainings. Many organizations will save more without the need to gather folks from the around the world in a hotel or conference center.
• Environmentally friendly: Reduce your carbon footprint by reducing air travel.
• Develop related technology skills: DevResults has worked remotely for a decade; we’ve tested countless tools and practices for online collaboration. We take a holistic approach to trainings and include guidance on making the most of distance learning.

Have any questions for us, or just want to know more about what we’re doing? Reach out to us any time at help@devresults.com

A blog post for anyone who writes indicators

Contents:

1. Percentages
2. Percent Change
3. Cumulative vs. Incremental Targets
4. Cumulative vs. Incremental Actuals
5. Disaggregation vs. Free Response
6. Cross-Disaggregation vs. Parallel Disaggregation
7. Deduplication

Background

In seven years as a data scientist at DevResults, I’ve worked with donor agencies, implementing partners, nonprofits, and private companies. I’ve had a front-row seat to M&E planning in the sectors of health and sanitation, education, democracy and government, humanitarian response, justice and law enforcement, agriculture, and economic growth. In that time, I’ve seen many indicators that were mathematically incorrect or fatally vague. Indicators get mangled in bureaucratic processes and often poorly reflect the original intent. This post will cover the most common problems I’ve encountered — problems that originate with the way indicators are defined and the way tools are set up to track results.

1. Percentages

Take a humble percentage indicator: Percent of people who passed a test. This is calculated as:

People who passed the test ➗ People who took the test

If 1 person passed out of 2 who took it, 50% passed. Simple enough. The problem arises when comparing results between different disaggregations, places, projects, or time periods.

Consider the example below, where many more people took the test in location B than A. If implementers report 50% and 99% for these locations, you might average them to get 74.5%. But this is clearly false. In total, 100 passed out of 102 who took it. This is a 98% pass rate. The only way to get to the correct result is to collect the numerator and denominator separately, NOT the percentage. You can always calculate the percentage yourself across disaggregations, places, projects, or time periods that you’d like to analyze.

Put another way, taking an average of percentages is always incorrect where denominators vary.

If you write indicators, you should require that the numerator and denominator are reported separately. If you're not in the position of writing indicators, it's always good practice to keep track of your numerator and denominator separately, even if you’re not required to.

Percentages are analysis done on data, not the data itself.

2. Percent Change

Calculating percent change presents the same challenges as described above: any aggregation of percent change would be incorrect if collected on its own. Instead, percent change should be calculated using underlying data for the two time periods to be compared. This is the same math as above.

But percent change offers additional pitfalls. Percent change from what? Here are a few options for change analysis with dramatically different results:

Each of these percentages is valid, but each means something completely different. Our advice is to collect the underlying data, enabling any of these kinds of analysis.

Good indicator: # fuel-efficient cookstoves in use
Resulting analysis: % increase in fuel-efficient cookstoves

If you use the first indicator:

• You can aggregate current and former data before calculating percent change, ensuring accuracy.
• You can always compare current data to any previous data.

If you don’t use the first indicator and only have people report their analysis:

• Comparisons across projects and places will be necessarily incorrect. They won’t be properly weighted by the number of cookstoves previously in use in different places or different projects.
• A location going from 1 to 2 cookstoves would have a 100% improvement, while a location going from 2 to 3 would have a 50% improvement. If the only data you have are the percentages, they seem very different, even though these results are nearly the same.

In short, percent change is analysis done on data, not the data itself.

3. Cumulative vs. Incremental Targets

Some organizations and projects think of targets as “This is what should be accomplished from the start of the project up to this date.” We call those cumulative. Some organizations and projects think of targets as “This is what should be accomplished only during this reporting period”. We call those incremental. Each is equally valid, but to avoid major problems in interpreting progress to target, the choice must be clear.

In the example below, the cumulative and incremental options are mathematically equivalent.

• For the cumulative row, there should be 50 things done by the end of Q1, then another 50 by the end of Q2 for a total of 100, and so on.
• In the incremental row, it’s simply stating that 50 things should be done each quarter. Both mean that 200 things should be done during the year.

Without the labels, you’d probably intuitively understand what each row meant.

But take the example here. Are these cumulative or incremental?

They could be either. If you interpreted them as cumulative, the goal would be to do 180 things by the end of the year:

But those targets could also be interpreted as incremental. In this case, the goal would be to do 180 things in the last quarter alone. The goal for the whole year would be 430:

If you assign targets, make sure you identify whether each indicator’s targets are cumulative or incremental. Each is equally valid, but if not clearly defined, analysis of progress to target could be dramatically wrong. If you're not assigning targets, it's always worth clarifying whether the targets you're working toward are cumulative or incremental.

4. Cumulative vs. Incremental Actuals

The meaning of cumulative vs. incremental for actuals is the same as for targets, but the considerations for handling actuals is different. In this case, the question is not how to interpret written targets, but how to enter the data requested. Whether or not indicator results should be recorded incrementally or cumulatively is dictated by the indicator definition.

In this respect, there are two kinds of indicators — those that measure work done over a period of time, and those that observe a state at a point in time.

Measurements of work done:

# schools built
# kilometers of roads cleared
# vaccinations administered

Observations:

# schools in community
# kilometers of clear roads
# community members who are vaccinated

Measurements of work done should be reported incrementally. Each reporting period, implementers should report just what work was done since the end of the last reporting period.

In contrast, observations are necessarily cumulative. At the end of a reporting period, you report the state of affairs at that moment.

Consider the indicator # people trained. Training someone is “work done” during a certain time period, so it should be reported incrementally. Features of “work done”, or incremental indicators:

• The baseline is zero. Before you train anyone, you’ve trained 0 people.
• A sum of all the reported data gives you the total (see below in yellow).

If you really want to report this data cumulatively, then the indicator should be named clearly (see in red), but it’s inadvisable. It just makes it harder to analyze things like change over time and the total is meaningless. How many people were trained in the second half of the year? That’s easy if you look at the incremental data, but less intuitive looking at the cumulative data. (96).

Now consider the indicator # people with access to clean water. Handling this data cumulatively is straightforward -- you just report who has clean water at the end of the quarter. Features of “observations”, or cumulative indicators:

• The baseline is most often not zero. There will likely be some people with access to clean water prior to a program’s efforts.
• Cumulative data should never be added together. Any sum or total is meaningless. The end result is simply the last datapoint entered (yellow).

If you really want to report this data incrementally, then the indicator should be named clearly (see in red), but it’s inadvisable. Here, Q2 ended with fewer folks with access to clean water than in Q1. This is easy to read in the cumulative row. But a “-32” as an incremental change makes less intuitive sense.

If you write indicators, make sure the indicator name explains whether the data is incremental or cumulative. It should be clear whether someone is measuring work done or making a point-in-time observation.

5. Disaggregation vs. Free Response

Two pieces of information are required for defining a disaggregation.

1. The disaggregation, such as sex.
2. The disaggregation categories, such as male, female.

No disaggregation is complete without its set of disaggregation categories. If you want to disaggregate by age groups, what are the age groups? If you want to disaggregate by training type, what are the training types? Defining this information is required for any indicator definition.

Occasionally I talk to a project that plans to offer a free response option in a survey. The project wants to learn from beneficiaries what the categories should be (what type of fertilizer do you use?), or to hear what beneficiaries have to say in their own words (how do you feel about your local police?). The project doesn’t want to pre-suppose a narrow set of potential options to choose from. This is a perfectly valid survey method, but writing surveys is not the same thing as writing indicators. Disaggregating indicator results means taking each response/event/observation and putting it into a bucket; the bucket is the disaggregation category. You simply can’t put something in a bucket if the bucket doesn’t exist.

Take this set of survey responses. To a computer, all 18 responses are different. If you made a report disaggregated by fertilizer, you’d just have a list of every response with a count of “1”. This isn’t very useful for summarizing large datasets. If you consider all the differences in terminology (and spelling!) then 1000 people could give 1000 different responses. Listing them doesn’t equal disaggregation.

If you start with a list of free responses, you have to decide how to group them -- that is, what the disaggregation categories should be. There’s rarely one right answer for how to do this. In our example, you could simply remove the duplicates/synonyms, or group them with various levels of specificity.

The choice in how the responses are categorized can produce a range of results. Regardless, no result is possible without first settling on a finite set of disaggregation categories, thus completing the indicator definition.

Categorizing free responses can be laborious and can introduce the biases of the person doing the sorting, but it can also provide unexpected insights and avoid presumptions of an indicator-writer. It really depends on what you need the data for. If the goal is simply to understand the prevalence of inorganic vs. organic fertilization, you could skip the free response and go with multiple choice.

All the same warnings apply to offering a category like “Other (please specify)”. That’s a free-response list within a multiple-choice answer. It would be subject to the same requirements that the data be cleaned and grouped by meaningful categories. The same would even be true of a survey that invites paragraph-length narrative responses.

To indicator writers: indicator definitions are not complete until disaggregation categories are finalized. If disaggregation categories won’t be determined until after a free-response survey is given, make sure there is a process in place for defining them then.

6. Cross-Disaggregation vs. Parallel Disaggregation

Frequently, indicators have multiple disaggregations, each with their own set of disaggregation categories. In these scenarios, data can either be reported for category combinations (cross-disaggregated) or for each category in isolation (parallel-disaggregated).

Consider disaggregation by sex and age. With cross-disaggregation, each sex category is subdivided by all of the age categories. In this scenario, adding up all the results gives us the total number of teachers:

Since we have one disaggregation with two categories and one with three, the total possible combinations is 2 x 3 = 6. Imagine if you wanted to disaggregate by sex (2 categories), age (3 categories), training type (5 types), course level (4 levels), and location (10 places). That would be 2 x 3 x 5 x 4 x 10 = 1,200 combined categories. If you write indicators, be aware of the multiplication effect with heavily cross-disaggregated indicators. If such specificity is even warranted, make sure implementers have the resources they need for handling this quantity of data.

In contrast, with parallel disaggregation, the sex and age categories are handled separately. In this version, we don’t know how many females are under 18, or how many 30+ folks are male. The categories are not subdivided. With parallel disaggregation, our total number of teachers comes from just adding the females and males together. This has the benefit of reducing the reporting requirements but there’s a loss of specificity in the information.

An alternative use of parallel disaggregation is the case where the second disaggregation is not mutually exclusive. For example, you could vaccinate 100 people and give all 100 each vaccination:

This gives us information about how many people were vaccinated and the vaccinations received. Since we’re counting people, it would be meaningless to add up the vaccination counts to get 300. (That would be a different indicator, # vaccinations administered). Parallel disaggregation gives us the option to mix mutually exclusive and non-mutually exclusive categories, which is impossible with cross-disaggregation.

If you write indicators, make sure that indicators with multiple disaggregations are clearly defined as cross-disaggregated vs. parallel disaggregated. For cross-disaggregation, be especially aware of the number of potential category combinations.

7. Deduplication

Now the fun starts. Instead of numerical data, consider the raw records that they come from. Imagine keeping this log of trainees:

It’s very common for programs to be interested in deduplicated — or “unique” — counts of trainees or other beneficiaries. But this can be much more complicated than it sounds. First, let’s look at three ways of calculating a deduplicated count of trainees.

In the most basic sense, 4 people were trained: Anita, Bob, Crispin, and Delvan.

However, what happens when we report a unique count each quarter? Anita, Bob, and Cripin were trained in Q1, while Anita, Bob, and Delvan were trained in Q2. It would make perfect sense for an implementer to report Q1: 3 and Q2: 3. The problem is that the total would be 6, even though we know that only four people are involved.

Are you expecting to receive a report that says Q1: 3, Q2: 3, Total: 4? If so, you’ll need a tool that can handle such complexity. If you don’t have a tool that does that, then it’s better to handle quarterly unique counts as separate indicators from all-time unique counts.

But there’s another way to think of a unique/deduplicated count that leaves a lot less room for confusion. You could define the indicator as a count of the first time someone was trained. This necessarily removes duplicates and simplifies the math for totals.

Both “count unique per quarter” and “count first appearance” are valid strategies, but they produce different results. If you write indicators, you should make it impossible to misunderstand which type of count you’re looking for.

The same complexity arises with disaggregations. Let’s look at the table again:

How many people were trained in irrigation? Anita, Bob, and Crispin: 3. And Crop rotation? Anita and Delvan: 2.

So what’s the sum? Is it 5? Or is it the 4 actual people? Put another way, how do you feel about counting Anita twice since she was trained on two different topics? There’s no right answer, but implementers require clarity, tools, and guidance to arrive at the result you want.

Things only get more complicated when you combine the disaggregations you’re investigating, such as “per topic” and “per quarter”. In this example, the crop rotation total comes out matching the unique count above (2 people), but the irrigation total is different because we’re counting Bob twice, once in each quarter.

Since only three separate people were actually trained on irrigation, should the total be 3 or 4? Again, are you OK with counting Bob twice since he was trained on irrigation during two different quarters? If not, then the quarterly unique counts should probably be a separate indicator from the overall unique counts.

Compound units offer an opportunity to clarify unique counts. The most common compound unit is the “person-hour”. If 2 people work 5 hours, that’s 10 person-hours. If 1 person works 10 hours, that’s also 10 person-hours. The same logic applies to a deduplicated count of “person-topics”. Applied to our sample dataset, we get 5 unique person-topics. This takes the 7 rows of the table, disregards the second and third time that Bob was trained in irrigation, and gives us a result of 5. This value doesn’t tell us exactly how many separate individuals were involved (since Anita is counted for each topic), but it “gives credit” for every time a person was exposed to a new topic.

Another way to “give credit” would be to count person-trainings. In this case, the project would get credit for training Bob 3 times on the same topic.

With almost every organization I’ve ever worked with, there’s been a conversation along the lines of, “if you train Bob twice, do you count him twice? What if you train him in separate reporting periods? What if you train him on different topics?” It’s rare that people are sure of the answers or that everyone agrees. If you write indicators, the answers to these questions should be explicit. Here, we’re counting a tiny dataset by eye. It’s extremely complicated to use Excel for something like “count first”, or to “count unique per quarter but with an all-time unique total”. If you have a voice in choosing indicators for projects, ensure that they have the tools and know-how to produce accurate results.

Contact Us

If you’d like to learn about how our tools and expertise could help your organization, or if you’d just like to talk more about indicators, please don’t hesitate to reach out to us at info@devresults.com.

Helpdesks are fairly ubiquitous nowadays, but you may still wonder who is waiting on the other end of help@devresults.com to answer your questions. The current DevResults Helpdesk is the result of dozens of iterations, and it is still very much a constantly evolving work-in-progress. So what can it do for you?

Who staffs the support desk?

DevResults maintains an on-call team comprising one software developer and one data scientist at all times. It’s their job to field and resolve Helpdesk tickets, allowing the rest of the DevResults team to focus on new features and other projects. Engineers typically rotate every three weeks (the length of our development sprints) whereas the Data Team cycles out weekly, with liberal use of “tag-outs” as life or illness intervene.

In the early days, our Helpdesk was staffed by a single data scientist (sorry Leslie!) but that approach didn’t scale well. We added an engineer to the rotation to ensure that bugs and other technical issues got immediate attention without disrupting on-going feature development work. If there are no bugs to squash, that engineer has carte blanche to do so-called “grease work” — the odds-and-ends tasks that result in small quality-of-life changes for users but never rise to the level of a top priority.

DevResults has never hired specifically for a support or on-call role – it’s always been staffed by experienced, senior team members. There may be cheaper ways to respond to help tickets, but we find the regular cycle of user interaction to be beneficial and grounding for everyone; it forces us to get familiar with every corner of our platform and take note of how people are using it. If we get the same question two or three times, maybe we need to write a new article in our Knowledge Base. If we get the same question weekly, maybe we need to rethink how a page in the app is designed.

Rotating on-call duty is also an opportunity for us to go beyond the mechanics of the site and offer our clients advice on evaluation methodology, M&E strategy, and change management as our prior experience and expertise allow. We don’t offer any formal consulting, but we do what we can to make people’s lives easier when they ask for our opinion.

How to find us

Both the Helpdesk email — help@devresults.com — and the “Help” menu inside of DevResults go to our ticketing system. We encourage users to take advantage of the in-app “Contact help desk” wizard because it quietly reports a bunch of helpful information to us, like which browser you’re using or what page you were looking at when you contacted us.

There are lots of ticketing systems out there, but we like Freshdesk because it's easy to use and integrates with our company Slack #help channel. Most of our users prefer to use email for communicating with our Helpdesk, but Freshdesk does give users the option of tracking and responding to the ticket in Freshdesk itself if they prefer.

How to write better help tickets

Few topics arouse more passion from software professionals than the dos and don’ts of writing help tickets. We don’t expect anyone to follow our own internal rules for writing tickets, but it doesn’t hurt to know what information we require to track down a bug or register a feature suggestion. If you don’t tell us, we’ll just ask!

We have a simple template for (internally reported) bug reports:

• Steps to reproduce: how did you get to a bad place?
• What I expected to see: what should it have done?
• What I saw instead: what did it do?

Likewise, we have a slightly longer template for logging feature suggestions:

• Current functionality: what does it do or not do now?
• Use case, user story, or problem to solve: what do you need it to do instead?
• Proposed workflow or functionality: how do you suggest we solve the problem?
• Mockup: a simple sketch (Microsoft Paint works fine) of how and where?

The key with both bug reports and feature suggestions is to clearly define the problem to be solved, not necessarily the solution itself. The problem may seem obvious to you, but there are lots of reasons we need to know that context. Sometimes, there’s an alternative solution to the problem that may be non-intuitive but ultimately more elegant and robust for other users, including the ticket-writer. In other cases, there’s actually already a tool that addresses the issue, you may just not know about it yet.

Again, consider this as advice for getting the most out of your software and your vendors, not a requirement! It takes a bit of time to write a good help ticket, but in the end, your problem gets resolved faster with less back-and-forth if you invest the time upfront.

Setting up your own Helpdesk

The DevResults Helpdesk will always be a part of our service to clients, but many organizations — especially large ones — set up or already have in place their own ‘frontline’ internal help desk. We strongly encourage this practice.

We get a lot of help tickets that we can’t answer because they have to do with organizational policy or clients’ internal systems. Even when the question is specific to DevResults, we’re often unaware of the guidance or norms that have been established by site owners. And last but not least, there’s lots of stuff that even brand new site owners can answer, like how to reset a password.

For all these reasons and more, it’s in everyone’s interest to have an M&E or program help desk — nothing more than an email account shared by site owners — which can always elevate incoming, DevResults-specific tickets to us. Many of our clients who use these systems report that they are indispensable tools for program quality assurance!

Got a question about the app? Go ahead, get in touch! help@devresults.com

I just returned from parental leave following the birth of my second daughter. I was able to do so thanks to the generous support of my company, which offers (as a norm, not a limit) 12 weeks paid leave for employees that have a new baby at home or are experiencing some other life change that requires full-time attention. This is in addition to unlimited sick and vacation time, and as a remote-first company, we all enjoy unlimited work-from-home privileges.

The case for paid family leave

It has been well-documented that the US is the only country among its peers that has no nationally guaranteed paid family leave. Of course some family leave policies in the US are extremely generous, though typically only for large companies or already well-compensated employees. Even when family leave (paid or unpaid) is available, it’s not always used.

The benefits of paid leave to families are fairly obvious (a short list includes: more financial security, improved health and development outcomes, reduced gender pay gap), and made more obvious by the ongoing coronavirus pandemic. Equally obvious are the costs to employers and the potential for disruption when a key team member suddenly disappears, especially in a small organization. Morally, we believe that the intangible social benefits outweigh the costs at any scale — whether within an organization, at the state-level, nationally, or globally. Even so, employers that offer paid family leave realize tremendous benefits in the form of better employee morale and staff retention.

It might be tempting to consider high employee morale a "nice to have," but it permeates everything that an organization does. As a personal anecdote, I elected to take my leave in two chunks — the first immediately following my daughter's birth, the second a few months later. After the first few weeks off, I returned to work as soon as I was achieving functional levels of sleep each night. I quickly realized that I wasn’t giving the job or my colleagues my best. I may have been able to work, but pulling myself away from my child each morning was taxing. After the second break, I could tell immediately that my attitude and contribution to the team was much improved. I don’t regret the decision to take my leave in two parts, but for the sake of my company’s morale and effectiveness, I deeply appreciated (and needed) the extra time.

Paid family leave at DevResults

When my colleagues and I sat down to write our family leave policy, we weren’t sure where to start. We recruit about half of our employees from public- and nonprofit-sector backgrounds where little to no paid family leave is offered, but the other half (our engineers) come largely from the private sector where generous family leave is the norm. In the end, we chose a middle path, one that balanced our commitment to our clients, to our teammates, and to our families.

We crafted our policy with an eye toward three forthcoming paid leave benefits:

• The DC Universal Paid Leave Amendment Act of 2016 (UPLA) offers DC-based employees 8 weeks paid leave for the birth of a child, 6 weeks paid leave to care for an ailing loved one, and 2 weeks paid leave to care for one’s own serious health condition.
• The forthcoming Federal Employee Paid Leave Act (FEPLA) will offer (most) federal employees 12 weeks paid leave for parents that give birth, adopt, or foster a child, though it has not taken effect yet, nor will the details of the policy be clear until it goes through the regulatory process.
• The Families First Coronavirus Response Act (FFCRA) which guarantees employees (and incentivizes employers to provide) paid leave for Covid-19-related quarantine, illness, care taking, and childcare.

As a company striving to be more family friendly, we cheer these advancements, and hope that other organizations in our sector will join us and others in offering generous family leave.

Putting policy into practice

But having a generous leave policy isn’t enough. We have worked hard over the years to go further in supporting employees. We do everything we can to make sure no important knowledge or expertise resides solely in one person. We trade responsibilities, document workflows, and share task lists. Compared to our early days, this seems like a luxury, but it’s also vitally necessary to give people the freedom to disappear at a moment’s notice to care for a family member, spend time with a new baby, or just take a nap when they don’t feel well. It isn’t always easy — especially when you’re a team of 9 — but we make it work through a combination of volunteering, turn-taking, and looking out for one another.

I’m proud and grateful to work for a team that not only gives me time off, but also has my back when I take time off, so that I don’t have to look over my shoulder or down at my phone when I’m away. My little one doesn’t realize it yet, but she’s grateful too!

Over the last decade, DevResults has invested time and resources into learning what best practices enable us to be a lean company while still supporting some of the biggest government agencies, implementing partners, and non-profits in the international development and humanitarian sector. Recently, my colleague released a blog post on how we have as a company have created a remote work culture that has been going strong for ten years. While we have specific tools that help us stay connected in a digital work environment, we also believe in implementing principles that encourage efficiency and ensure productivity.

In programming it is common to hear people mention the DRY principle, “Don't Repeat Yourself”. This concept is a reminder that if you need to use some piece of code in more than one place that you shouldn't just copy and paste, but instead extract it out into a reusable module, which can be referenced where necessary. At DevResults, we strongly believe in the DRY principle because it greatly aligns with our desire to operate efficiently. By not repeating ourselves in our code we can minimize the amount of wasted time and effort required to maintain duplicate code. It also provides consistency in the way we perform particular actions, producing a predictable result every time. As a team, we’ve collectively taken the DRY principle beyond our code and we strive for efficient and scalable communications internally and externally with our partners and customers. If you have ever received a help link to one of our knowledge base articles, you've seen DRY at work!

We've made it a habit to try to leave tangible artifacts of our solutions and discoveries with the hope of preventing duplicate effort and also to share our knowledge with others. This comes in many forms including knowledge base articles, blog posts, and wiki pages. If you find yourself providing instructions for something or giving a detailed answer to a question, we would bet that there are good odds that this situation will arise again and someone, possibly even you, will be re-iterating, or even worse, once again searching for these answers. It can be well worth taking a few extra minutes to publish your work somewhere for future reference. It has been said that "an email helps one person, an article can help thousands." We find this to be very true and we hope that you'll also take it to heart.

What's that you say? This sounds like it's going to be difficult, and possibly expensive? It certainly doesn't need to be and there isn’t a sales pitch coming. It is true that it takes some effort to establish systems for storing the information, and the habits to keep using those systems, but it doesn't have to cost you anything. You're probably already using tools that could be leveraged to start building a library of information that will save you and your organization a ton of time down the road.

Let's explore a few types of things you might want to share:

Guides and How-To's

These are probably one of the best things to put in your library. You know, that report you have to run at the end of every quarter but always spend an extra 20 minutes trying to remember exactly what you did last time? Just take a few moments as you go to write down what you're doing and once you're done doing the thing you'll have a handy guide for next time. Bonus points for including screenshots of your process!

Some things we have guides for at DevResults:

• How to connect to the VPN
• How to install [software package X, Y, or Z]
• How to refresh the training site each month

Checklists and Templates

Sometimes it's just nice when things are consistent. Whether it's structuring meetings the same way, making sure a task is performed properly, or even letting your teammates know what you got done this week, there's a lot of value in having predictable output. Checklists and templates are a great thing to share with your team to make sure no matter who does the thing, it always includes the same bits you'd expect it to.

Some examples:

• Weekly all hands meeting template
• New employee onboarding checklist
• Packing list for the conference booth (never forget the swag again!)

Knowledge Is Power

Regardless of the tools you choose to use or how you get started, we think that having shared access to knowledge is a wonderful thing. As you grow the habits of collecting and using information you'll find that everyone becomes empowered and there's a lot less time spent (re-)explaining things. So next time you're about to hit send on that email full of instructions on how to do that thing no one can remember how to do, please consider writing it down somewhere that people can find it and sharing the link instead.

The International Aid Transparency Initiative (IATI) is a global initiative to improve the transparency of development and humanitarian resources and their results to address poverty and crises. Tracking the response to Covid-19 is no exception. Whether you’re required to publish IATI data by your donor or not, doing so is important, as we have learned from previous crises like the Ebola epidemic.

Over ten years since its founding, IATI provides the world with an impressive and unprecedented view of where development and humanitarian resources are spent and what they are achieving. Even so, many organizations small and large do not voluntarily publish their data, and many publish only the bare minimum as required by their donors. Under the best of circumstances, this is a shame, but in the midst of a pandemic, it vastly underestimates the scale and scope of our collective response efforts.

Data on a crisis, for a crisis

Organizations face many challenges when deciding how (or whether to) publish IATI data: hesitancy to open up internal data, privacy and security concerns, lack of technical clarity with the IATI data standard and available tools. We take these challenges seriously at DevResults, which is why we have advocated for more attention and investment in open data, done our utmost to guard our users’ data security and privacy, and worked hard to reduce the technical burden of publishing IATI data.

The new guidance from IATI — developed in coordination with the World Health Organization (WHO), UN OCHA Financial Tracking Service (FTS), and the Humanitarian Data Exchange — spells out three simple additions to standard IATI activity files to ensure that data on Covid-19 is discoverable by users:

1. Add “COVID-19” somewhere in the activity title
2. Mark relevant Covid-19 response activities and transactions as ‘humanitarian’
3. Link to the Humanitarian Response Plan and GLobal unique disaster IDEntifier (GLIDE) codes pertaining to Covid-19

DevResults enables you to set and forget these items, automatically ensuring the required data appears in all subsequent activity files that are generated.

But to what end?

Why should organizations take the time to publish details on their programming using some arcane data standard to an anonymous audience? Where does this data go, and who uses it?

Perhaps it would be useful to see how this data is utilized and displayed downstream:

• Development Portal (D-Portal), helps users explore IATI data to find out who is doing what, where, and to what effect. A simple search for ‘Covid-19’ reveals that several institutions are already publishing their outbreak response efforts.
• Donor dashboards showcase IATI data from activities they fund, such as DFID’s DevTracker, or even data from across the development cooperation landscape, such as USAID’s new Beyond USG tool on the Foreign Aid Explorer.
• UN OCHA’s Financial Tracking Services (FTS) aims to present a complete picture of all international humanitarian funding flows. Organizations can report data to FTS using their IATI files to avoid duplicate efforts. The FTS summary of the Covid-19 Humanitarian Response Plan shows data on funding linked to the appeal.
• Aid Management Platforms (AMPs) are a communication tool that enables donors and recipient governments to open and share aid data to enhance collaboration and information sharing. Many AMPs utilize IATI data, and some have public dashboards, such as Madagascar and Senegal.
• Researchers use IATI data to track and study responses to crises such as Ebola or Covid-19 to determine what was achieved, if pledges were fulfilled, and whether trust and accountability were created in the process.

...and this is just a taste of what’s out there!

The economics of data

IATI and other open data platforms can be thought of as public goods, and as such they suffer from free-riding; the private cost (to your organization) of publishing open data exceeds the private benefit you derive from the published data. As actors in the business of creating public goods — in healthcare, governance, crisis response, public services, and more — we should be willing to bear these costs for the collective benefit of all, not to mention our own adaptive management and learning.

Our driving mission at DevResults is to reduce the costs — in time, in effort, in funds, and in technical knowledge — to act in the public interest. If ever you’ve considered opening up your programming data or publishing IATI data, now’s the time!

How we can help

DevResults is committed to providing our platform for organizations managing the Covid-19 response and will rapidly deploy a fully-functional DevResults site available within 24 hours. For more information on our platform and the work we are supporting around the world, please contact us at info@devresults.com.