2015 Retrospective

The last year saw a lot of changes here at DevResults, and in good M&E fashion we've decided to review a lot of that to craft our strategy and focus for calendar year 2016. Overall, 2015 was a year of internal change for us--new staff, systematizing a lot of our support and work processes, and technical infrastructure changes. All of these have been iterative processes, and we wanted to highlight some of those efforts and lessons learned. Expect a follow-up post with our 2016 vision!

Broadly, our work tends to fall in one of three buckets: people, process, and product. Here are some of our highlights and lessons learned for 2015.

People

Near the end of 2014, we added 2 new staff to the roster (Diego and me), and at the start of 2015 we added one more engineer, Scott¹, taking us up to a full-time staff of 9. This seems like a small change (6 to 9), but it made us re-evaluate a lot of our internal processes. Our team remains mostly remote, so we have spent the year testing out various in-person ways of getting together to get more done (company retreats, team meet-ups, etc.). In 2015, we tried Buffer's model of having three company retreats. We realized that this was just too much for our little company, particularly since most of our staff have kids. We're trying a single retreat in 2016 with more visits for smaller teams to push through major features. (For example, this blog post came out of a smaller team meeting in DC.)

In 2015, we felt like we could be delivering more to clients but were being hampered by our own processes. Now, though we continue to refine our processes, we feel like we're being hampered by a lack of people power. We are debating hiring another 1-2 engineers and are in the midst of interviewing for a third data hire to join Leslie and me, so 2016 will likely see us break into double digits for the first time. We're hoping that our process changes will make integrating these new hires fast and relatively painless.

Process

We are a software company; why on earth am I talking about process before product? Because we've seen the most significant amount of change on the process side, and in many ways, I think those changes have an even bigger impact on customers than product changes do. Growing our staff and client base has challenged us and made us re-evaluate how we do all kinds of things, and we've tried out a number of changes to be better:

1 Support Requests: In-app Help and Trackable Tickets

When we began 2015, customers were still emailing individual DevResults staff directly to report bugs. Our belief with support had been that having a designated person was what most customers wanted. But this wasn't sustainable for all kinds of reasons: what happened when your person went on vacation? What if your contact was Leslie or me but you were really reporting a software bug that should be looked at quickly by an engineer? It put a ridiculous amount of pressure on checking inboxes and tracking emails, which is horribly inefficient, and though we did our best, some questions or problems inevitably fell through the cracks. Plus, requiring customers to use email meant that we often had to have several exchanges before we knew what page you were on, which DevResults site you were accessing, and so on, which made the resolution process that much slower.

We have approached solving this problem in two ways: first, we implemented the Something's Wrong tool, which allows you to submit a help request directly within DevResults. This captures quite a bit of information, which greatly reduces the amount of back and forth at the start of each ticket. Second, after we played around with a group mailbox (which suffered from most of the same problems and a bit of confusion over who was responsible for what), we implemented a help ticketing system, built on FreshDesk. We didn't actually blog about this, so it might be news to you. But every request submitted through Something's Wrong and every email sent to help@devresults.com goes into this ticketing system. (And if you haven't actually created an account on the site yet, you should! It's a much easier way to track all the requests/bugs you've sent us than trying to keep track of emails. Just use the email address you use to log into DevResults and all your older tickets will be displayed once you create an account.)

Both these changes have greatly improved support turnaround times and have drastically reduced the number of communications that disappear into the email ether. We are continuing to refine our ticket workflows, but overall this is head and shoulders above where we were at the start of 2015.

2 Engineering Workflows

In the last year, we've tried several changes to how we decide and prioritize which new features and functionality we add to the software. We started off the year deciding that our non-engineering staff (Josh, me, and Leslie a.k.a. JKL) should begin to set development priorities, since we had the most customer interaction. What we learned rather quickly is that setting priorities is only useful if people are held accountable for them. Over the summer, we experimented with more structured sets of work: "sprints" that were roughly a month long package of work, a mixture of large and small changes with the goal of getting them all out by the end of the month.

We discovered that, while sprints helped keep the team focused, we struggled estimating difficulty/effort on a lot of tasks, and we were getting hung up on that. We spent the fall with a much looser structure: 3-4 high priority items for each month and then a prioritized list of small but important tasks that engineers could draw from at-will. This seemed to work a bit better, and we are continuing to adapt this structure moving into 2016. (There's a lot more to be said about this--check out the 2016 vision post once it comes out!)

3 Better Communication

It's pretty easy to tell when we shifted to sprints, because that's when I started doing blog posts all the time. One of the biggest gaps we noticed in 2015 was people either not being aware of new features or not understanding how to use them. This was some of the best feedback we could have gotten. I'm not sure why we thought people would just either a) obviously notice, or b) psychically know when changes rolled out, but it turns out that you all are just as human as we are and were busy with other things. So we decided to try to close that gap in a fairly easy way: blog posts for new features and at the end of each sprint to highlight smaller changes that might impact you, with site banner announcements to let you know they were available.

This decision was informed by two things. One of the pieces of software we use regularly, Asana, does blog posts for features/updates, and we liked that model. It's an easy way to reach all of our customers at more or less the same time. We also liked the idea of doing blog-driven development. Although we don't follow that practice to the letter, knowing that Leslie or I'll be writing a post encourages more communication between us and the engineers doing the work. This increased collaboration has inspired a number of smaller tweaks before we make changes live, improving the quality of our releases. It also lets the engineers make sure we tell customers about the things they're most proud of. So far the response seems to be that it's working for you a lot better than the old system, so we plan to keep doing it!

On a small but no less noteworthy note, we transitioned our own help documentation to a new provider which helps us better highlight brand new or updated help pages. This should make it easier for you to update any internal documentation you have by checking what we've updated or added recently.

Looking ahead, we're also going to be contacting site Owners in advance of major new features/changes. This is a fairly significant change; in the past, we've mostly just rolled out new features. Our intent here is to give you a bit of advanced notice with access to a copy of your site with the new functionality. This way, you can test it out, give us feedback, update your own workflows or documentation, and so on. We'll have more info on this in the 2016 vision blog post.

4 Client Onboarding

Since most of you were already clients in 2015, you won't have noticed this much, but a couple of you will! Going into 2015, our onboarding process went something like this:

1. Josh worked out all the nitty gritty details on contracts and got things signed.
2. He gave Leslie or me a run-down on the client and usually an email introduction.
3. Leslie/I would work with clients to find out who their implementation lead(s) were and walk them through setup, asking a lot of questions along the way.

This wasn't awful, but it meant that some of the key players weren't involved before a contract was signed: Leslie/me and the client implementation leads. This meant that clients didn't always have a clear idea of what they were signing up for and along the way we often discovered surprises. We don't mind surprises, but we don't like finding them out when we've already built a site and are in-country training folks. We felt like we needed a more seamless hand-off to ensure everyone was on the same page and expectations were properly set for everyone involved.

Our solution to this has been the creation of our Readiness Assessment. This is an online survey to collect a bit of information up front from new clients before they begin the setup process. It has helped us better identify and describe key considerations and pitfalls during DevResults setups and implementations. It's also gotten the data science staff interacting with client implementation leads well before contracts are signed, which helps flush out more of those surprises well in advance. Now hand-offs from Josh to Leslie/me are much smoother and far more detailed:

1. Josh gets to know a potential client and sends them the readiness assessment.
2. They fill it out; Leslie or I review it to follow up with them on any areas we need additional clarification.
3. We send a full write-up evaluating them on readiness in several different categories, including risks, change management considerations, and training needs.
4. Josh works out any remaining nitty gritty details on contracts and gets things signed.
5. As soon as a contract is signed, we have a kick-off meeting with the implementation leads we've already been in contact with and can provide a setup template and sandbox immediately because we already have all the information we need.

We're so excited about this that I'll actually be writing an entirely separate blog post on it, so if you're interested, keep an eye out for that.

Product

Changes in the software itself are the most obvious, which is part of why I've left them for last. If you've been actively using the software or following the blog posts, you have a pretty good idea of what's in here, anyway. But here are some of the highlights of what we've done in the last year:

• Entirely New Functionality: This year saw us release three completely new features in DevResults: Diagnostics, Single Indicator Pito, and Custom Queries. We're already halfway through our next major new feature: Reporting Cycles.
• Performance Improvements & Rewrites: 2015 was the year of performance improvements. We optimized several scheduled tasks that seemed to be causing slowness for customers in certain time zones. More importantly, we rewrote several key pages, including the Indicator Definition page and the Index pages. Both of these endeavors are part of our work to speed up response times and have DevResults be more helpful, and both achieved these goals. The index page performance is roughly 4x faster now. The Indicator Definition page is a good showcase for patterns we intend to add everywhere: more warnings/alerts to point out when something's misconfigured and helpful validation/confirmation of changes. We'll be leveraging work and lessons learned from both efforts for every future page rewrite.
• Migration to Microsoft Azure: We moved all of our servers and databases to Microsoft Azure earlier in the year, and have been continuing to refine our architecture as a result. This has at times been a bumpy process as we've had to get used to Azure's quirks, but it has allowed us to do things like move all file storage into a separate storage container, which greatly improves our database backup and recovery processes (see this blog post for a bit more on that). We are still evaluating how we could maximize benefits from Azure.

What's Next?

A whole bunch! We've factored in a lot of our lessons learned as we've planned for 2016, and more of that will be revealed in our upcoming 2016 vision blog post.

Have feedback on any of the 2015 changes or what you'd like to see in 2016? We welcome comments below!