Using the Weekly API Science Report to Improve Customer Experience

Your API Science account automatically provides you with the “Weekly API Science Report” which is delivered by email to the email address associated with your account. Here’s an sample email for an account that runs two API Science monitors:

weekly-report-2The report proceeds from a performance overview for all of your monitors down to the details of performance for each API monitor. If the overview sentence states that you had zero issues, then you are assured that your customers probably experienced few or no outages for your product in the past week.

Of course, even if your monitors experienced no outages, that doesn’t guarantee that your customers experienced no outages. For example, if you run your monitors once per hour, and a critical API went down for 30 minutes after your last check and before your next check, your customers would have experienced an outage that would not be seen by your monitors.

This is a good reason to have fine-grained monitoring for your product’s most critical APIs (internal or external). API Science provides the capability to run API checks once a minute. Running your critical monitors at this cadence would capture these anomalies.

A primary benefit of the Weekly API Science Report is that it can alert you to problems your team didn’t notice (as they worked on upgrading your product, etc.). These problems will have been noticed by your customers who were trying to use your product at the times when the outages occurred.

The “Monitor Overview” section combines the check results for all of your monitors into a single table. The table lists:

  • Total – the number of monitors included in the report
  • 100% Uptime – the number of monitors that passed all checks during the week
  • Issues – the number of monitors that had at least one failed check
  • Failed Checks – the total number of checks that failed for all monitors
  • Overall Uptime – the percentage of all checks performed for all monitors during the week that were successful

This table is the basis for the summary statement that appears above it. In this example, with a 99.41% Overall Uptime, the stated conclusion is: “Awesome!” In other words, if you’ve designed your monitoring to accurately represent you customers’ experience, and you receive this kind of weekly API Science report, then your team can probably continue working on improving your service, without worrying about recent performance issues.

But, say the top of your weekly report looks like this:

weekly-report-64

“Uh oh…” the summary line concludes. Indeed! Because if these monitors represent the core of your product’s utility for your customers, then, as the Monitor Overview table states, your product appeared down to your customers more than 14% of the time in the past week — which translates into more than three hours per day, or an entire day out of the week! This is an “uh oh” situation, because you have competitors, and maybe your customers went to their sites when your product was down.

When this happens, it’s time to proceed into the Monitor Details section of your weekly report. The monitor overview states that three different monitors reported issues, with a total of 170 failed checks. I won’t post the entire table due to its size, but scrolling down the Monitor Details table, we find:

weekly-report-64-detail-heading

weekly-report-64-detail

We’re running these three monitors once an hour. The Facebook and iTunes monitors had zero failures in the past week, but the Twitter API monitor never succeeded. If a core feature of your product involves this Twitter API, then from your customers’ point of view, your product has probably been down the entire past week! Uh oh… yes.

Note that you can have a pretty high overall uptime percent — but if the core APIs that your product depends on have a very low uptime percentage, then your product is mostly down for your customers — and you’re surely losing some of them.

Most start-ups have relatively small teams, and keeping tabs on performance as seen from the customer’s point of view can occupy valuable time. The Weekly API Science Report can alert your team to issues that weren’t noticed as the week progressed and the team worked toward the next product release. Receiving a weekly report that points out problems will alert you that effort has to be, at least temporarily, diverted toward addressing downtime issues.

–Kevin Farnham