7 Ways to Validate that Your APIs Are Working Correctly

Online products that provide a complex service typically rely on Application Programming Interfaces (APIs), utilizing online resources to provide basic information. The added value is in how their own software uses this information to create a new product that is of value to their users. The APIs that are utilized to create this unique product often include internal APIs developed by the company’s own developers.

A failure of any of the APIs that are involved in creating the final product means that the product will appear either down or partially down to the company’s user base. Therefore, all APIs that are critical components of a company’s product must be monitored, so that appropriate actions can be taken when an outage of some sort occurs.

There are many ways in which an API can be not fully functional. These include:

  • Invalid response codes
  • Invalid response headers
  • API time-outs
  • Slow API response with respect to response data bytes
  • Incorrect required data in JSON responses
  • Missing required text in response bodies
  • Slow response for customers in specific locations

In this article I show how to develop API monitors to validate that all of your API monitors (external or internal) are fully functioning with respect to these seven key problems. In order for your product to succeed, your team needs to know if any of these problems becomes an issue at any point in time.

Validating HTTP Response Codes

JavaScript can be applied for fine-grained validation of the HTTP Status Code for each API response.

The status code normally associated with a successful response is 200 (“OK”). However, all 2xx status codes indicate that from the server’s point of view a valid request was received, it was processed, and an appropriate response was delivered to the client. The 2xx status codes that are greater than 200 provide additional information about the processing performed by the targeted server, or by a proxy server along the Internet path between the client and the target server.

The Internet is a complex system. Request and response traffic can be routed in many different ways, across multiple chains of proxy servers that relay the information packages. Another complication is caching, wherein data is stored in temporary locations along the path between a client and server. Caching is done to reduce the overall traffic level, thereby increasing the speed individual clients experience. Routing and caching are just two of the factors that can cause widely variable response times when the same request is repeatedly sent from the same client to the same server (as in one of your API Science  monitors). For example:

response_time_variationThe 2xx HTTP status codes report information about what happened at the server and what happened as the response package traversed the path between the endpoint server and the requesting client.

This three-digit code can be used by your monitor to let you make decisions about when your product will consider the API up, and when it will consider the API down. If the status code indicates that, while the request was successfully processed, the response body will not include information your product depends on, then there’s no need to even attempt to process the body: from your product’s point of view, the API is down.

Here’s a brief review of what the 2xx HTTP status codes mean (in addition to “Success”):

  • 200 OK: no additional information is provided
  • 201 Created: a new data item was created on the server
  • 202 Accepted: the server has received the request, but hasn’t finished processing it
  • 203 Non-Authorative Information: a proxy server sent the response based on a response from the endpoint server, but the response isn’t identical to the response the proxy received
  • 204 No Content: the server processed the request and returned no content
  • 205 Reset Content: like 204, but the requester should reset its data view
  • 206 Partial Content: the content is a subset of the entire data set because the request asked for this
  • 207 Multi-Status: the response body is an XML document containing additional status information
  • 208 Already Reported: a follow-on related to 207
  • 226 IM Used: the response is a representation of one or more instance manipulations (IMs) applied to the current instance

If the response from this API is critical for your product, clearly status codes 200 and 201 are perfect, but status code 202 makes the API down. The remaining status codes are more complex: depending on the software that creates your product, API responses with some of these status codes may represent responses that can be processed.

Assume we’d like to have the API check fail if the HTTP status code is any value other than 200 or 201. Here’s how we implement that using API Science’s JavaScript validation. Edit the monitor, click “Show settings” and select “JavaScript” in the “Validations” pull-down menu:

xkcd_javascript_entry

In the text box, enter:

var statusCode = context.response.meta.statusCode;

assert( statusCode == 200 || statusCode == 201, 
        "Expected a 200 or 201 response");

Then, at the bottom of the page, click the “Save Monitor” button.

We’ve now created a JavaScript validation that will say the API check failed if the response HTTP status code is anything other than 200 or 201.

Validating HTTP Headers

Examining the HTTP Headers that are returned with an API’s response is another way to validate that an API is functioning properly. I’ll use the XKCD API (a simple API that returns data about online comics) to illustrate how to do this.

Here are the response headers from a typical call to the XKCD API:

xkcd-response-headers

You might ask: “Why check the response headers? Isn’t the body all that really matters?” One reason is that, while the body is indeed what matters most in terms of data, the body of an API response can have any structure; whereas HTTP headers are defined by RFCs (Requests for Comments) that are agreed upon by the global World Wide Web community. Currently RFCs 7230 through 7237 define the requirements for HyperText Transfer Protocol (HTTP). Thus, unlike the structure of a response body (which can be entirely freeform), the structure of the HTTP headers segment for every API response is known. This makes analyzing the response headers relatively straightforward, whereas analyzing the response body may require more complex coding.

The HTTP response headers for my monitor’s call to the XKCD API provide numerous opportunities for validating that the latest API check succeeded in the manner my application requires. For example, the “Content-Type” header provides the MIME type with which the response body is formatted (“MIME” is an acronym for “Multi-purpose Internet Mail Extensions”).

The Content-Type header illustrates the value of applying JavaScript to validate headers. If your application expects to receive a JSON (JavaScript Object Notation) response body, then from the point of view of your application, this API is down if the response’s Content-Type header doesn’t include application/json.

You can verify this using an API Science JavaScript validation. Reviewing the XKCD response headers in the example above, we see that, indeed, the Content-Type header includes application/json.

Here’s how we set up a JavaScript monitor validation that will have an API check fail if the Content-Type header doesn’t include this. I edit my XKCD monitor and select “JavaScript” in the Validations pull-down menu:

xkcd_javascript_entry

In the text box, I enter:

var contentType = context.response.headers['Content-Type'];

assert( contentType.indexOf("application/json") >= 0, 
        "Not a JSON response");

The contentType variable retrieves the value that corresponds to the Content-Type entry in the response header. That string is then evaluated using a JavaScript indexOf command. This command identifies the location of the requested string (application/json) within the entire string (contentType). The value 0 means the requested string is at the very beginning of the full string. If the requested string does not exist in the searched string, then a value of -1 is returned.

The assert statement checks if the JavaScript indexOf command returns a value greater than or equal to zero. If the assert fails, a “Not a JSON response” message is issued.

With this validation in place, your team will know if a critical API suddenly ceases to deliver body content in the format your application requires.

Timing-Out Slow-Performing API Requests

API performance varies depending on many variables, including the response of the Internet (for example, resolve time and transfer time) and the connect time and processing time that occur on the remote server. For example, my own experience with the World Bank Countries API has revealed that calls sometimes exhibit abnormally slow performance.

Unpredictable, though occasional, slow performance when you call an external API will make your own product appear to exhibit poor performance. It may not matter if the delay is relatively small (a second or two). But my monitoring of the World Bank Countries API has shown several instances where a single call to the API took up to 10 seconds to complete. No customer is going to want to wait 10 seconds (or a multiple of this if the API must be called multiple times) to see a response on their device.

The API Science platform provides a “Max Response Time (ms)” validation that lets you specify the maximum amount of time allowed for receiving the response to a request. If the response is not received before the specified time expires, the call is considered to have failed.

We can test this using my World Bank BRIC monitor. This multi-step monitor accesses the World Bank API for information about the countries economists call the “BRICs” (Brazil, Russia, India, China) — nations they predict will become prevailing economic forces in the coming decades.

Clicking “Edit” (or the pencil icon on the Dashboard) brings up the monitor’s edit page. Click “Show Settings” to see the details:

wb-bric-edit

The “Validations” pull-down lets you select the type of validation you’d like to perform:

  • Response Code
  • Regex
  • Validate JSON
  • JavaScript
  • Max Response Time (ms)

Select “Max Response Time (ms)” and enter a millisecond value:

wb-bric-max-resp-1000

Clicking “Test Now” runs the test, which in this case produced a valid result:

wb-bric-max-resp-1000-result

The Call Summary shows that the total time for completing this request was 57.68 msec. If we modify the “Max Response Time” value to a very low value (for example, 10 msec), then the monitor run should fail. This is indeed what happens:

wb-bric-max-resp-10-result

The server responded to the request and returned its result 26.15 msec after the request was initiated. But, since we defined a 10 msec maximum response time, and the actual response time exceeded this, the test fails.

Flagging Abnormally-Slow World Bank Countries API Calls

World Bank BRIC monitor tests typically complete in less than a second. However, there are occasional aberrations where the test takes up to 10 seconds to complete. Rather than have users wait 10 seconds after a button click, assume we want to get whatever BRIC data is quickly available, but tell the user “data is not currently available” for any request that takes too long. This would ensure that your user always receives a prompt response.

The table below lists the timings for a typical WB-BRIC monitor call, along with the millisecond timings for a call that took an abnormally long time (1796 msec) to complete:

Monitor Step Typical Long
1. Brazil 281 285
2. Russia 177 471
3. India 175 541
4. China 173 503

We see that, under normal conditions, the calls for an individual country occupy less than 200 msec (the timing for Brazil is affected by a longer resolve time, since that request is executed first by the monitor).

Assume we don’t want our users to have to wait more than twice as long as normal to receive their result. We can configure our World Bank BRIC monitor to validate that each monitor step must complete within 350 msec. If any of the monitor steps takes more than 350 msec to complete, that test will be flagged as failed, due to the maximum response time criterion.

I’ve configured all four steps in the BRIC monitor to validate that the Max Response Time is 350 msec. If any step fails this test, then the entire monitor run fails.

wc-bric-max-resp-350

I’ll let the updated monitor run now, expecting to see any excessively slow performance by the World Bank Countries API flagged by the new validation settings. This will let me formulate a plan for addressing World Bank API anomalies in the applications I provide to customers.

Time-outs with Respect to Data Size

Now it’s time to write some JavaScript that validates the API’s responses.

The API Science monitoring platform provides a developer with access to all available information related to an individual monitor check, including timing information, HTTP headers, and the body of the API response. This information can be evaluated using JavaScript to create detailed levels of validation that will let you know immediately if an API that is critical for your product’s performance goes down or becomes unreliable.

Timing and Data Size

Your product uses data extracted from an API (external or internal). Your customer clicks something and expects to see a result within the next second or two.

But, what happens if your customer has to wait 10 seconds to see the result they requested? Will they wait that long? Even if the page they requested from your app is ultimately received, if they had to wait 10 or 15 seconds to see that result, might they conclude that your product is, effectively, down?

For this reason, you need to know if an API that serves critical data for your product is suddenly experiencing unusual delays (even if the API ultimately delivers the requested information).

API Science’s monitors provide a “Max Response Time (ms)” validation. If the response from the API takes longer than n milliseconds, the API check fails, for example, this setting:

max_response_time_validation

will cause a check on my “br Ireland” monitor, when I call the World Bank’s Countries API from Ireland, if a response is not received within 100 milliseconds.

World Bank Countries API queries return a fixed content size. So, simply specifying a time limit can suffice for assessing whether the API is up or down with respect to your product’s needs.

But, what if the situation is more complex? What if your product can provide quick looks in response to some requests, but your customers can also request large, detailed reports? In this case, it’s safe to assume that when your customer requests a quick summary, they’ll want to see it appear quickly on their device; meanwhile, if they then click your “Show Me All the Details” button, they’ll be willing to wait longer to receive the result.

In this case, a mathematical equation defines whether your customer will view your product as up or down. Your reasonable customer will consider: “how long did it take to receive the data I requested, compared with the amount of data I asked to receive?”

Can you apply your API Science monitors to evaluate your product’s performance based on this type of question? The answer is: “Yes!” — if you use API Science’s JavaScript validation capability.

Editing my example “XKCD Monitor” and clicking “Show Settings,” I see:

xkcd_editIn the “Validations” pull-down menu, I select “JavaScript,” which opens a box into which I can type the JavaScript programming that I’d like to apply to each instance when my monitor runs a test on the xkcd.com API:

xkcd_javascript_entry

Here’s what I enter:

var timing = context.response.timing.total;
var size = context.response.meta.downloadSize;

if (timing > 5000) {
        assert(size > 10000, "Download size vs speed issue");
}

The timing variable returns the total number of milliseconds that took place between the API request being initiated and the full response being received. This is the value that defines success or failure when you create a “Max Response Time (ms)” validation (see above).

In this case, however, we want to measure time with reference to how much data our customer requested (assuming they will be more patient if they requested more data). So, we create a size variable that stores the number of bytes that were downloaded in order to fulfill the request.

Now, using JavaScript and API Science’s built-in Chai Assertion Library, we can calculate the bytes per millisecond for the API call. If it took more than 5 seconds to download the data, then we check the download data size. If the data size was greater than 10,000 bytes, we consider that fine, and the API check is considered successful; if the data size was 10K bytes or smaller, we consider that a problem that might make our customer consider our own product to be “down.”

Applying JavaScript validation to your API Science API monitoring lets you make these distinctions, and lets you alert your 24/7 team when an issue arises.

Using JavaScript to Validate API Response Bodies

Next, we investigate how you can use the API Science JavaScript validation facility to ensure that the API response body is valid.

We’ll use my XKCD monitor to see how this can be done. This monitor queries the XKCD API using a simple GET request sent to this API endpoint:

http://xkcd.com/1481/info.0.json

When the API is up, it returns an API response body similar to this:

{
  "month": "2",
  "num": 1481,
  "link": "",
  "year": "2015",
  "news": "",
  "safe_title": "API",
  "transcript": "((This is a faux-screenshot of a technical document))\n[[A figure sits at a computer upon a desk, apparently engrossed in the document which we now see before us.]]\nTITLE: API GUIDE\nRequest URL Format: domain\nuser\nitem\nServer will return an XML document which contains the requested data and documentation describing how the data is organized spatially.\nAPI KEYS: To obtain API access, contact the x.509-authenticated server and request an ECDH-RSA TLS key...\n\nCaption: If you do things right, it can take people a while to realize that your \"API documentation\" is just instructions for how to look at your website.\n\n{{Title text: ACCESS LIMITS: Clients may maintain connections to the server for no more than 86,400 seconds per day. If you need additional time, you may contact IERS to file a request for up to one additional second.}}",
  "alt": "ACCESS LIMITS: Clients may maintain connections to the server for no more than 86,400 seconds per day. If you need additional time, you may contact IERS to file a request for up to one additional second.",
  "img": "http://imgs.xkcd.com/comics/api.png",
  "title": "API",
  "day": "2"
}

This is a JSON response body: a list of keys, followed by a colon, followed by a value. In your API Science monitor, you can use JavaScript to validate any of these fields.

Let’s say your API at this moment relies on the response “day” value to be “2”. If the API isn’t returning this JSON value, then, from the point of view of your product, the API is down.

To configure this validation into your API Science monitor, edit the monitor, and click “Show Settings”. In the “Validations” pull-down menu, select “JavaScript”:

xkcd_javascript_entryEnter the following code into the text window:


var body = JSON.parse(context.response.body);
var day = "2";

assert.equal( body.day, 
              day, 
              "Wrong day");

This code inspects the API’s JSON response body’s “day” variable, and tests if its value is “2”. If the value is “2” then the API check validation passes. If the values isn’t “2” then the check fails.

What will you see in your API Science monitor if this type of validation fails? “2” is the expected response for my XDCK monitor. If I edit the second line of my JavaScript validation code to be:

var day = "3;

and I then run my monitor, I will see this in the result:

xkcd_wrong_dayThe API is down from your product’s point of view, because your product depends on the value of the “day” variable for this particular query to be “2”.

JavaScript is a vast programming language that supports highly-detailed investigation of content that is returned in API response bodies. API Science provides unlimited ability for you to analyze response bodies to determine if an API you depend on is up or down.

Using Regular Expressions and JavaScript to Validate API Responses

Regular expressions (RegEx) are “a sequence of characters that define a search pattern, mainly for use in pattern matching with strings, or string matching.” Amazingly, the RegEx concept was invented back in the 1950s, by U.S. mathematician Stephen Cole Kleene, who was a student of Alan Turing, among others.

In his book Mastering Regular Expressions, Jeffrey E. F. Friedl writes:

Regular expressions are the key to powerful, flexible, and efficient text processing. Regular expressions themselves, with a general pattern notation almost like a mini programming language, allow you to describe and parse text. With additional support provided by the tool being used, regular expressions can add, remove, isolate, and generally fold, spindle, and mutilate all kinds of text and data.

Consider a response body from an API that your product depends on. Your product sent a request to the API, and a response was returned. But does that response contain the information that your product needs in order for your product to appear “up” to your customers?

API Science’s JavaScript validation capability provides you with the opportunity to apply regular expressions to analyze the content of an API’s response body, even down to the level of the text that’s in a particular JSON field.

My XKCD API monitor queries the XKCD API. A successful response has a body that looks something like this:

{
  "month": "2",
  "num": 1481,
  "link": "",
  "year": "2015",
  "news": "",
  "safe_title": "API",
  "transcript": "((This is a faux-screenshot of a technical document))\n[[A figure sits at a computer upon a desk, apparently engrossed in the document which we now see before us.]]\nTITLE: API GUIDE\nRequest URL Format: domain\nuser\nitem\nServer will return an XML document which contains the requested data and documentation describing how the data is organized spatially.\nAPI KEYS: To obtain API access, contact the x.509-authenticated server and request an ECDH-RSA TLS key...\n\nCaption: If you do things right, it can take people a while to realize that your \"API documentation\" is just instructions for how to look at your website.\n\n{{Title text: ACCESS LIMITS: Clients may maintain connections to the server for no more than 86,400 seconds per day. If you need additional time, you may contact IERS to file a request for up to one additional second.}}",
  "alt": "ACCESS LIMITS: Clients may maintain connections to the server for no more than 86,400 seconds per day. If you need additional time, you may contact IERS to file a request for up to one additional second.",
  "img": "http://imgs.xkcd.com/comics/api.png",
  "title": "API",
  "day": "2"
}

This is a JSON response body. The API Science JavaScript validation capability enables you to inspect JSON responses on a field-by-field basis.

Let’s say the information that’s critical for you product is located in the transcript field in the JSON body: if the word “faux” isn’t in the body, then the API is down as far as your product is concerned, because your product integrates what follows that text into the result you present to your customers.

How can you verify that the XKCD API is “up” for your product using your API Science monitor? It’s easier than you might think, using JavaScript and regular expressions…

Edit your monitor, and click “Show Settings”. In the “Validations” pull-down menu, select “JavaScript”:

xkcd_javascript_entryEnter the following into the text window:


var body = JSON.parse(context.response.body);

assert( body.transcript.search(/faux.*/i) > -1, 
              "transcript field does not contain 'faux'");

The regular expression is the text contained within the parentheses following search. What does this RegEx mean?

The /faux.*/ means the text should be searched for the text “faux”; if that’s not found, the search fails. The “.” means that “faux” must be followed by at least one character (i.e., the “faux” that was found cannot be at the end of the string). The “*” means that any number of characters can be in the string after “faux” (these characters would likely be the ones your product depends on). The “i” after the last slash means that the string should be analyzed in a case-insensitive manner (which might or might not be fitting for your particular application).

Save the updated monitor. Now, if a response is ever received where the JSON body’s transcript field doesn’t contain the word “faux” followed by other characters, your API Science monitor will declare the XKCD API “down” — and it will be time for your team to take appropriate action on your customers’ behalf!

Why Location Matters in API Performance Testing

In his 2014 book Flash Boys, Michael Lewis talks about a massively expensive project that burrowed through mountains, tunneled under riverbeds, dug trenches beside country roads, with the objective of creating “maybe the most insistently straight path ever dug into the earth” between Chicago and New Jersey. Why? So that “a one-and-a-half-inch-wide hard black plastic tube designed to shelter four hundred hair-thin strands of glass” could be inserted into the path. The gain? A reduction from 17 msec to 13 msec in round-trip network communication time between the company’s Chicago data center and a New Jersey stock exchange.

Latency matters. And not only for high-frequency traders in the financial markets. In reality, latency matters for any company whose product is an Internet-based service. What do your customers see when they click a button on your web page? If they don’t immediately see their web browser, tablet, or phone displaying the expected result, they’ll likely soon be looking for a new solution (i.e., visiting your competitors).

Your product probably uses internal APIs. You test these in your data center, and the latency, as expected, is low. But what does your customer who lives far away experience?

When you create a new API Science API monitor, you are provided with options to select the geographic location from which you’d like to run the API testing. To test the effect of the location from which you’re calling the World Bank’s Country API, I created four monitors, performing the exact same API test (requesting the country information for Brazil) from four different locations: Ireland, Oregon (northwest US), Tokyo, and Washington DC (east coast US).

The current results on my API Science Dashboard tell a story:

20160513_fig1

Uptime is 100% for all the tests. This is good. But the customer’s experience is also dependent on the delay between clicking the button and receiving the next page. In this case, the average response time for the World Bank Countries API when it is called from Washington DC USA is substantially lower than the response times from Europe, the western US, and Asia.

As it turns out, the World Bank is located in Washington, DC. So, calling the World Bank Countries API from your data center located in Washington DC provides great performance, where performance is measured by latency.

This vividly illustrates the effect of calling location on API performance. You can’t afford to test the performance of your API only from a computer located in your data center. That definitely will not show you what your customers around the globe experience.

There’s much more to discuss on this topic. For example, what happens if your product also utilizes third-party APIs, integrating calls to those APIs to provide data you then pass into your own API? In that case, the latency of their APIs will affect the latency your own customers experience around the globe.

If your customers are global, not just local, test the performance of your APIs from different locations around the world, using API Science’s “Run from” monitor configuration pull-down. You need to know what your customers are experiencing, whatever their location.

Conclusion

If you want to act when customers suddenly see your product as down, you need to know the current status of the APIs your product depends on. The API Science platform provides many methods by which you can analyze the status of your key APIs.

–Kevin Farnham