Does It Really Work? Testing API Calls (Part II)

Testing APIs is complex, but in the end, we’re making a request, look at the response, and need to consider – did it work? Did it affect the system in the way I expect it?

Last time, we created a record, with a POST call, which looked like it worked. But then we looked for the created record in two ways: we issued a specific GET for that record, and a GET all records. Neither brought it back.

A mystery waiting to be solved? Here’s the solution.

If you read through the documentation, you’ll see that JSONPlaceHolder is really a mocked site (it even says so in their subtitle). It returns mocked answers. So adding that blog post record didn’t change anything in it. It will always return the original 100 records. It ignores adding records, or deleting them, or changing their data.

Which is a nice anecdote, but what have we learned? First, read the docs. But let’s talk about the general case of testing APIs.

APIs Can Be Deceiving

First, even if one request looks like it’s working, it doesn’t mean it actually does.
We looked for clues if a request is working (looking at the status code and response). But in order to trust that it actually does, we needed to check THE FEATURE. Not just the specific API.

The “feature” here was basic CRUD information – storing new data, and looking for it. Only by failing the GET we understood that the POST lied to us.

While we stopped when we smelled a rat, we didn’t try a handful of other methods that the site says it supports: Patching, updating and deleting – which are all part of the blog post management feature. When checking real features like that, we should check those too.

Here’s an example of DELETE-ing a record:

As you can see, it lies to our face – a 200 status code for deleting a non-existent record.

In real life applications, to check the feature, I would test for all these data related operations, and if they work as I expect, then I will report the feature is working. Not just the return code, but also if that data is manageable, and if I can do something with it later.

But that also, doesn’t always tell the whole truth.

The HTTP Honor System

When we did the HTTP POST request, we looked at the status code, saw that it was 201, and felt that it’s working – we thought we’ve added a record, and a 201 “Created” response is a good response.

Now, what happened if it had returned a 200? Just “OK”. Would that be a bug?

Well, it depends. After all, nothing is really working. Returning 200 instead of the 201 is really a small bug compared to what we have here.

But let’s pretend it did work. And let’s pretend the documentation doesn’t tell us what we should expect. Is a 200 code a wrong status code for creating a record?

Legally speaking, nope. There are no rules as to what requests should return. There are conventions – it would be nice to get a 201 for a CREATED record. But nobody enforces this. There is no HTTP police.

The whole thing is built like an honor system, and is completely in the hands of developers. If the API is a contract for external users, we can treat a wrong status code as a bug, but again, that is probably not a big problem as other things. If it’s an internally used API, that matters even less.

Another example is with deleting records. In financial or medical applications, anything that’s regulated, you can’t just delete data. That means that even if the DELETE request returns a 200 code, the data may not have been completely erased. We need to understand that feature in context, to evaluate if it’s working correctly.

The way we think about features, not focusing on the APIs, holds true for bugs too. If the users perceive broken functionality, it comes from the features, not from specific APIs. Oh, and usually the fix is not around a single API either.

We need to think about features and behaviors across multiple APIs, when we’re testing. Don’t believe your eyes, as single APIs don’t tell the entire story. And don’t trust status codes either, they don’t always tell the (whole) truth.

And while you contemplate right and wrong, check out my “Testing APIs with Postman” workshop, where the right amount of doubt is discussed, and how to manage it.