Argot: a lightweight composable test framework for Go

By: on April 21, 2017

In a current project we’re writing a number of fairly small REST HTTP servers. There are probably going to be around 10 of these in total so I guess that makes these ‘deci-services’. As part of the testing approach, we wanted to be able to write some end-to-end tests and soak tests, and so went looking for a nice test library with support for making and testing HTTP calls. These sorts of tests are quite nice because if they’re written carefully then it’s perfectly safe to run them against the production deployment to check everything’s working as expected. Now the Go standard HTTP package is pretty nice so there’s no real hardship in simply using that directly, but it would be nice to have some asserts around response codes, matching on the response body, and so forth. A little bit of searching later and we came across baloo—the API looked nice, and the feature set pretty much matched what we needed, so it seemed like a decent first choice.

A couple of weeks passed and as we gained experience with baloo we noticed some issues. Firstly, I’m not really a fan of the builder pattern (or fluent interfaces) for this purpose: one conceptual issue and one concrete. The conceptual one is that really a builder pattern is configuring some set of parameters which you’re later going to have interpreted in some way. My issue is that using builder pattern to construct a list of instructions is quite a different thing. Plus in the examples you can see a slightly awkward mixture of configuration and steps. For example (code comments are added by me):

func TestBalooClient(t *testing.T) {
  test.Post("/post").                      // config of call
    SetHeader("Foo", "Bar").               // config of call
    JSON(map[string]string{"foo": "bar"}). // config of call
    Expect(t).                             // force the call
    Status(200).                           // assertion step
    Type("json").                          // assertion step
    AssertFunc(assert).                    // assertion step
    Done()                                 // tidy-up step
}

To me, it makes much more sense to have an explicit list of steps and not abuse the builder pattern like this.

The concrete issue is that using the builder pattern makes extension very hard. Yes, in Go you could do an anonymous embedded field to do a subclass-like-thing, but the builder pattern would still break because now anything that is doing a return this in the parent type is returning the wrong thing. Even in languages with complex type systems, this doesn’t work out properly.

Finally, there was a real deal-breaker: the whole library was not thread-safe. We found that out the hard way when writing soak tests that just hammered the deci-services with concurrent requests. Now by no means is this meant to be condemnation of the library or its authors. The authors have clearly put lots of work in, and the library worked well initially for us. We probably just have requirements that were outside the scope of the baloo project, and it’s pretty likely we’re fussier and more demanding about API design and functionality.

So at this point we had a better idea of what we wanted and needed, and so we had a choice: either spend hours trying to search around finding something better, or spend hours writing our own. Now I’m quite a big fan of reinventing the wheel: it can always be made rounder, and you tend to learn things as you do it. People typically harp on about NIH (Not Invented Here) syndrome but fail to consider the risks of using libraries. Obviously, it’s a bit glib and cheap to mention the leftpad debacle, but the point remains that every time you add a library to a project, you are giving up an element of control.

The odds are that a candidate third-party library doesn’t fit your needs perfectly, so that means that you’re going to have to learn the library in some detail: where the warts are, where the leaky abstractions are, where the extension points are, and how to live and work with it. These are things you’d have to learn if you wrote your own too. It could also be that the library is huge and you only need a tiny bit of it, which may mean that the learning curve is substantial. That will also place a knowledge burden on everyone who looks after this project after us. And of course you’re just hoping that the library authors will continue working on it and looking after it for quite some time and never make a really bad mistake. In some cases, you really need to think about the security implications too. For example, I convinced a colleague not to use a library that wrapped the AWS S3 API and presented a nicer API. Why? Because that code will have access to everything a client has stored in S3, and none of us can vouch for the current intent of the authors of that library, let alone the future authors. Sure, the AWS APIs are not pretty, but I didn’t want to take the risk. So, as always, it’s a trade-off: there are pros and cons of re-implementing libraries and functionality.

Given I had a reasonable picture of what we wanted, I decided that spending a few hours one Saturday morning attempting to write my own was a sensible use of time, given the various trade-offs. If it didn’t work out then we’d go searching for something else.

Presenting Argot

Argot consists currently of two parts. The first is the execution engine, which is mind-numbingly simple. You construct a list (or slice) of Steps. Then you run them in order. That’s basically it. Yes, in the Haskell world, this is pretty much just sequence. Execution stops when you run out of steps, or with the first step that returns an error, and it returns the error and all the steps that have been executed.

In typical Go fashion, we have a basic interface for a Step so that anyone can cheaply build a step:

type Step interface {
  Go() error
}

And then we also do the typical type aliasing of a function so that we make construction even cheaper:

type StepFunc func() error

func (sf StepFunc) Go() error {
  return sf()
}

To aid debugging (e.g. when you get back a list of steps that were executed you probably want to log them if there was an error), it would be nice if a Step could have a name too, so we resort to a struct, anonymously embed a StepFunc in it, and have a simple constructor function for good measure:

type NamedStep struct {
  StepFunc
  name string
}

func (ns NamedStep) String() string {
  return ns.name
}

func NewNamedStep(name string, step StepFunc) *NamedStep {
  return &NamedStep{
    StepFunc: step,
    name:     name,
  }
}

Finally, we do an alias of a slice of Steps and have the engine function which runs it all:

type Steps []Step

func (ss Steps) Test(t *testing.T) (results Steps, err error) {
  ...
}

So the engine is trivial, and the intention is that you perform composition by building up longer lists of Steps. More complex things are certainly possible, but this seems fine for now.

Completely independently from the engine is a basic state container for an HTTP call: HttpCall. Initially, we have some assert methods that should be used within a Step where you need to ensure the HttpCall is in a certain state; for example that there is a Request but there is currently no Response. There are then some idempotent methods that use these assertions and make changes to the state of the HttpCall: for example ensuring the actual HTTP call has been made, or ensuring that a response body has been read.

With those preliminary building blocks out of the way, we can then use these to build Steps: e.g. creating a request, or requiring a particular response status code, or that a response header contains a string, and so forth.

All of this means that we keep quite a nice high-level declarative structure to our tests, but: a) the engine is now nicely decoupled from the state container and so the engine could easily be reused for other purposes; b) it’s clear we’re building up a list of instructions to be executed; c) and if necessary, by anonymously embedding an HttpCall in a further struct, we can easily extend the set of available Steps relating to an HttpCall. Putting it all together, we can write tests like this:

func Testify(t *testing.T) {
  req := argot.NewHttpCall(nil)
  defer req.Reset()
  argot.Steps([]argot.Step{
    req.NewRequest("POST", server+"/api/v1/foo/bar", nil),
    req.RequestHeader("Content-Type", "application/json"),
    req.ResponseStatusEquals(http.StatusOK),
    req.ResponseHeaderEquals("Content-Type", "application/json"),
    req.ResponseHeaderNotExists("Magical-Response"),
    req.ResponseHeaderNotExists("No-Unicorns"),
    req.ResponseBodyEquals("Attack ships on fire off the shoulder of Orion...\n"),
  }).Test(t)
}

And of course it’s all thread/go-routine safe now, so we can write simple end-to-end tests this way, and we can write soak tests that hammer the deci-service from dozens of different concurrent go-routines. In general, I quite like the construction of lists of instructions and then a mini-interpreter of those instructions and I’ve written about this before: it opens up many options for generating tests mechanically; for example ensuring that every permutation of an API is exercised, which can be extremely powerful for catching bugs.

So that’s all it is. Argot is available from GitHub under the Apache License version 2.0.

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

*