chettriyuvraj

Understanding Interfaces Through Golang: Mocking HTTP Server Response

Feb 21st, 2024

Rant

If you have ever run through a university-level course on a programming language (say Java), you have likely come across the concept of interfaces.

Even more likely is the fact that you have been given an example akin to:

interface Bicycle {
    void speedUp(int increment);
}

class NormalBicycle implements Bicycle {
    int speed = 0;
    void speedUp(int increment) {
        speed += increment
    }
}

class GearBicycle implements Bicycle {
    int speed = 0;
    void speedUp(int increment) {
        changeGear();
        speed += increment
    }
}

Can we be critical of such an example? An introductory programming course is like being hit with a barrage of ideas all at once, trying to piece things together. With that context in mind, such an example does kinda-sorta touch upon the concept of single interface, multiple implementations.

The trouble lies come the end of the course. Having never been exposed to how interfaces are used in real systems, the exam sheets end up being filled with examples like:

Birds flying()
Vehicles starting_engine()
Cats attacking()

I mean all of these are logical, if somewhat convoluted deductions from the Bicycle example, right?

Thoughts

If all you are left with at the end of a course/book/tutorial are these examples, is your understanding any further than someone who doesn’t code? Present the Bicycle example to any smart person and they can easily deduce the other examples.
Conversely, when you come across a real-world opportunity to use an interface, it won’t click. You end up with a problem to which you have seen the solution, but still don’t know how to solve.
Heck, even that would be a good programming assignment - NOT using an interface and then being shown how one would naturally fit
You become like that child who rote learnt multiplication tables. You know 6 x 7, but blank out on 7 x 6

Considering the above, I would think that how a lot of people understand interfaces is when they come across a well-written one, and life suddenly makes sense all at once.

Why not do exactly that, then?

We will use a fairly routine task as the basis for our understanding: mocking responses from an HTTP server to a client sending requests.

Real Life Scenario

You were asked to create an HTTP client that hits a server endpoint and gets the response
“Easy peasy breezy”!
You created a client. Grabbed the response. Ran it through a bunch of handlers depending on the HTTP status code.
Now you want to test it without hitting the real endpoint because, let’s say they charge 1000$ per hit

Solution 1: You can create an actual server using the NewServer() function defined in the httptest package, which is a reasonable choice.

Solution 2: Cleverly make use of an existing Golang interface: the RoundTripper. Of course we’re going to use this solution! What the hell was I on about otherwise? Trying to look smart with interfaces and all..

We will proceed in a bottom-up fashion, starting from the RoundTripper interface itself and building our way to the top of the pyramid: using RoundTripper to mock an HTTP server response.

RoundTripper and interfaces in Go

The net/http package in Go’s standard library defines the RoundTripper interface as

type RoundTripper interface {
    RoundTrip(*Request) (*Response, error)
}

It is an interface with a single method RoundTrip that takes in a Request and returns a Response.

For those of you unfamiliar with Go’s interface mechanism. Go uses the notion of _Duck Typing_ for interfaces. To match structs to interfaces, it uses the the duck test. According to the duck test, if it walks like a duck and it quacks like a duck, then it must be a duck. What does this mean? Any struct that has a RoundTrip method is automatically a RoundTripper!

How is this different from other languages? Take Java for example: objects have to explicitly declare the interfaces they implement i.e. class NewObject implements Roundtripper. Interfaces in Go are implicit - no need for such declarations.

HTTP Requests in Go

The first step was to understand the RoundTripper interface.

Next, we will look at how client requests are sent to the server in Go. Our goal is to mock an HTTP server response, but what for? To test our client requests! Understanding how client requests are sent will be key to this goal.

Go’s _net/http_ documentation provides the following example along with some commentary

For control over HTTP client headers, redirect policy, and other settings, create a Client:

client := &http.Client{
    CheckRedirect: redirectPolicyFunc,
}
resp, err := client.Get("http://example.com")


For control over proxies, TLS configuration, keep-alives, compression, and other settings, create a Transport:

tr := &http.Transport{
    MaxIdleConns:       10,
    IdleConnTimeout:    30 * time.Second,
    DisableCompression: true,
}
client := &http.Client{Transport: tr}
resp, err := client.Get("https://example.com")

Simple enough, right?

The way to execute an HTTP request is simply to create a client. Define (Optionally) a transport. Execute your request using the client’s baked in methods such as client.Get() or client.Post()

Clients and Transports: A closer look

Let’s take a closer look at the Client and the Transport struct

Our [Client](https://pkg.go.dev/net/http#Client) struct is defined as

type Client struct {
    Transport RoundTripper
    CheckRedirect func(req *Request, via []*Request) error
    Jar CookieJar
    Timeout time.Duration
}

Notice anything? Transport is a RoundTripper. Interesting!

That means it is implementing the RoundTrip method. Let’s us confirm this: the Transport struct is located in the transport.go file. A quick search for RoundTrip gives us no method definitions for RoundTrip. Weird..

Try it yourself: Go to transport.go, try to find the method definition for RoundTrip. No luck!

We do have a roundTrip though (hmm):

A bit more digging and we find RoundTrip here: a roundtrip.go file with only one method definition: for Transport.RoundTrip which calls the internal Transport.roundTrip we saw above.

Why this weird structuring? I have absolutely no clue! Like any other codebase, Go’s source code has rabbit holes you don’t want to get into. Hit me up if someone does find the commit and the reason for this, though!

So far, we have:

Seen that Client has a field called Transport, which is a RoundTripper
Confirmed that Transport is a RoundTripper by wading through Go source code
Understood that real codebases can be weird.

Clients and Transports: An even closer look

Let’s continue digging to see if we can make sense of what Clients and Transports actually are

The Client struct documentation additionally states:

A Client is an HTTP client. Its zero value (DefaultClient) is a usable client that uses DefaultTransport.

A Client is higher-level than a RoundTripper (such as Transport) and additionally handles HTTP details such as cookies and redirects.

The second line is particularly interesting. A Client is higher level than a RoundTripper (such as a Transport) and additionally handles HTTP details such as cookies and redirects. This seems to imply that Client is simply a wrapper around the Transport with additional options to control certain settings.

Let’s find out if this is true. We’ll go all the way into Client.Get() and see how an HTTP request is actually made.

Client.Get() seems to be a wrapper around Client.Do(). This must also apply to other methods like Client.Post() (go and check):

Client.Do() calls into the internal client.do(), which looks like a hot mess:

Amidst all the muck, we can make an educated guess that Client.send() is executing the request:

Client.send() does a small couple of things, but we are interested in the send() function that it calls. It also passes to the function, our Transport. This is getting interesting!

send() again is doing a bunch of stuff, but the comment tells us we are on the right track. Somewhere around here there is an HTTP request being sent:

Look closely and.. it’s our Transport! rt.RoundTrip seems to be the call that is executed to get the response. We can’t follow the trail anymore because RoundTrip is an interface method, remember? Every RoundTripper will have it’s own internal implementation of RoundTrip

What did we learn?

Client is simply a wrapper around Transport, which is where the actual magic of an HTTP request happens.
The RoundTripper interface and its RoundTrip method is basically an abstraction which defines the concept of sending a request and getting a response.

I am hoping you are beginning to see the outline of why interfaces are powerful..

Mocking an HTTP server response

If you haven’t joined the dots already. Here’s how you can test your HTTP Client Requests using the RoundTripper interface.

Here’s what we have. A client making HTTP requests, getting the response, and performing some manipulations on the response:

type CustomClient struct {
    client http.Client
}

func NewCustomClient(rt http.RoundTripper) *CustomClient {
    return &CustomClient{
        client: http.Client{
            Transport: rt,
            Timeout: TIMEOUT * time.Second,
        },
    }
}

func (f *CustomClient) MakeRequest(URL string) (string, error) {
    fmt.Fprintf(os.Stderr, "\n\nMaking new request to %v...", URL)

    resp, err := f.client.Get(URL)
    if err != nil { /* request itself returns an error */
        return "", ClientUnexpectedError
    }
    defer resp.Body.Close()

    switch sc := resp.StatusCode; sc {
        case http.StatusOK:
            return f.handle200(resp)
        case http.StatusTooManyRequests:
            return f.handle429(resp)
        default:
            return f.handleUnexpectedResponse(resp)
    }
}

/**** HELPERS ****/
func (f *CustomClient) handle200(resp *http.Response) (string, error) {
    body, err := io.ReadAll(resp.Body)

    // can do all sorts of manipulations on this response..
    // extract vowels.. 
    // convert this response to lowercase..
    // in our case, we will only convert it to uppercase..

    return upperCaseBody, nil
}

Our server returns a 200 status code and the response “This is Sparta!” Our handler will convert this to uppercase. Just to restate: this handler logic is what we are looking to test, which is why we need to mock the server response.

First, define a custom RoundTripper

type MockRoundTripper struct {
    t *testing.T
    responses []*http.Response
    reqIndex int
}

func NewMockRoundTripper(t *testing.T) *MockRoundTripper {
    return &MockRoundTripper{
        t: t,
    }
}

func (m *MockRoundTripper) StubResponse(statusCode int, header *http.Header, body string) {
    resp := &http.Response{
        StatusCode: statusCode,
        Proto: "HTTP/1.1",
        ProtoMajor: 1,
        ProtoMinor: 1,
        Header: *header,
        Body: io.NopCloser(strings.NewReader(body)),
    }
    m.responses = append(m.responses, resp)
}

func (m *MockRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
    m.t.Helper()
    require.Less(m.t, m.reqIndex, len(m.responses), fmt.Sprintf("error: number of requests %d, number of responses %d", m.reqIndex, len(m.responses)))
    resp = m.responses[m.reqIndex]
    m.reqIndex += 1
    return resp, nil
}

Explanation:

MockRoundTripper: Has 3 fields, t stores the testing state from Go’s test package; responses stores the mock responses that we want our server to return; reqIndex stores the current request index i.e. first response will be returned for the first request, second for the second and so on.
StubResponse(): Accepts statusCode, headers and the body for the response we want from our server. It then stuffs it into an http.Response object and adds it to our response array.
RoundTrip: Simply returns the response depending on the reqIndex and increases the reqIndex.

Finally, the test for our handler looks like

func TestHandle200(t *testing.T) {
    body := "This is Sparta!"
    mockRoundTripper := testutils.NewMockRoundTripper(t)
    f := NewFetcher(mockRoundTripper)
    mockRoundTripper.StubResponse(200, &http.Header{}, body)
    handledResp, err := f.MakeRequest("http://www.dummyurl.com") // URL doesn't matter 
    require.NoError(t, err)
    require.Equal(t, handledResp, strings.ToUpper(body))
}

Ba Dum Tss!

Putting the pieces together

Why did I wade into the muddy waters of Go’s standard library?

To show how an actual interface, the RoundTripper interface was nicely wrapped around and then used deep inside the wrapper to perform quite a simple task: execute an HTTP request.

We then defined a custom implementation for this interface and used it to perform a routine task: mock an HTTP server response. This shows the actual meaning of the phrase one interface, multiple implementations.

Where do you go from here? Try going through some more of Go’s interfaces such as io.Reader and io.Writer and how they can be used. Consider this scenario: how would you use another one of Go’s interfaces to test a CLI program that outputs to stdout

Resources

The MockRoundTripper is borrowed from Daniel Wagner Hall’s implementation here. It reads like poetry.