Ulrich Kautz Blog

Practical introduction to Go: Application development (3/3)

· 52 minute read

This introduction is written for developers with some experience in Go. I will assume:

If any of the above is not (yet) true, please check out the previous articles in this series:

After this course you will:

Overview

Since I believe in learning by doing this course will walk you through the development of an application. I will explain on the way how things are done - and why. The application we’re going to write will be an HTTP server, providing a REST API for a simple todo management application.

My approach might be coloured by my own opinions and experiences, but - as far as I know - it follows mostly best practices.

Either way, here we go, I hope you enjoy:

Planning

Before we dive into writing our Go application, let’s take a quick breather and consider what we want to achieve. As mentioned above, the goal is to write an HTTP REST server which provides an API to manage todos. So what do we need to make that happen?

Components

Ok, having this spelled out, we can alreadu derive some requirements for later implementation. Let’s group the requirement into logical components, as in “a thing that does a specific thing”:

With these components, we can also forsee some requirements for the data structures we will need to deal with. Usually these data structures are called models:

That covers the general ideas and concepts. Let’s step further into the details:

Todo model

To keep it simple, let’s agree a todo will have the following attributes:

User model

Also as simple as possible, at least we need:

API specification

We set out to create a REST API, so we need to define the endpoints. Before we do that, let’s agree to use JSON whenever data (transport) encoding is required.

That should suffice for a first iteration.

Structure

The first step is always the hardest. To get started with a new application it makes sense to give it a name. That name will be the module name.

Since version 1.11 Go comes with a built-in package manager and introduced Go modules as a concept. This not only allows you to easily manage dependencies (aka “using 3rd party modules”), but also suggests a framework where your Go code can live, so that it can be used by 3rd parties (or yourself - always keep in mind: d.r.y. don’t-repeat-yourself).

A module, in the Go sense, is a set of Go packages under the same namespace. Those packages need to live somewhere. That somewhere is often a public repository host like github.com, but can also be a company-internal web server or your local file system.

Initialize Go module

Let’s jump right into it. Change into your projects directory (=wherever you store your code on your local machine) and create a new directory for your application. I will call it todo-app. In that directory initialize your Go module:

$ cd ~/Projects
$ mkdir todo-app
$ cd todo-app
$ go mod init github.com/ukautz/go-intro/todo-app

The last line creates the Go module with the name github.com/ukautz/go-intro/todo-app. This name is not arbitrary, it provides a lot of useful information:

If you now list the directory contents, you fill find a new file named go.mod. That file should look like this:

module github.com/ukautz/go-intro/todo-app

go 1.14

That reflects exactly what we did:

About Go version compatibility

Go comes with a promise of compatibility. In short this promise is a commitment that all Go 1.x code will be able to run on any future Go 1.y version, if 1.y > 1.x. So far this holds true.

However, the reverse is not guaranteed! Every new Go release comes with at least some newly introduced functionality in the standard library. If your Go program uses these new functions, older Go versions won’t be able to compile the code anymore.

Directory structure

There is no official way to structure Go code, but there is the standard library, written by the authors of Go (Rob Pike, Robert Griesemer, Ken Thompson at Google), which is often used to derive good practice and standards from. However, this mostly pertains to structuring Go libraries and helps only somewhat with structuring Go applications.

If you search for “Go application structure” you will find the Go standard project layout, which collects common layout patterns for Go (application) projects.

The structure I am going to present to you here is mostly adhering to that standard project layout.

The pattern, in essence, separates “business logic” (=what your application does) from “command logic” (=how your application is being used) and you might have seen similar strategies in other languages (it’s really nothing new). To that end, start out by creating the following directories:

$ mkdir -p cmd/server pkg

This leaves your with the folder structure:

cmd/
  server/
pkg/

To phrase that in plain English:

If you are confused about the cmd/server, instead of just cmd: Consider we might adding another command line program, which uses the same library code in pkg. I am thinking here of a client application, which talks to the HTTP API of our server. It would then live in the directory cmd/client. In anticipation of that, because the standard project layout recommends it and because it feels wrong for me break that pattern, I went ahead and complicated things for you ;)

Main

Any program needs to start somewhere. The main package in Go is that starting point for Go programs. Open up cmd/server/main.go in your editor of choice and copy the following as a starting point of our soon-to-be HTTP REST API server.

package main

import (
	"fmt"
	"net/http"
)

func todo(w http.ResponseWriter, r *http.Request) {
	w.Write([]byte("todo"))
}

func main() {
	fmt.Println("Starting API server")
	if err := http.ListenAndServe(":12345", http.HandlerFunc(todo)); err != nil {
		panic(err)
	}
}

This is our main Go file. Go runs the main function in the main package, when you run it or execute a compiled binary:

$ go run cmd/server/main.go
Starting API server

In another terminal, you can now curl your running HTTP server and see the word todo returned:

$ curl http://localhost:12345
todo

Recap: Program anatomy

In case you have read the previous article in the series, feel free to skip this section.

package main

Every .go file is in a package and must have a package directive on top of the file. Usually package name match the directory name (so it would be server here). The package name main we used is a special case and required here, as this is our executable part of the code and Go mandates it. There are a few other rules you should keep in mind:

import (
	"fmt"
	"net/http"
)

These are package import statements. As we are going to use fmt.Println and various http.* things a few lines below, we first need to import both packages.

You might notice fmt and even net/http are somewhat shorter than our github.com/ukautz/go-intro/todo-app. This is because both are part of the standard library. While we could have also chosen a short module name (eg only todo-app), it is considered bad practice to chose “local names” (as in short, without url of location) for your own packages. Reason: (among others) those names are used by the Go package manager to download from. So, use location based modules names - or expect headaches in any deploy pipeline.

The last block:

func main() {
	fmt.Println("Starting API server")
	if err := http.ListenAndServe(":12345", http.HandlerFunc(todo)); err != nil {
		panic(err)
	}
}

Well, in short: that prints out the message Starting API server and then starts the HTTP server - and runs until you press ctrl + c.

Models

We already have generic specification of our model. Translating that into Go we will be using struct (“structure”), which you can think of something like class in other languages, tho not fully; Go has it’s own approach to things. You will see. For now just take with you: struct is especially useful to model anything that has properties or attributes. Like your Todo or User.

Todo Model

Create a new file named todo.go in the pkg folder, with the following contents:

package todo

import "time"

type Todo struct {
	ID          string
	Title       string
	Description string
	Created     time.Time
	UserID      string
}

Ignore all but the type Todo struct {..} for a moment. First: This implements the model from above in Go. Title, Description and Created should be self-explanatory. The user reference is implemented via the UserID field, which will point to an ID attribute of a following User model. I’ve added the ID here as well, so that we have some guaranteed unique value to utilize. This will be simplify reading and deleting a specific Todo via the REST API.

Why todo, not pkg?

The name of the folder the todo.go file is in is pkg. So, per convention, the name of the package should be pkg, not todo.

The reason is that pkg is actual the “root folder” of our library code. There is no library code “above” it. Now think of using multiple libraries that you import, that follow that directory structure and are named pkg… Although you can rename package namespaces during import within the scope of the import, it makes great sense to name it package todo to begin with.

I choose todo as the name as it describes the concern of the package best in my opinion.

Structs in Go

A bit more about structs in Go in general: You might be familiar with object oriented (OO) programming as a concept. While Go has it’s own flavor of how to implement OO, ultimately it provides the required features:

[objects] can contain data and code: data in the form of fields (often known as attributes or properties), and code, in the form of procedures (often known as methods)

Struct fields

In the code above you can see the first part: Objects contain data in the form of fields / attributes. We declared a type with the following fields:

A struct can define any amount of fields - or none, for that matter. It can also have “complex” attributes, which are themselves structs or slices or slices of structs or interfaces, or any variation thereof. For example:

type Nothing struct {}

type Something struct {
	Field1 string
	Field2 struct {
		SubField1 string
		SubField2 struct {
			SubSubField1 int
		}
	}
}

The Field2 attribute is itself a (anonymous) struct, that contains an scalar value (string) and yet another anonymous struct.

In practice, you would often define structs that have properties that are other structs and can contain them yet again. Along the lines of:

type Address struct {
	Name string
	Street string
	City string
	Zip int
	Country string
}

type BasketItem struct {
	Amount int
	ItemID int
}

type Order struct {
	Address   Address
	Items     []BasketItem
	Timestamp uint
}

Instantiating structs

To make it a bit clearer what you can do with our Todo type, let me give you a quick usage example:

t := Todo{
    ID:          "todo-0001",
	Title:       "go shopping",
	Description: "buy 3 pizzas, 6 beers and 1 avocado",
	Created:     time.Now(),
}
fmt.Printf("Todo: %s, created %s", t.Title, t.Created)

The above declares a new variable t (as an instance) of the type Todo and specifies values for the fields of the type, then accesses its fields (here: Created and Title) below.

Note: There is no (programmatic) need to fill in all the available fields. You could also write:

t := Todo{
	Description: "buy 3 pizzas, 6 beers and 1 avocado",
}

or even

t := Todo{}
// synonym to
var t Todo

As a consequence, all “not mentioned fields” are initialized with their empty value (what that is depends on the attribute-type). In the above case t.Title, as it is not specified, would contain the empty string "".

The struct fields are also write-accessible after creation:

t.Title = "do something"

User Model

While we are at it, let’s also define the above mentioned User model. For that create the file pkg/user.go with the following content:

package todo

type User struct {
	ID       string
	Name     string
	Password string
}

That is even simpler. Note that I diverted from our initial specification a bit:

Struct Methods

The second property of the OOP definition Objects have functionality / code, in the form of procedures / methods is fully supported with structs.

As an example, consider what I wrote above:

fmt.Printf("Todo: %s, created %s", t.Title, t.Created)

That would print out something like: Todo: <The-Title-Content>, created <Date>.

If there is a need to print out a Todo multiple times (e.g. think of logging), it’s a good idea to create a function that renders the string, so it can be re-used. Since Todo is already a struct, we can simply add a method to it:

func (t Todo) String() string {
    return fmt.Sprintf("%s, created %s", t.Title, t.Created)
}

This is close to a function definition, but it has an additional statement after func. The (t Todo) expression specifies the receiver. With t being the receiver instance and Todo the receiver type.

Now that the “rendering” is wrapped up in a method, we could write:

fmt.Printf("Todo: %s", t.String())

Actually, there is even a shorter way:

fmt.Printf("Todo: %s", t)

This works, because the fmt.Printf function is able to use the String() string method automatically. To understand why, you need to know how Go implements interfaces.

Interfaces in Go

Interfaces are a common concept in many high-level programming languages. In plain terms, an interface is a contract. To use a real-life example, consider the power button ⏻ you will find on many electronic devices. The contract that button implies is: Pressing it powers a stopped device on and a running device off. It toggles it’s running state. In the same way an interface in programming is a contract. Consider the following piece of Go code:

type OnOffSwitch interface {

	// TogglePower switches a running device off and a stopped device on
	TogglePower()
}

The above specifies an interface named OnOffSwitch. The contract states basically: Anything that has a method TogglePower() is of the interface OnOffSwitch. This means: In Go implementation of the methods of an interface, implements that interface.

A code example, implementing the above OnOffSwitch interface, would be:

type RedButton struct {}

func (r RedButton) TogglePower() {
	// do something
}

The RedButton struct now implements the OnOffSwitch interface, because it has the TogglePower() method.

In most other programming languages, that I know off, and that have interface concepts, do it the other way around. They have an explicit interface declaration, like:

class SomeClass implements OnOffSwitch {
    TogglePower() {
        // ..
    }
}

Go does not do that. It has implicit interface implementation. If you implement all the methods, the interfaces specifies then you are done. The interfaceis implemented.

And this is why log.Printf("something %s", todo) is able to use the String() string method automatically: it just checks if an provided argument is a string, or does have the String() string method. Knowing that Go uses implicit interface implementation, the question “does something have method X” and “does something implements an interface Y that specifies method X” are synonym.

To put that into code, here the Stringer interface from the fmt package of the standard library:

type Stringer interface {
	String() string
}

With the above interface: Everything that implements Stringer has the String() string method. Also in reverse: everything that has the String() string method implements the Stringer interface. If the interface would have more than one method, then things would need to implement all of the methods - partially doesn’t help.

Now back to log.Printf: The implementation of that function would then check: Does the argument I got implements fmt.Stringer? If so, then it has the String() string method, and that can be used.

This is a good segway to the next topic. To check “if something implements a method”, you need an “any type” - a type that can be multiple things, can be of multiple types. In Go, that is the empty interface interface{}:

The empty interface

As shown previously, implementing an interface in Go just requires to implement the methods specified in that interface. If additional methods are implemented, Go doesn’t care. It just needs to have also the methods of the interface.

Now, with that in mind: What if an interface does not specify any methods? An empty interface, so to speak. Well, from Go’s point of view that means: Anything automatically implements that interface, as it cannot be missing any of the methods specified in that interface, as there are none. Or in short: Anything implements the empty interface.

Why is this useful? Consider the log.Printf again. It allows you to pass parameters of any type (after the first, which must be a string). To use a simple example, consider a greeter function, that generates a hello message:

func Greet(anyone interface{}) string {
	switch value := anyone.(type) {
	case string:
		return "Hello " + value
	case fmt.Stringer:
		return "Hello " + value.String()
	default:
		return "Hello unsupported"
	}
}

The above uses a common Go pattern, in the context of the empty interface: The type switch. Basically: Do something different, based on and the type of an argument that can have any type.

The above Greet function would work with string type arguments - or anything that implements the fmt.Stringer interface, so has that String() string method:

type Person struct {
	Name string
}

func (p Person) String() string {
	return p.Name
}


// --%<--

func main() {
	fmt.Println(Greet("Me")) // prints out "Hello Me"
	fmt.Println(Greet(Person{Name: "Alice"})) // prints out "Hello Alice"
}

You will see interface{} being used quite often. To give you some examples:

// as function parameter
func something(v interface{}) {

	// as a variable
	var w interface{}

	// as a slice variable
	a := make([]interface{}, 0)

	// as a map
	m := make(map[string]interface{})
	n := make(map[interface{}]interface{})

	// ..
}

Components

That was a lot theory, back to the problem at hand. Now that we have the models, let’s get to the components.

First let’s look at the HTTP server. Go already comes with an excellent HTTP server in the standard library. We already used it in the above main.go. So what we need is an HTTP router, which decides based on the incoming requests which API action is to be executed. That router must then:

Ok, let’s start with a draft of the HTTP router and find out in implementation what it requires from the other components.

HTTP router draft

Previously interfaces where introduced. With that knowledge, let me show you the probably most important interface in Go HTTP handling, the Handler from the net/http package.

It’s quite short, have a look:

type Handler interface {
	ServeHTTP(ResponseWriter, *Request)
}

This means we just need to create a custom type, which has a ServeHTTP(ResponseWriter, *Request) method, and we have an HTTP router!

Let’s get started then. Create the file pkg/router.go and add:

package todo

import "net/http"

// Router handles HTTP request routing for the Todo REST API server
type Router struct {
}

// ServeHTTP implements the http.Handler interface
func (r Router) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
	// TODO: authenticate each request
	// TODO: handle create, get, list and delete
	rw.Write([]byte("from the router"))
}

As described in the // TODO comments, we’re not done yet, but let’s go ahead and integrate the Router in the main.go:

package main

import (
	"errors"
	"fmt"
	"net/http"

	todo "github.com/ukautz/go-intro/todo-app/pkg"
)

func main() {
	fmt.Println("Starting API server")
	router := todo.Router{}
	if err := http.ListenAndServe(":12345", router); err != nil {
		panic(err)
	}
}

The server should now return from the router on all requests:

$ curl http://localhost:12345
from the router

Authentication interface

To make sure only “valid users” can access the application, we will need something that makes sure that:

Not considering implementation, the following expresses that in Go:

package todo

import "net/http"

// Authentication permits HTTP requests for known users with valid credentials and rejects all other
type Authentication interface {

	// Authenticate returns ID of identified user creating the HTTP request
	Authenticate(req *http.Request) (userID string, err error)
}

Besides the error, which will be returned in case the access is not permitted, or any other internal error ocurred, the userID string will contain the unique identifier of the User that issued the request.

Integrate authentication

Before implementing any actual authentication, let’s first integrate it. Going back to pkg/router.go:

package todo

import (
	"log"
	"net/http"
)


// Router handles HTTP request routing for the Todo REST API server
type Router struct {

	// Authentication validates that requests are from permitted users
	Authentication
}

// ServeHTTP implements the http.Handler interface
func (r Router) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
	// end with an error for all not authenticated requests
	_, err := r.Authenticate(req)
	if err != nil {
		r.handleError(rw, req, err)
		return
	}

	rw.Write([]byte("you are authenticated"))
}

// handleError prints out errors in the logs and lets the request fail
func (r Router) handleError(rw http.ResponseWriter, req *http.Request, err error) {
	log.Printf("Error in %s %s: %s", req.Method, req.URL, err)
	rw.Header().Set("content-type", "application/json")
	if errors.Is(NotAllowedError, err) {
		rw.WriteHeader(http.StatusForbidden)
		rw.Write([]byte(`{"error":"forbidden"}`))
	} else {
		rw.WriteHeader(http.StatusInternalServerError)
		rw.Write([]byte(`{"error":"internal server error"}`))
	}

First take a look how Authentication ise being used in the type Router struct:

Sweet! Although we could not really run that, because we have no implementation yet, of the Authentication interface, the security concern is already addressed in the routing.

Before going into any implementation, let’s continue a bit further with the next component:

Persistence interface

We can already forsee what we will need from our Persistence component from our API specification: Create, List, Get, Delete on the Todos.

Again, let’s start with the interface before implementation. Create pkg/persistence.go with:

package todo

// Persistence is a storage for todos
type Persistence interface {

	// Create stores a new Todo and returns the ID
	Create(todo Todo) (string, error)

	// Delete removes a single Todo identified by it's ID. Returns os.ErrNotExist if not found
	Delete(id string) error

	// Get fetches a single Todo identified by it's ID. Returns os.ErrNotExist if not found
	Get(id string) (*Todo, error)

	// List returns all Todos
	List() ([]Todo, error)
}

This interface matches our requirements. I’ve extended the functionality a bit, to make it more convenient in our use-case.

If you were wondering why I chose to (re-)use os.ErrNotExist for our purposes: It’s convenient. We could have also defined our own “custom” error (or error type) for that, but I think this is semantically clear.

Integrate persistence in Router

With this, we can draft out the rest of the Router (back in pkg/router.go). We need to:

  1. build a decision tree to handle the incoming requests (=the routing)
  2. call the respective Persistence methods to implement the REST endpoints

First to the decision tree. Modify the ServeHTTP method like so:

// ServeHTTP implements the http.Handler interface
func (r Router) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
	// end with an error for all not authenticated requests
	userId, err := r.Authentication.Authenticate(req)
	if err != nil {
		r.handleError(rw, req, err)
		return
	}

	// handle 
	// - POST and GET for /todo 
	// - DELETE and GET for a path looking like /todo/<id>
	path := req.URL.Path
	if path == "/todo" {
		switch req.Method {
		case http.MethodPost:
			r.create(rw, req, userId)
			return
		case http.MethodGet:
			r.list(rw, req)
			return
		}
	} else if strings.HasPrefix(path, "/todo/") {
		id := path[len("/todo/"):]
		switch req.Method {
		case http.MethodDelete:
			r.delete(rw, req, id)
			return
		case http.MethodGet:
			r.get(rw, req, id)
			return
		}
	}

	// anything else, we don't now
	rw.WriteHeader(http.StatusNotFound)
	rw.Write([]byte("not found"))
}

This show-cases a minimal HTTP router in Go implementing our specific requirements. With larger projects you would probably start using a 3rd party library with a more convenient path routing setup. For our use-case this is not required.

Some things of note:

Now to the above mentioned methods, which will implement the calls to the Persistence interface which was defined earlier. Add below the ServeHTTP method:


// --%<--

// Router handles HTTP request routing for the Todo REST API server
type Router struct {

	// Authentication is used to filter out requests that are not from valid users
	Authentication Authentication

	// Persistence is used to access Todos
	Persistence Persistence
}

// --%<--

func (r Router) create(rw http.ResponseWriter, req *http.Request, userId string) {

	// read Todo from JSON body of HTTP request
	var todo Todo
	decoder := json.NewDecoder(req.Body)
	if err := decoder.Decode(&todo); err != nil {
		r.handleError(rw, req, err)
		return
	}

	// create Todo in Persistence
	todo.UserID = userId
	todoID, err := r.Persistence.Create(todo)
	if err != nil {
		r.handleError(rw, req, err)
		return
	}
	r.json(rw, req, map[string]string{"id": todoID})
}

func (r Router) list(rw http.ResponseWriter, req *http.Request) {
	todos, err := r.Persistence.List()
	if err != nil {
		r.handleError(rw, req, err)
		return
	}
	r.json(rw, req, todos)
}

func (r Router) delete(rw http.ResponseWriter, req *http.Request, todoID string) {
	err := r.Persistence.Delete(todoID)
	if err != nil {
		r.handleError(rw, req, err)
		return
	}
	r.json(rw, req, map[string]string{"id": todoID})
}

func (r Router) get(rw http.ResponseWriter, req *http.Request, todoID string) {
	todo, err := r.Persistence.Read(todoID)
	if err != nil {
		r.handleError(rw, req, err)
		return
	}
	r.json(rw, req, todo)
}

// json prints out a JSON HTTP response
func (r Router) json(rw http.ResponseWriter, req *http.Request, data interface{}) {
	rw.Header().Set("content-type", "application/json")
	err := json.NewEncoder(rw).Encode(data)
	if err != nil {
		r.handleError(rw, req, err)
		return
	}
}

Since we agreed to use JSON for transport encoding, the last method json() prints arbitrary (JSON-transformable) data as an HTTP response with a JSON body.

In the create method the decoding (unmarshalling) of a Todo instance from the JSON body of the incoming HTTP request is show-cased. Then it is created via Persistence.

The other methods do not provide any surprises: They call the respective Persistence method and handle the error with the previously implemented handleError method. If needed they use the json() method to print out JSON formatted responses.

Authentication implementation

Ok, now that we go the basic Router logic in place we need to provide an actual implementation for the Authentication interface. For simplicity sake, let’s do HTTP basic authentication.

So what our implementation must do is:

So two things need to be decided:

Still in the spirit of simplicity let’s use a JSON file containing the user credentials.

Structs to JSON

Encoding or decoding Go types, especially structs like our User or Todo , from and into transport encoding like JSON is quite simple. To that end, the standard library provides encoding/json.

Assume you have an instance of User like so:

u := User{
	ID:       "u01",
	Name:     "alice",
	Password: "secret",
}

To encode that into JSON, using encoding/json:

encoded, err := json.Marshal(u)
if err != nil {
	// ..
}
fmt.Println(string(encoded)) // encoded is []byte

This would yield the following output:

{"ID":"u01","Name":"alice","Password":"secret"}

Note: Only public (UpperCase) attributes can be accessed by the encoder. Private (lowerCase) attributes are not part of any resulting encoding.

Commonly, you write JSON attributes lowercase. This is easily done in Go. You just need to tell the JSON encoder, what the field names should be. Here is how that looks like, using the User type in the pkg/user.go file:

type User struct {
	ID       string `json:"id"`
	Name     string `json:"name"`
	Password string `json:"pass"`
}

Each of the attributes has now an added json:"<something>" in backticks `. These expressions are called struct tags and can be used in in reflection, which allows you to inspect a variable or type at runtime.

In this case, when we encode an instance of User into JSON, the used encoder will be able to get the information from the struct tag and will use it as the JSON attribute/key name, instead of the attribute name (ID -> id). To make that plain:

With the above struct tags, that encoded to JSON would yield:

{"id":"u01","name":"alice","pass":"secret"}

The reverse, decoding (aka unmarshalling) from JSON into Go, for example into the User struct, looks like this:

var user User
encoded := []byte(`{"id":"u01","name":"alice","pass":"secret"}`)
err := json.Unmarshal(encoded, &user)
if err != nil {
	// ..
}
fmt.Println("ID", user.ID) // "u01"

Load JSON file

Now to the JSON file from which the valid users are supposed to be loaded from. Create a data folder in the root of the todo application directory and then create a file named users.json within. Fill it with the following:

[
  {"id":"u01", "name":"alice", "pass":"secret1"},
  {"id":"u01", "name":"bob", "pass":"secret2"}
]

An easy implementation of the Authentication interface which uses that JSON file can be found when looking it as two separate problems:

Let’s start with the Authentication by appending to the existing file in pkg/authentication.go.

// NotAllowedError is returns when access is not permitted
var NotAllowedError = errors.New("access not permitted")

// UsersAuthentication checks credentials against a list of users
type UsersAuthentication []User

// Authenticate extracts HTTP basic auth user credentials and returns whether a user
// in the list has a matching username and password
func (a UsersAuthentication) Authenticate(req *http.Request) (string, error) {
	name, pass, ok := req.BasicAuth()
	if !ok {
		return "", fmt.Errorf("missing credentials: %w", NotAllowedError)
	}
	for _, user := range a {
		// found a user!
		if user.Name == name && user.Password == pass {
			return user.ID, nil
		}
	}
	return "", NotAllowedError
}

The UsersAuthentication type extends the []User (slice) type. The defined Authenticate method then implements the Authentication interface. That is pretty neat, as we now are just left with loading the JSON file into a []User-type variable and then can use that directly.

Note: I added NotAllowedError so we have a specific error to differentiate, for example, from file read problems.

// LoadAuthenticationFromJSON reads a JSON file, returns an Authentication implementation
func LoadAuthenticationFromJSON(filename string) (Authentication, error) {

	// define a slice of users & fill it from a JSON file
	var users []User
	encoded, err := ioutil.ReadFile(filename)
	if err != nil {
		return nil, err
	} else if err = json.Unmarshal(encoded, &users); err != nil {
		return nil, err
	}

	// cast the slice of users into an Authentication implementation
	return UsersAuthentication(users), nil
}

That gives us a good-enough Authentication component for the moment. Since we used an interface, we can later write something using LDAP, AWS DynamoDB or whatever makes sense and just switch it out.

Persistence implementation

Continuing with JSON as the encoding for our structs, let us first add some struct tags to the Todo struct type. Open pkg/todo.go again and modify:

type Todo struct {
	ID          string    `json:"id"`
	Title       string    `json:"title"`
	Description string    `json:"description"`
	Created     time.Time `json:"created"`
	UserID      string    `json:"user_id"`
}

Basically CamelCase to snake_case. With that decided, let’s agree on the file system as the storage, in the manner:

<base-directory>/<id>.json

So bascially: All todos will live in the same directory in a file named after their ID plus the .json suffix.

Considering the Persistence interface, our directory-based implementation will do roughly

Use 3rd party libraries

Our Todo struct contains an ID attribute, which should contain a unique identifier. An UUID is a good choice. There are many Go implementations, I suggest github.com/google/uuid, which is well tested in production.

While Go downloads dependencies whenever you run it (or build, or test, ..), I prefer explicit download of dependencies from the command line (so I see the dependencies it might have):

$ go get -v github.com/google/uuid
go: github.com/google/uuid upgrade => v1.1.1
github.com/google/uuid

Note: This will extend both go.mod and go.sum

$ cat go.mod 
module github.com/ukautz/go-intro/todo-app

go 1.14

require github.com/google/uuid v1.1.1 // indirect


$ cat go.sum 
github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY=
github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+y

JSON file Persistence

Open the file pkg/persistence.go again and add:


// --%<--
// in imports add:
	"github.com/google/uuid"
// --%<--

// DirectoryPersistence implements Persistence with a local file system directory
type DirectoryPersistence string

// Create stores Todo in <directory>/<id>.json file
func (p DirectoryPersistence) Create(todo Todo) (string, error) {
	if todo.ID == "" {
		todo.ID = uuid.New().String()
	}
	encoded, err := json.Marshal(todo)
	if err != nil {
		return "", err
	}

	path := p.path(todo.ID)
	err = ioutil.WriteFile(path, encoded, 0640)
	if err != nil {
		return "", err
	}

	return todo.ID, nil
}

// Delete removes <directory>/<id>.json file
func (p DirectoryPersistence) Delete(id string) error {
	return os.Remove(p.path(id))
}

// Get reads Todo from <directory>/<id>.json file
func (p DirectoryPersistence) Get(id string) (*Todo, error) {
	return p.read(p.path(id))
}

// List reads all Todos from <id>.json files in <directory>
func (p DirectoryPersistence) List() ([]Todo, error) {
	todos := make([]Todo, 0)
	err := filepath.Walk(string(p), func(path string, info os.FileInfo, err error) error {
		if err != nil {
			return err
		} else if info.IsDir() {
			return nil
		} else if filepath.Ext(path) != ".json" {
			return nil
		}

		todo, err := p.read(path)
		if err != nil {
			return err
		}

		todos = append(todos, *todo)
		return nil
	})

	return todos, err
}

func (p DirectoryPersistence) path(id string) string {
	return filepath.Join(string(p), id+".json")
}

func (p DirectoryPersistence) read(path string) (*Todo, error) {
	var fh 
	encoded, err := ioutil.ReadFile(path)
	if err != nil {
		return nil, err
	}

	var todo Todo
	if err = json.Unmarshal(encoded, &todo); err != nil {
		return nil, err
	}

	return &todo, nil
}

That should be largely self-explanatory. The following statement might be a bit surprising:

type DirectoryPersistence string

Reason: The type could be structured instead like type DirectoryPersistence struct { Directory string }, but that would not gain much, but add complexity.

The respective public methods (with uppercase first letter) implement the the Persistence interface as intended.

To not repeat code, the following helper methods are implemented:

Bring it all together

Now that we have both working Authentication and Persistence implementations, all can be setup together in the main method. Since this is about practical application development, let me introduce the urfarve/cli library to you. It’s a very commonly used CLI framework.

$ go get -v github.com/urfave/cli/v2
go: github.com/urfave/cli/v2 upgrade => v2.2.0

This framework has a nice interface to rapidly develop simple or complex command line applications. I even use it for very small tools, just because it supports very readable code structures. I recommend to look into the extensive documentation, with many examples.

package main

import (
	"log"
	"net/http"
	"os"
	"path/filepath"

	todo "github.com/ukautz/go-intro/todo-app/pkg"
	"github.com/urfave/cli/v2"
)

func main() {
	app := cli.NewApp()
	app.Name = "server"
	app.Usage = "HTTP API for todos"

	app.Flags = []cli.Flag{
		&cli.StringFlag{
			Name:    "storage-directory",
			Aliases: []string{"d"},
			Usage:   "Path to directory to store todos",
			Value:   filepath.Join("data", "store"),
		},
		&cli.StringFlag{
			Name:    "users",
			Aliases: []string{"u"},
			Usage:   "Path to JSON file containing user credentials",
			Value:   filepath.Join("data", "users.json"),
		},
		&cli.StringFlag{
			Name:    "address",
			Aliases: []string{"a"},
			Usage:   "Listen address for the HTTP server",
			Value:   "127.0.0.1:12345",
		},
		&cli.StringFlag{
			Name:    "path-prefix",
			Aliases: []string{"p"},
			Usage:   "Prefix for ",
			Value:   "/v1",
		},
	}

	app.Action = func(c *cli.Context) error {
		listenAddr := c.String("address")
		routePrefix := c.String("path-prefix")

		// init storage
		store := todo.DirectoryPersistence(c.String("storage-directory"))

		// load users for authentication
		usersFile := c.String("users")
		auth, err := todo.LoadAuthenticationFromJSON(usersFile)
		if err != nil {
			return err
		}

		// setup router
		router := todo.Router{
			Prefix:         routePrefix,
			Authentication: auth,
			Persistence:    store,
		}

		// run server
		log.Printf("Starting API server at http://%s%s, storage directory: %s",
			listenAddr, routePrefix, store)
		return http.ListenAndServe(listenAddr, router)
	}

	err := app.Run(os.Args)
	if err != nil {
		panic(err)
	}
}

Ok, before going through all of that, best run the application once with the -h help flag:

$ go run cmd/server/main.go -h
NAME:
   server - HTTP API for todos

USAGE:
   main [global options] command [command options] [arguments...]

COMMANDS:
   help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --storage-directory value, -d value  Path to directory to store todos (default: "data/store")
   --users value, -u value              Path to JSON file containing user credentials (default: "data/users.json")
   --address value, -a value            Listen address for the HTTP server (default: "127.0.0.1:12345")
   --path-prefix value, -p value        Prefix for  (default: "/v1")
   --help, -h                           show help (default: false)

As you can see, this is a quite nice interface already. Have a look at the rendered out options, their default values, aliases etc and how that matches their implementation.

To run the application, first make sure:

Then run:

go run cmd/server/main.go 
2020/05/20 17:39:42 Starting API server at http://127.0.0.1:12345/v1, storage directory: data/store

In another terminal, you can now use the API with curl:

# create a new todo
$ curl -s -u "alice:secret1" -X POST http://127.0.01:12345/v1/todo \
	-d "{\"title\":\"my first todo\", \"description\":\"api must be played with\"}"

# list all todos
$ curl -s -u "alice:secret1" http://127.0.01:12345/v1/todo

# get a specific todo
$ curl -s -u "alice:secret1" http://127.0.01:12345/v1/todo/SOME_ID

# delete a specific todo
$ curl -s -u "alice:secret1" -X DELETE http://127.0.01:12345/v1/todo/SOME_ID

Testing

While the application is “feature complete”, according to the requirements, the development is not done. Any software, which needs to be maintained (which is any software you don’t throw out immediately), should have a test-suite.

Go provides an easy to use tooling and language support and there is a large number of 3rd party libraries to satisfy any taste of writing tests.

Writing tests in Go

Before going into the specific tests for this application, let me give you first an introduction on how to write tests in Go in a more general sense.

It’s not hard, the Go tooling comes already with all you need, including conventions on how to structure files and name tests.

Test files should always live in the same directory as the code they are testing. The convention is to create a new file named the same as the file containing the actual code, just with a suffix _test. For example, if you wish to write tests for code in a file named app.go, you would do so in the file app_test.go.

Let’s look at some code to make that clear. Assume the following code, which is to be tested:

// file: app.go
package myapp

func Multiply(a, b int) int {
	return a * b
}

In the same directory:

// file: app_test.go
package myapp_test

import "testing"

func TestMultiply(t *testing.T) {
	// here come the tests
}

We will fill in some actual test code in a moment, let’s quickly go through the test file:

Now, fill that test with something senseful:

// file: app_test.go
package myapp_test

import (
	"testing"

	"github.com/acme/myapp"
)

func TestMultiply(t *testing.T) {
	result := myapp.Multiply(5, 10)
	if result != 50 {
		t.Errorf("expected 50, got %d", result)
	}
}

To run this test we can use the go test command. I am adding the -v parameter for verbosity and the path to the test file:

$ go test -v app_test.go
=== RUN   TestMultiply
--- PASS: TestMultiply (0.00s)
PASS
ok  	command-line-arguments	0.002s

This looks good, the test is green!

Note: If you would omit the path to the test file (app_test.go) and just write go test -v, then all tests in all _test.go files in the current directory would be executed.

Note: If you want to run all the tests in the current directory and all sub-directories do: go test -v ./... with ./... meaning: traverse this and all sub-directories recursively

So you have seen it, let’s make the test fail once by changing the condition:

func TestMultiply(t *testing.T) {
	result := todo.Multiply(5, 10)
	if result != 40 {
		t.Errorf("expected 40, got %d", result)
	}
}

The output would look like this:

$ go test -v app_test.go
=== RUN   TestMultiply
    TestMultiply: app_test.go:13: expected 40, got 50
--- FAIL: TestMultiply (0.00s)
FAIL
FAIL	command-line-arguments	0.002s
FAIL

Pattern: List tests

While this is only an artificial example, there is already something to optimize. The above test only checks for a single, arbitrary chosen multiplication. In “real life”, you want to test more thoroughly. You could either create multiple test functions (TestMultiply1, TestMultiply2, …) or do a list test like so:

func TestMultiply(t *testing.T) {
	expects := []struct {
		a, b   int
		result int
	}{
		{1, 1, 1},
		{1, 2, 2},
		{2, 2, 4},
		{3, 3, 9},
		{3, 4, 12},
		{333, 777, 258741},
	}

	for _, expect := range expects {
		t.Run(fmt.Sprintf("expect %d * %d = %d", expect.a, expect.b, expect.result), func(t *testing.T) {
			result := todo.Multiply(expect.a, expect.b)
			if result != expect.result {
				t.Errorf("expected %d, got %d", expect.result, result)
			}
		})
	}
}

This is a very common pattern you will find in many Go test files. The construct setting up the tests expectations might be a bit unfamiliar, so let me explain:

expects := []struct {
	a, b   int
	result int
}{
	{1, 1, 1},
	// ...
}

This creates a list of anonymous structs, which have three int attributes (a, b and result) and initialize them with a set of values. The first line {1, 1, 1} sets a, b and result to 1.

This pattern is quite helpful, both for brevity and readability. When executing the test the t.Run(..) command will print out the sub-tests it ran:

$ go test -v app_test.go
=== RUN   TestMultiply
=== RUN   TestMultiply/expect_1_*_1_=_1
=== RUN   TestMultiply/expect_1_*_2_=_2
=== RUN   TestMultiply/expect_2_*_2_=_4
=== RUN   TestMultiply/expect_3_*_3_=_9
=== RUN   TestMultiply/expect_3_*_4_=_12
=== RUN   TestMultiply/expect_333_*_777_=_258741
--- PASS: TestMultiply (0.00s)
    --- PASS: TestMultiply/expect_1_*_1_=_1 (0.00s)
    --- PASS: TestMultiply/expect_1_*_2_=_2 (0.00s)
    --- PASS: TestMultiply/expect_2_*_2_=_4 (0.00s)
    --- PASS: TestMultiply/expect_3_*_3_=_9 (0.00s)
    --- PASS: TestMultiply/expect_3_*_4_=_12 (0.00s)
    --- PASS: TestMultiply/expect_333_*_777_=_258741 (0.00s)
PASS
ok  	command-line-arguments	0.002s

3rd party test libraries

While the standard library testing package comes with all you need to write tests, there are various 3rd party libraries making things easier and providing different flavours.

The most important ones, in my experience, are:

I am a big fan of testify, due to it’s neat and readable API and will use it in the tests for this application. As any external dependency I prefer to install it before using it:

$ go get -v github.com/stretchr/testify    
go: github.com/stretchr/testify upgrade => v1.6.1

Rewriting the above TestMultiply using testify would look like:

package todo_test

import (
	"testing"

	"github.com/stretchr/testify/assert"
	"github.com/acme/myapp"
)

func TestMultiply(t *testing.T) {
	assert.Equal(t, 50, myapp.Multiply(5, 10))
}

I think this is much more expressive than the standard Go testing tooling. Changing that 50 to the (wrong) expectation 40 again, the testify library also provides a much better readable failure reporting:

=== RUN   TestMultiply
    app_test.go:11: 
        	Error Trace:	app_test.go:11
        	Error:      	Not equal: 
        	            	expected: 40
        	            	actual  : 50
        	Test:       	TestMultiply
--- FAIL: TestMultiply (0.00s)
FAIL
FAIL	command-line-arguments	0.002s
FAIL

All in all, I prefer testify much over the standard tooling, hence I am going to use it in the actual test of this application. Check out the available assertions in the documentation.

Test: Authentication

To test our authentication we need to test the method Authenticate(*http.Request) (string, error) of the UsersAuthentication implementation.

To describe that intention in a Go test function name you will find usually the naming pattern Test<Type>_<Method> in testing code, as in:

func TestUsersAuthentication_Authenticate(t *testing T) {
	// todo
}

Create pkg/authorization_test.go with the following contents:

package todo_test

import (
	"net/http"
	"net/http/httptest"
	"testing"

	"github.com/stretchr/testify/assert"
	todo "github.com/ukautz/go-intro/todo-app/pkg"
)

func TestUsersAuthentication_Authenticate(t *testing.T) {
	auth := todo.UsersAuthentication{
		{ID: "u01", Name: "alice", Password: "secret1"},
		{ID: "u02", Name: "bob", Password: "secret2"},
	}

	expects := []struct {
		name    string
		request *http.Request
		id      string
		allowed bool
	}{
		{"missing basic auth forbidden", createBasicAuthTestRequest("", ""), "", false},
		{"unknown credentials forbidden", createBasicAuthTestRequest("foo", "bar"), "", false},
		{"invalid credentials forbidden", createBasicAuthTestRequest("alice", "invalid"), "", false},
		{"allow valid user u01", createBasicAuthTestRequest("alice", "secret1"), "u01", true},
		{"allow valid user u02", createBasicAuthTestRequest("bob", "secret2"), "u02", true},
	}

	for _, expect := range expects {
		//expect := expect
		t.Run(expect.name, func(t *testing.T) {
			//t.Parallel()
			userID, err := auth.Authenticate(expect.request)
			if expect.allowed {
				assert.NoError(t, err)
				assert.Equal(t, expect.id, userID)
			} else {
				assert.Error(t, err)
			}
		})
	}
}

func createBasicAuthTestRequest(user, pass string) *http.Request {
	req := httptest.NewRequest(http.MethodGet, "http://localhost:12345/bla", nil)
	if user != "" {
		req.SetBasicAuth(user, pass)
	}
	return req
}

Let’s run that test before I explain how it works:

$ go test -v pkg/authorization_test.go
?   	github.com/ukautz/go-intro/todo-app/cmd/server	[no test files]
=== RUN   TestUsersAuthentication_Authenticate
=== RUN   TestUsersAuthentication_Authenticate/missing_basic_auth_forbidden
=== RUN   TestUsersAuthentication_Authenticate/unknown_credentials_forbidden
=== RUN   TestUsersAuthentication_Authenticate/invalid_credentials_forbidden
=== RUN   TestUsersAuthentication_Authenticate/allow_valid_user_u01
=== RUN   TestUsersAuthentication_Authenticate/allow_valid_user_u02
--- PASS: TestUsersAuthentication_Authenticate (0.00s)
    --- PASS: TestUsersAuthentication_Authenticate/missing_basic_auth_forbidden (0.00s)
    --- PASS: TestUsersAuthentication_Authenticate/unknown_credentials_forbidden (0.00s)
    --- PASS: TestUsersAuthentication_Authenticate/invalid_credentials_forbidden (0.00s)
    --- PASS: TestUsersAuthentication_Authenticate/allow_valid_user_u01 (0.00s)
    --- PASS: TestUsersAuthentication_Authenticate/allow_valid_user_u02 (0.00s)
PASS
ok  	github.com/ukautz/go-intro/todo-app/pkg	0.002s

Excellent, the test passes and the test output is rather expressive, too! Let’s go through it then, starting with the createAuthTestRequest helper function:

func createAuthTestRequest(user, pass string) *http.Request {
	req := httptest.NewRequest(http.MethodGet, "http://localhost:12345/bla", nil)
	if user != "" {
		req.SetBasicAuth(user, pass)
	}
	return req
}

As we want to check whether requests with valid credentials are passed through and requests with missing or invalid are not, a helper to create those requests with or without credentials is useful. The URL and the method type are of no consequence, so they are set to dummy values. The username and password in the request are set as basic auth credentials, as we need for the tests.

As you can see I am making use of net/http/httptest, which provides a lot of helpful tooling around testing HTTP services and functionality. I especially recommend to have a look at the test HTTP server, it might come in handy in your future endeavors.

Now, in the TestUsersAuthentication_Authenticate function: First an instance of the todo.UsersAuthentication type with two valid users is created. All the checks will use that instance:

auth := todo.UsersAuthentication{
	{ID: "u01", Name: "alice", Password: "secret1"},
	{ID: "u02", Name: "bob", Password: "secret2"},
}

Next, all test expectations are being declared. Since we are going to run “sub-tests”, each of the expectations has:

expects := []struct {
	name    string
	request *http.Request
	id      string
	allowed bool
}{
	{"missing basic auth forbidden", createAuthTestRequest("", ""), "", false},
	{"unknown credentials forbidden", createAuthTestRequest("foo", "bar"), "", false},
	{"invalid credentials forbidden", createAuthTestRequest("alice", "invalid"), "", false},
	{"allow valid user u01", createAuthTestRequest("alice", "secret1"), "u01", true},
	{"allow valid user u02", createAuthTestRequest("bob", "secret2"), "u02", true},
}

Lastly the actual test execution, which iterates the expectations and tests them sequentially by calling the auth.Authenticate method:

for _, expect := range expects {
	t.Run(expect.name, func(t *testing.T) {
		userID, err := auth.Authenticate(expect.request)
		if expect.allowed {
			assert.NoError(t, err)
			assert.Equal(t, expect.id, userID)
		} else {
			assert.Error(t, err)
		}
	})
}

Taking a deeper look at the body of the encapsulated Run method call:

userID, err := auth.Authenticate(expect.request)
if expect.allowed {
	assert.NoError(t, err)
	assert.Equal(t, expect.id, userID)
} else {
	assert.Error(t, err)
}

This first line executes the Authenticate method with the HTTP request from the expectation. Depending on whether we expect it to pass (expect.allowed), we either do not want it to error (assert.NoError) and return the ID (which must metch the expectation) - or make sure that it returns an error.

Test: Persistence

Now to testing the Persistence implementation: DirectoryPersistence. As we want to test the whole public interface, we need to write tests for all four public methods: Create, Delete, Get and List.

In the interest of brevity I will guide you here only through the tests for Create and Get. To see the full code have a look at https://github.com/ukautz/go-intro/blob/master/todo-app/pkg/persistence_test.go

Ok, let’s dive in:

Test Todo creation

func TestDirectoryPersistence_Create(t *testing.T) {
	p := createTestDirectoryPersistence(t)
	id, err := p.Create(todo.Todo{
		Title:       "the-title",
		Description: "the-description",
		UserID:      "u01",
	})
	require.NoError(t, err)
	require.NotEmpty(t, id)

	testFile := filepath.Join(testPersistenceDir, fmt.Sprintf("%s.json", id))
	defer os.Remove(testFile)

	raw, err := ioutil.ReadFile(testFile)
	require.NoError(t, err)

	var td todo.Todo
	require.NoError(t, json.Unmarshal(raw, &td))

	assert.Equal(t, id, td.ID)
	assert.Equal(t, "u01", td.UserID)
	assert.Equal(t, "the-title", td.Title)
	assert.Equal(t, "the-description", td.Description)
}

The first step is to create a new instance of DirectoryPersistence. Since we need to do that in all the tests (for all the public methods), I created the createTestDirectoryPersistence helper function, which makes sure a test (fixture) directory exists and then returns an instance using that test directory (have a look at code for the implementation).

p := createTestDirectoryPersistence(t)

Now we need to call the Create method, with a “dummy Todo”, and make sure the creation worked:

id, err := p.Create(todo.Todo{
	Title:       "the-title",
	Description: "the-description",
	UserID:      "u01",
})
require.NoError(t, err)
require.NotEmpty(t, id)

The two expressions with the require. prefix call functions of a (sub) package of the testify library. The only and main difference between the assert.* and the require.* functions is that require.* calls fatal (meaning: end the test with an error immediately), if they fail, while assert.* just print out the error and continue the test.

Since the following test code would not be able to execute / would not make sense, if the create already fails using require.* is opportune.

The following two lines give us the path to the JSON file, which we expect will be created by the DirectoryPersistence implementation, and make sure it’s removed after the test execution ends.

testFile := filepath.Join(testPersistenceDir, fmt.Sprintf("%s.json", id))
defer os.Remove(testFile)

Note especially the defer expression, which makes sure os.Remove() on that file is being called after the function ends - however this function ends: Either if it runs to the end (with a success) or ends anywhere in the middle (with a failure), the os.Remove will be executed thereafter. This way, we make sure the state before the test (no exiting JSON file) is guaranteed after the test.

To the next block, in which we’re going to read the contents of the file (we assume has been created) and decode (unmarshal) these contents into a new Todo instance:

raw, err := ioutil.ReadFile(testFile)
require.NoError(t, err)

var td todo.Todo
require.NoError(t, json.Unmarshal(raw, &td))

Again, require. is used instead of assert. as we need not continue with the test code, if anything on the way is already failing.

Finally to the the last assertions, where we check whether the JSON written by the Create method contains all the same, provided data:

assert.Equal(t, id, td.ID)
assert.Equal(t, "u01", td.UserID)
assert.Equal(t, "the-title", td.Title)
assert.Equal(t, "the-description", td.Description)

And that’s it. Now we can be reasonably sure that Create of the DirectoryPersistence:

Test Todo reading

func TestDirectoryPersistence_Get(t *testing.T) {
	defer os.Remove(assertJSONTodoFile(t, 2))

	p := createTestDirectoryPersistence(t)
	td, err := p.Get("todo-02")
	require.NoError(t, err)
	assert.Equal(t, todo.Todo{
		ID:          "todo-02",
		Title:       "todo 02",
		Description: "the todo number 02",
		UserID:      "u02",
		Created:     time.Date(2010, 11, 12, 13, 14, 15, 0, time.UTC),
	}, *td)
}

This is way shorter than the previous test, but contains a condensed statements at the start, which needs explanation:

defer os.Remove(assertJSONTodoFile(t, 2))

First, quickly about the custom assertJSONTodoFile helper function: This creates a JSON file in the test directory containing an encoded Todo, so it can be assumed in the tests. The path to that file is then returned. In this case, the 2 indicates to create the test file test-02.json. See the source code for more details.

Now here is what the whole defer-expression does:

This might be a bit confusing, but since it is a common trick in Go it is worth to take the time to understand it properly.

To make it plain, consider the following small program (which you can run here in the playground) first to show how defer itself works:

package main

import "fmt"

func main() {
	defer fmt.Println("deferred hello")
	fmt.Println("hello")
}

If you execute the above, you will see the following:

hello
deferred hello

I hope this makes it clear that defer executes the statement when the code block, it is in (in this case: the main function), ends.

Now, let’s extend that example (also in the playground):

package main

import "fmt"

func greet() string {
	fmt.Println("returning greetings")
	return "greetings"
}

func main() {
	defer fmt.Println(greet())
	fmt.Println("hello")
}

This will print out:

returning greetings
hello
greetings

Here is how the execution works:

This “trick” can be often found, especially in the context of (cleanup in) testing.

Now to the rest of test body:

p := createTestDirectoryPersistence(t)
td, err := p.Get("todo-02")
require.NoError(t, err)
assert.Equal(t, todo.Todo{
	ID:          "todo-02",
	Title:       "todo 02",
	Description: "the todo number 02",
	UserID:      "u02",
	Created:     time.Date(2010, 11, 12, 13, 14, 15, 0, time.UTC),
}, *td)

As before, we create a new instance using createTestDirectoryPersistence and then call the Get method using the ID todo-02. A JSON file, representing that Todo, was created earlier with the assertJSONTodoFile(t, 2) call.

Again, using require to make sure the test ends if any error was returned from Get and lastly a comparison of the returned *Todo with our expectations.

That’s it. Have a look into the other tests, which will use similar patterns to the ones explained here.

Test: Router

At last, we need to test our HTTP router. To that end, we need to test whether

Test Fakes / Stubs / Mocks

As we are writing unit(‘ish) tests, we are constraint to test the code in isolation. Since the Router requires both Persistence and Authentication, we need something to fulfill that need.

Luckily, we defined Persistence and Authentication as interfaces, which allows us now to use “test doubles”, which implement those interfaces, but are not the “actual implementations” (from above) and thereby not violate our isolation constraint (much).

If you are familiar with software testing patterns in other languages you might have heard of mocking or stubbing or test dummies or test fakes. There are a few definitions about in the interwebs. I decided to go with the one from Martin Fowler, specifcially what he calls a (test) Fake.

Note: If you wish to use mocks in Go, consider the mock package from the testify library.

Let me show you a quick example. For that we first need an interface for which we then write a test fake. Consider the following interface:

type Filesystem interface {

	// Write stores a file under the given path with the given content
	Write(path string, data []byte) error

	// Delete removes a file from the file system with the given path
	Delete(path string) error
}

Assume we have an actual implementation (which would actually remove or write a file on the file system), here is how a fake implementing that interface could look like:

type FakeFilesystem map[string][]byte

func (f FakeFilesystem) Write(path string, data []byte) error {
	f[path] = data
	return nil
}

func (f FakeFilesystem) Delete(path string) error {
	delete(f, path)
	return nil
}

As you can see, the FakeFilesystem simulates an actual file system. Further: it contains state, by storing the written “files” in a map - and by having the deleted “files” removed from that map. Why is that helpful? Easy: In a test, which uses that fake, where you want to test if a file was created, you can just check the map after the test run, whether it now contains a key with that file path and whether it contains the (expectedly) written contents.

Going back to our Router testing, let me show you the Authentication fake implementation, as you can find in pkg/router_test.go:

type testAuthentication map[string]string

func (a testAuthentication) Authenticate(req *http.Request) (userID string, err error) {
	user, pass, hasBasic := req.BasicAuth()
	if !hasBasic {
		return "", errors.New("no basic")
	}
	if known, has := a[user]; has && pass == known {
		return user, nil
	}
	return "", errors.New("not allowed")
}

This is a pretty common “in-memory” pattern, using a map as the underlying “storage”.

The fake for Persistence is a bit more extensive, since it needs to implement four public methods. Please have a look in pkg/router_test.go, the type is called testPersistence.

HTTP testing

Now with both Fakes in place, we can write the actual tests. I won’t be going through all the tests of our REST API here, but let me walk you through the interesting bits, starting with a test constructor to create a todo.Router instance, which uses the above test Fakes and fills them with useful contents for the test:

func testNewRouter() todo.Router {
	return todo.Router{
		Authentication: testAuthentication{"the-user": "the-pass"},
		Persistence: testPersistence{
			"todo-01": {
				ID:    "todo-01",
				Title: "todo 01",
			},
			"todo-02": {
				ID:    "todo-02",
				Title: "todo 02",
			},
		},
	}
}

Test HTTP request authentication

With that in mind, consider the following test to make sure the Router is using the Authentication to reject requests with invalid credentials:

func TestRouter_ServeHTTP_RejectInvalidCredentials(t *testing.T) {
	req := httptest.NewRequest(http.MethodGet, "/todo", nil)
	req.SetBasicAuth("invalid", "invalid")

	router := testNewRouter()
	rec := httptest.NewRecorder()
	router.ServeHTTP(rec, req)

	res := rec.Result()
	require.Equal(t, http.StatusForbidden, res.StatusCode)
}

The first two lines create a new HTTP request and set invalid HTTP basic auth credentials:

req := httptest.NewRequest(http.MethodGet, "/todo", nil)
req.SetBasicAuth("invalid", "invalid")

In the next block you see a typical HTTP testing pattern:

router := testNewRouter()
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)

Now all we need to do is check the HTTP response status that simulated HTTP request handling returned:

res := rec.Result()
require.Equal(t, http.StatusForbidden, res.StatusCode)

And done. We expect to see a 403 Forbidden response status code and the test will fail if that is not the case.

Test fetching a single Todo

Now towards testing the actual API. Keep in mind the above Persistence fake, which contains two items in the “database”. In the following test, we are checking whether the Router returns the expected Todo for GET requests to the path /todo/todo-01:

func TestRouter_ServeHTTP_Get(t *testing.T) {
	req := httptest.NewRequest(http.MethodGet, "/todo/todo-01", nil)
	req.SetBasicAuth("the-user", "the-pass")

	router := testNewRouter()
	rec := httptest.NewRecorder()
	router.ServeHTTP(rec, req)

	res := rec.Result()
	require.Equal(t, http.StatusOK, res.StatusCode)

	ret, err := ioutil.ReadAll(res.Body)
	require.NoError(t, err)
	out := todo.Todo{}
	require.NoError(t, json.Unmarshal(ret, &out))
	assert.Equal(t, todo.Todo{
		ID:    "todo-01",
		Title: "todo 01",
	}, out)
}

The test starts similar to the previous one, where we checked whether invalid credentials are rejected. Of course we’re now using the “correct” credentials and querying a differnt path (/todo/todo-01) and expected an 200 OK response. Assuming that we get it (and require.Equal makes sure the test aborts if we don’t), the test then reads the returned HTTP response body, decodes it from JSON into a todo.Todo instance (which will fail if it’s not, not valid or not compatible JSON). Finally, we also compare the decode todo.Todo instance with what we put into the fake Persistence above:

ret, err := ioutil.ReadAll(res.Body)
require.NoError(t, err)
out := todo.Todo{}
require.NoError(t, json.Unmarshal(ret, &out))
assert.Equal(t, todo.Todo{
	ID:    "todo-01",
	Title: "todo 01",
}, out)

If that test does not fail, we can be reasonable sure that:

And that’s it. Again, please take a look into the rest of the test source code to find the tests for the other HTTP endpoints.

Please also keep in mind that this is foremost an example and you could be easily optimized. I leave that to the reader as an exercise, if so desired.

Finish

Phew. That was quite a lot. Thanks for reading! I hope you learned something and had a bit of fun on the way. I might continue this series in the future, going deeper into detail topics.

Image of Ulrich Kautz

Ulrich Kautz

ukautz GitHub account ulrich-kautz LinkedIn account