Writing Go Web Clients for Pythonistas

Setting the scene

The history of Python can generally be divided into three eras, each conveniently aligning, more or less, into the three major versions. The first thereof, started at the start of 1994 when Python 1.0 was first released. Most of us have never seen code from this period, unless we’ve deliberately searched it out, but the language in those days found its common use as a simple albeit niche scripting language.

Python 2.0 finally came out in late 2000, marking the second era, and this was when it transitioned from being relatively niche to a heavyweight. When the TIOBE Programming Community Index began in June 2001, Python measured a humble 1.25%, while the Java behemoth scored a colossal 26.49%, over twenty times more popular. In the decade that followed, Python started to gain heavy traction in web development circles, especially with the introduction of Django in 2003 and Flask in 2004, names that every Python developer worth his salt should recognize…

Unless of course you adopted Python during the third and final era of its existence. While Python 3.0 was technically released at the end of 2008, its controversial breaking changes meant that the line between the second and third era is understandably blurred, as vast swathes of the Python landscape assumed the daunting task of porting their existing code to a new language. During that time, while investment in Python’s web development scene didn’t exactly diminish, it was overshadowed by swelling interest in data science. Python has, in the years since, risen to the top of the TIOBE index, and there is little evidence to suggest that its dominance is yet to be challenged.

Be that as it may, Python fits into a broad ecosystem that is the web, and nowadays it has lost much ground to Node.js. But the web is more than just hosting websites and RESTful servers. In the former case, the web browser is needed to render the ever more complex and disorganized HTML that permeates the farthest reaches of the Internet, while in the latter case, web clients make it possible for consumers of your API to write nice, clean code for interacting with your server.

One server, many clients

If you are an experienced Python developer, in all likelihood, you have written a backend server in Django and perhaps a web client to boot. This is no particularly bold or unusual feat, but given someone using Django already likely possesses a firm command of the language, it may be tempting to write that web client in Python and call it a day.

The truth is that for every language for which a web client is written, the greater breadth of possible users one can attract. It is well and good to write a web client for your Python server in Python, but that won’t be of any benefit to a Go developer.

Do not pass Go

For those who don’t know, Go is the brainchild of Robert Griesemer, Ken Thompson (yes, that Ken Thompson), and Rob Pike (of Plan 9 fame). Motivated primarily by a mutual disgust for C++, these men created a C-like language with a heavy focus on networking and concurrency, serving the needs of their Google overlords. Perhaps given the involvement of titans like Thompson and Pike, a language like Go could never fail… Yet one way or another, it distinguished itself from other less-fortunate C derivatives (think Zig or Vala) and has become very popular.

I will not attempt to persuade you that Go is a good language. it is enough only to know that it is widely popular in the web. But what of the poor Django developers who have never read a single line of Go code, let alone written a web client? If you are reading this blog, the existing Go documentation will likely suffice for you to gain a basic understanding of the language, but for any advanced Python developer, you will have to unlearn as many things as you need to learn.

Unlearning Python

Python, while inspired by C, is not a C derivative. Though Go has important differences with C, the gist is the same.

Classes and structs

Generally, a person writing a web client in Python will define their client as a class more or less like the following:

class MyClient:
    def __init__(self, base_url: str) -> None:
        self.base_url: str = base_url

        # The session might be mutated based on SSL verification, etc.
        self.session: requests.Session = requests.Session()

In this way Go is fairly similar, but with some key differences. To begin with, because Go is a C derivative, it represents a block of data as a “struct”. Each member of the struct can either be provided at instantiation, or set thereafter.

type Client struct {
	BaseURL    *url.URL
	HTTPClient *http.Client

	validate *validator.Validate
	logger   slog.Logger
}

The * preceding the types indicates that the struct does not contain the bytes of the pertinent data, but a memory address to where the data is stored. If we were to create a new Client using var client Client, the logger would store the bytes of the slog.Logger, but the other variables would be null pointers, indicating that they do not point at any memory address.

Of course, as a courtesy to the user of your web client, you will provide a function that creates a new Client and sets its members. In Python, __init__ tends to serve a dual purpose of defining both structure and initialization, but in Go, these are distinct concepts from one another. Defining an integer without an explicit value will set it to zero. Defining a string without an explicit value will set it to an empty string. Defining a struct without explicit members will set its members to their corresponding definition-time values. You get the general idea.

Given that you may not wish for your default values to be zero, you can create a function that defines the struct and then sets the values accordingly:

func New(config *ClientConfig) (*Client, error) {
	parsedURL, err := url.Parse(config.BaseURL)
	if err != nil {
		return nil, fmt.Errorf("failed to parse URL: %w", err)
	}

	transport, ok := http.DefaultTransport.(*http.Transport)
	if !ok {
		return nil, fmt.Errorf("failed to assert default transport")
	}
	clonedTransport := transport.Clone()

	var TLSClientConfig *tls.Config
	if clonedTransport.TLSClientConfig == nil {
		TLSClientConfig = &tls.Config{InsecureSkipVerify: config.InsecureSkipVerify}
	} else {
		TLSClientConfig = clonedTransport.TLSClientConfig.Clone()
		TLSClientConfig.InsecureSkipVerify = config.InsecureSkipVerify
	}
	if config.RootCAs != nil {
		if config.InsecureSkipVerify {
			logger := loggerOrDefault(config.Logger)
			logger.Warn("InsecureSkipVerify enabled, nullifying provided RootCAs")
		}
		TLSClientConfig.RootCAs = config.RootCAs.Clone()
	}

	clonedTransport.TLSClientConfig = TLSClientConfig
	httpClient := http.Client{
		Timeout:   config.Timeout,
		Transport: clonedTransport,
	}

	return &Client{
		BaseUrl:    parsedURL,
		HTTPClient: &httpClient,
		validate:   validator.New(),
		BaseUrl:    config.logger,
	}, nil
}

Obviously there’s a fair few things happening here, but New is defined as a function which takes an existing configuration struct. The user presumably creates the ClientConfig variable, populates it, and then uses it to instantiate a Client.

Syntactic porridge

Go upholds a minimal feature set, so it doesn’t allow functions to receive “keyword” arguments or defaults. Python’s syntactic sugar is noticeably missing. Go has syntactic porridge instead. You might not serve it at a restaurant, but it is wholesome and nourishing. For example, in Python, mandatory positional arguments are opt-in, like so:

def minus(x: int, y: int, /) -> int:
    return x - y

Mandatory positional arguments are much less commonly used than keyword-optional or keyword-mandatory arguments. Yet in Go, this is the only way to write such a function. Order is always meaningful, so the more arguments your function has, the less clear it is which function’s arguments are which. So what is the Go equivalent of a Python client like this?

def create_user(
    self,
    *,
    first_name: str,
    last_name: str,
    age: int,
    gender: Literal["male", "female"],
    year_joined: int | None = None,
) -> User: ...

Presumably, if the user doesn’t provide year_joined to the function, the function determines this itself. Of course, we could rely on explicit positional arguments, but we sacrifice clarity at the call sites:

myUser := client.CreateUser("Fred", "Jordan", 12, "male", None)

Which argument is which? 12 could be anything. Why is there a random None? Other questions come to mind. We also don’t have a nice default for the value representing when the user joined, and providing it explicitly could be confusing. Worse still, if we change this signature at all, every single call site will need to be updated.

The better way is to determine which arguments are semantic (i.e. can be inferred from the function’s name and argument’s position), and which arguments are too unclear. In the case of CreateUser, we have ample reason to pass in a struct containing all the values.

func (client *Client) CreateUser(
	ctx context.Context,
	credentials *Credentials,
	payload *CreateUserPayload
) error {
	// Implementation here
}

The CreateUserPayload can then be defined before the CreateUser method is called. Unlike functions, members of structs can easily be defined by name.

createUserPayload := CreateUserPayload{
	FirstName: "Fred",
	LastName:  "Jordan",
	Gender:    "male"
	Age:       12,
}
myUser := client.CreateUser(context.Background(), credentials, &createUserPayload)

The order of Gender and Age being swapped did not matter, nor did it matter that YearJoined is not explicitly provided. If, as we discussed earlier, the struct’s member type is *int, then it is set to nil when not provided, and the implementation can easily determine that the user wishes for the method to determine this value for itself, presumably by finding the current year.

Some context, please?

Many web clients written in Python do too much. They hide things from the user, promising to handle everything under the hood and expose only what is needed of the user. That sounds good, except that often users of Python libraries find themselves having to resort to inventive measures to bypass the “help” they received by the authors of the library in question. This is, of course, not endemic specifically to Python, but because Python makes it so easy to write code quickly, it makes it just as easy to write bad code as it is to write good.

A user then may be frustrated when seemingly obvious considerations are forgotten or assumed to be unimportant, despite being really quite important. For example, a web client may define methods that provide minimal control over the timeout. Perhaps they assume the user must provide a single timeout value for all their methods, or worse still, they don’t implement a timeout and allow methods to hang, with no viable alternative.

Good Pythonic code doesn’t do this, but good Pythonic code is an endangered species nowadays, so some Go conventions may serve to teach us a thing or two. In Go, it is conventionally considered better for lifetime and cancellation control to come explicitly from the user each time they call a method. In the example Go code’s CreateUser method, we took an argument called ctx typed as context.Context, which comes from Go’s standard library.

A user who wants a five-second timeout on the function call can pass in a ctx like this:

ctx := context.WithTimeout(context.Background(), 5*time.Second)

Another example might be if the user, while waiting for a method, wishes to cancel a request it previously made.

ctx, cancel := context.WithCancel(context.Background())
defer cancel()

At some point in an asynchronous action, the user can call the cancel() function, which will abort the request with an error, which the web client method can then pass back to the user.

While this level of control is generally best in the hands of the user, too many Python developers think themselves wiser than their users, and resultantly fail to write what the user actually needs, which is explicit control. And yes, while there is nothing internal to the language forcing context to be in the hands of the user, it is a very strongly enforced convention.

Validation without Pydantic

We’ve talked at rather great length about function signatures, and what the ideal call site should look like for your users, but there’s one other implementation detail which should not escape our attention. Since Pydantic emerged on the scene nearly ten years ago, handling JSON has gone from being insufferable to quite manageable, and even rather pleasant. Take the following User model, for example:

class User(pydantic.BaseModel):
    first_name: str
    last_name: str
    age: int
    gender: Literal["male", "female"]
    year_joined: int

Any Pythonista worth his salt can tell at a quick glance what values are accepted or not. The model itself is much like a dataclass, except with that special sauce which let’s it validate raw JSON or Python primitives.

But while Python has Pydantic, Go has no Godantic, and it simply cannot given that Go has structs and not classes. Structs don’t have inheritance. The idea of a user-defined model is incoherent. There are only structs. However, structs in Go have one thing which their C counterparts do not, and that is tags.

Tags are effectively struct metadata that code handling your struct can use to reason about your data. For instance, the Python code above could be quite simply translated to Go like so:

type rawUser struct {
	FirstName:  *string     `json:"first_name" validate:"required"`
    LastName:   *string     `json:"last_name" validate:"required"`
    Age:        *int        `json:"age" validate:"required"`
    Gender:     *UserGender `json:"gender" validate:"required"`
    YearJoined: *int        `json:"year_joined" validate:"required"`
}

type User struct {
	FirstName:  string
    LastName:   string
    Age:        int
    Gender:     UserGender
    YearJoined: int
}

UserGender here would be a new type like so:

type UserGender string
const (
	UserGenderMale   UserGender = "male"
    UserGenderFemale UserGender = "female"
)

You are possibly wondering why we would use two structs to represent what seems like the same thing, and why they differ in certain ways. The answer isn’t too obvious. Suppose the server that our web client targets promises always include a first name in the user payload. It is probably for the best that our web client should reject a server response that excludes the first name field.

The json:"first_name" tag you saw on the rawUser struct plainly tell the JSON decoder that when decoding a JSON object with the field key first_name, the FirstName member of rawUser should be populated by the deserialized value for that key. That’s simple enough.

But what about the validate:"required" tags? Well, those tags don’t actually affect deserialization at all. They are used during post-deserialization validation. That’s why they might look confusing. Interpreted by the third-party validator library produced by Go Playground, it is used to perform validation on existing structs based on their pertinent tags.

Why does that matter? Because required is the library’s way of saying: are the bytes for this struct member zero? In other words, performing validation on something that is required will return an error when it encounters:

• Integers or floats that are zero.
• Strings that are empty.
• Booleans that are false.
• Slices with a length of zero.
• Nil pointers.
• Structs for which all fields are unset.

So if someone were to use required on a bare string (without a pointer), a struct decoded from this would fail validation:

{"first_name": ""}

In this case, that might be acceptable, though there are many genuine cases where we might accept an empty string for a field. Even if empty strings are unacceptable, the precise error you return could be misleading if the right validation tag is not used.

This is the precise reason why I showed two structs earlier. rawUser can be used for decoding and validation before being converted to User for use by the caller.

There is also admittedly more boilerplate associated with parsing of enums (the UserGender type needs an UnmarshalJSON method), but performing deserialization and validation in Go is more effort than the batteries-included approach with Python and Pydantic.

To boldly Go where no Pythonista has gone before

It is quite possible that I have over-emphasized how important it is for one’s web server to have a Go web client, but Go remains an important and popular language, and one it does not hurt to have in one’s arsenal. Even if it isn’t critical for your project, per se, it may be critical for you, and writing a web client is in many respects an excellent way to learn a language.