Concerning Quality

Bug Bash 2025 Conference Experience

2025-04-12T00:00:00+00:00

The inaugural Bug Bash conference was really special. I’ve been to many conferences, but this was legitimately the first that I felt “a part of,” because the subject matter greatly overlapped with what I’m interested in and what I write about here. There are various combinations of testing conferences, devops conferences, and formal methods conferences, sure, but this still felt like a new stake in the ground. Possibly because of the undeniable connection to deterministic simulation testing, or possibly because it just consisted of a bunch of people on a similar wavelength at the moment. But I’ve personally never been in a room where almost every single person raised their hand when a speaker asked: “who’s familiar with property-based testing?” So it certainly felt like something interesting was in the air.

I left feeling more than ever that generative / autonomous testing is the future, even though I don’t need much encouragement there. There was a broader message though: if we really want correctness and reliability in the presence of radical software complexity, we should be open to everything. From formal verification to testing in prod. From hand-crafted unit test cases to fault-injected end-to-end tests. From sprinkling a few asserts around our codebase to building new infrastructure components to better support deterministic testing. Every approach has different assumptions, tradeoffs, and strengths, so we had better start breaking down the walls between separate and even historically at-odds communities in order to elevate the correctness and reliability of our systems.

That’s a message I can get behind, and that’s the reason I left feeling rejuvenated, and dare I say inspired.

The keynote given by Will Wilson set the stage for the conference as a whole, and perfectly introduced this overarching message. After diving into the question of “why do we have bugs to begin with?”, he posited that testing, verification, and observability are fundamentally not at odds, but rather variations on a theme. At the extreme, each of these seem totally incompatible with one another. Testing can be worlds away from formal verification, because the very act of writing a test is an admission that we can’t model the real execution environment with enough precision. Observability even further distances itself from testing by entirely punting on input data generation and allowing that to be handled by the natural operation of the production system.

The line becomes totally blurred with small variations though, like going from hand-written to generated test inputs. This is much closer to user-generated inputs in prod because we don’t know exactly which inputs will be produced. Observability monitors are also just a generalization of test assertions, and formal properties generalize them both. In the world of generated inputs, testing, observability, and formal methods all start to blend together. Testing and observability particularly blend together when it comes to tracking down the actual cause of a generated test failure.

Cue the Antithesis tool, which is not “just” a deterministic hypervisor, but truly a test execution and analysis platform. Will showed a demo of using it to find a bug in etcd by querying over execution histories that the tool stores. He coined this workflow “pre-observability”: the idea that we can take the same root cause analysis techniques from post-deployment observability and apply them to the massive amount of execution traces produced by simulated system actions.

On top of this, this bug was found much more regularly in the test environment due to fault injection techniques, highlighting one of the tradeoffs of observability vs. testing: sometimes a failure scenario is rare enough in production that it’s inefficient to sit and wait for it to happen. Fault injection speeds up the bug finding process by triggering rare scenarios more frequently. It also highlights the unifying nature of formal methods: fault injection to me has always seemed a practical manifestation of prophecy variables, which were invented precisely to deal with situations like nondeterministic failures in distributed systems.

All in this one talk, we spanned testing, observability, formal methods, and tied it together with a ribbon of determinism. I knew then that this was gonna be a good one.

Now, I’ll describe the overall feel and themes of the conference, since you’ll be able to watch all the talks once they’re posted (which I recommend that you do).

Challenge the Status Quo

One big theme I heard throughout the talks was that we should absolutely challenge the status quo. The obvious example of this is Antithesis itself. I, along with everyone else in the programming world, have been complaining about flaky tests for years. But what I have not done is write a deterministic hypervisor to simply avoid the problem at its root.

Generative testing is also inherently in opposition to the current mainstream state of quality techniques. In his talk about the adoption of the Hypothesis property-based testing library, Zac Hetfield-Dodds mentioned that only 5% of Python users use Hypothesis according to their measurements. When most people think of checking for functional correctness, they think of lots and lots of hand-written example-based tests running in CI, with maybe some linting or static type checking layered on top. They typically don’t think in terms of properties and generating inputs.

Zac remains convicted that testing properties is how we should be testing (I agree), so rather than just give up he shared his overall view on why people don’t adopt it as a practice. His message was that we should primarily focus on the human aspect of property-based testing, for example by making the tests easier to write and improving their error messages. On the easier writing front, Hypothesis added a ‘write’ command that can help bootstrap tests for you. We have a ways to go there, but I appreciated that the maintainer of such a prominent library was taking a step back and really analyzing the situation.

The most interesting manifestation of this theme though was in Kyle Kingsbury’s talk on the most recent Jepsen analyses. He was describing the analysis done on Datomic, where he encountered a surprising behavior that didn’t really fall into any of the existing terms we have for transaction anomalies. An eerie vibe came over the room when he suggested that we may have to create new terms for situations like these. If aphyr himself, one of the only people I know of who can tell the difference between G1a and G2-item phenomena, doesn’t have the words to describe a distributed system scenario, what the heck are we supposed to do?

This may be a more personal revelation, but that moment made me realize: we are the adults in the room now. It’s not enough to just rehash things that Leslie Lamport or Barbara Liskov discovered 30-50 years ago. We need to be the ones doing new research, building new tools, or thinking of how we can create infrastructure that allows us to gain more control over our software. If we’re unhappy with the state of the world, we wield the power to change it. And many people are actively working on this.

Test End-to-End

Another common theme was end-to-end testing. End-to-end testing can be a dirty word in some circles, but this group of speakers went all-in on it. Mitchell Hashimoto didn’t get to true end-to-end testing until the end of his talk about making hard-to-test code testable, but he gave a great variety of advice on applying the functional-core-imperative-shell style of design to a codebase. This enables tests of interacting components, in this case stopping only at actual GPU instruction execution. This approach implies that mocking should only be done at strategic system boundaries, which I think is fantastic advice in general.

But then he went on to talk about full end-to-end testing via NixOS VM testing for the final bits that you just don’t want to abstract away. This was actually the first time I heard about Nix’s VM testing, and this looks like a great tool for anyone plagued by the inconsistency of e2e test infrastructure management. I’m definitely going to give it a further look.

Stephanie Wang spoke about all of the reliability lessons she learned while building MotherDuck, and someone from the audience asked how some of these were verified. She replied that they performed lots of chaos testing using a mock network interface for controllability, of course. When she spoke about minimizing data movement via caching as a win for both reliability and performance, I can’t think of a unit test that would give any kind of confidence about that. And this was the main assertion in Ben Egger’s talk about testing in prod at OpenAI, which is the most extreme form of end to end testing: no matter how well you model your system, prod is the concrete instantiation of it, and you shouldn’t ignore the ways that all of your components interact in the production setting.

End to end testing is the perfect example of where testing and observability have a lot in common. The further toward prod your test moves, the more you have to worry about collecting information from hard-to-reach infrastructure components, and the more the semantics of these components influence system behavior. This part about hard-to-reproduce semantics is the whole reason Jepsen takes the end to end testing approach (“as God intended it” as Kyle likes to say). Because unit tests are great and all, but will they catch anomalies caused by weak transaction isolation?.

Formal Methods Push The Limits of Testing

This was my personal favorite theme because it so perfectly puts my intuition into words. This was spoken almost verbatim by Ankush Desai in his talk about formal and semi-formal methods at AWS. His point was that the value of formal techniques aren’t just limited to actual formal verification. The logical approach that formal methods take can be used to come up with testing techniques as well.

This to me is the sweet spot of formal methods, at least in today’s landscape. Formality gives us the framework and strategy for figuring out what exactly we should be looking for and how we should think about systems, but we can use tests in place of proofs when it comes to the checking part. We sacrifice the completeness of the checking in the name of practicality and efficiency: generative tests can’t prove a property, but they provide a much higher level of confidence than a few hand-crafted example scenarios. This is an idea that others have shone light on as well: the Cogent sub-project of seL4 wrote a paper about using property-based tests as an intermediary on their way to proofs in the verification of a filesystem implementation.

In Ankush’s talk, he introduced PObserve, a framework for checking a production system against a specification written in P, a language that he created, and one in use within AWS. Instead of using the spec to prove the implementation correct, it takes logs from the real system and checks that they adhere to the specification. This is similar to model-based tests which many property-based testing libraries support, but it instead takes the observability-inspired approach of checking execution traces extracted from the actual running system. It also reminds me of Tracetest.

This was another case of the trifecta of testing, observability, and formal methods all working together, this time with a focus on validating implementation behavior against a model. We sometimes refer to such approaches as “lightweight formal methods,” and this is the area that I see being most likely to be implemented in a practical setting.

On top of the actual talks, there was a very social vibe to the conference in general, which is a hard thing to fake. I got the sense that there’s very much an appetite for the particular brand of autonomous testing, lightweight formal methods, and observability techniques being presented there. Overall, it was a great experience, and really galvanized and clarified ideas that I’ve been mulling over for a while now. Thank you to Antithesis for shepherding this conversation and getting everyone together under one roof to contribute to it. If there’s a Bug Bash 2026, I will certainly be first in line for a ticket.

Branch Coverage Won’t Prove The Collatz Conjecture

2025-01-26T00:00:00+00:00

The Collatz conjecture is the prime example of the limitations of thinking in terms of branch coverage. It can be written as a recursive function in 5 lines of code with only three branches. That’s great, except we have no idea if it’s true or not, and no amount of tests can prove either way.

Here’s the code for generating the Collatz process:

function collatz(n: number): boolean {
    if (n === 1) {
        return true;
    }

    return n % 2 === 0 ? collatz(n / 2) : collatz(3 * n + 1);
}

It’s quite simple, by almost any metric. Just a couple of conditionals, and some plain arithmetic. The conjecture is that this always returns true: no matter the starting number, all paths should end at 1, says Collatz.

There’s only a few lines of code. Let’s just test all the branches. To make this a little more explicit, let’s unwind the ternary into an if-else:

function collatz(n: number): boolean {
    if (n === 1) {
        return true;
    }

    if (n % 2 === 0) {
        return collatz(n / 2)
    } else {
        return collatz(3 * n + 1);
    }
}

We only need 2 test cases to hit all 3 branches: n=2, and n=3. Here’s the sequence of n values that result in each case, just to get a feel for how the state progresses:

n=2 -> n=1 ==> true
n=3 -> n=10 -> n=5 -> n=16 -> n=8 -> n=4 -> n=2 -> n=1 ==> true

That was easy. All the branches are covered. There’s just one problem: since it was proposed in the 1930s, the entirety of the math community has been unable to prove this true or not. We don’t know if this is just a pattern up to some gigantic value of n, after which it breaks down, or if it’s the real deal and we can finally watch it grow up into a real theorem. We simply don’t know for sure if it’s always true, or even within what bounds it is true. The issue is that the state oscillates. If we could show that every iteration of the recursion produced a smaller value, then we’d be sure that we’ll always get down to 1. But when n is odd, we go up. The progress is inconsistent. It, pretty surprisingly given its apparent simplicity, completely eludes our species.

Look back at the above test cases and how they create sequences of n values. Sequences like this are what software behavior boils down to. A program is really two things: its code, along with the set of all behaviors that it produces. Branch coverage is a statement about the code, but it doesn’t touch the full breadth of the runtime behavior of the program. And the runtime behavior is what determines correctness.

This is why a tiny little function can lead to an unknowable question. There are lots of numbers, so lots of possible sequences of n, and in this case the code branches keep getting revisited until the program terminates. That is, if it terminates.

Branch coverage will get a small glimpse of your code’s behavior, but it isn’t enough to prove the Collatz conjecture.

Simulating Some Queues

2025-01-03T00:00:00+00:00

System performance boils down to the timing behavior of various interacting queues. Queues are one of those incredibly simple but powerful concepts, but they have some unintuitive or non-obvious behavior when only thinking about them mathematically. Simulating queueing scenarios gives us a better picture about how queues operate in practice.

The Unit Queue

Let’s introduce the simplest possible queue as a reference point, which we’ll call the unit queue. Requests arrive once per second, and each request takes one second to process. There’s only one processor that services requests. Here are some quick definitions about the operations of this queue:

Arrival Rate: the rate that requests come into the queue. Here, it is 1 / second.
Processing Time: the time it takes to process a request. Here, it’s 1 second.
Wait Time: the amount of time a request waits after arrival and before processing begins. Here, the wait time for all requests is 0.
Latency: the total time it takes to process a request after arrival. Here, it’s 1 second.
Active Request: a request that’s currently being processed.
Queued Request: a request that’s waiting to be processed after arrival.
Queue Length: the number of queued requests. Here, it’s always 0.

This queue is in an equilibrium state: as soon as a request is done being processed, a new one comes in. And before the next one comes in, the current request has enough time to complete. This means that a request never has to wait to be processed, and it begins processing as soon as it comes in. Because of this, the queue length is always 0 and never grows.

Discrete Event Simulation

This won’t be a deep dive into discrete event simulation, but it helps to know a few things about it to understand the data that we’re generating in our simulations. You can read more in the SimPy docs.

The basic idea is that we emit events for system changes, and the system is assumed to be in the same state between events. Because of this, we can “fast forward” time by only considering the events and not waiting for time to pass. It’s another manifestation of the state machine model of a system, only here we can keep track of the duration of each transition instead of only worrying about the states that changed.

In the case of a queue, we’ll broadcast one event for each of:

request arrival
processing start
processing end

With just these events, we can calculate:

wait time (processing start - request arrival)
processing time (processing end - processing start)
latency (processing end - request arrival)

We can also record queue lengths whenever a request arrives. We’ll look at some code in a bit, let’s just focus on the behavior that the simulation gives us for the moment. Simulating 5 minutes of the unit queue leads to the following graphs:

Visually, equilibrium is a bunch of straight lines. The reason the lines can remain straight is because the queue length is always 0, so no wait time ever gets introduced. Let’s see what happens when we break this.

Queue equilibrium relies on the following inequality always being true:

\[processing\ time \leq arrival\ rate\]

If the processing time ever exceeds the arrival rate, the queue length will begin to grow, and thus some wait time will be added to the latency of subsequent requests. Let’s simulate the same 1 / second arrival rate, but with a 2 second processing time (note that 300 requests aren’t recorded any more. This is because fewer requests can complete in the fixed time window when queueing is introduced):

The processing time remains constant at 2 seconds, but latency, wait time, and queue length all increase. It’s actually worse, they increase indefinitely. This queue will never catch up, because the processing time exceeds the arrival rate. It is saturated.

The effect is brutal. After 100 requests arrive in the queue, only 50 have been processed, so there’s a queue of 50 requests. Requests 101 and onward wait for 100 seconds before even beginning processing, and their total latency reflects this.

The lesson here is: there’s no ideal processing time or arrival rate. Their relationship is what matters, so we need to know both. Even if there’s no change in the processing time of a request, an increase in arrivals will lead to queueing and increased latencies across the board.

Processing Distributions

Here’s where simulations really become useful. We obviously won’t have a system with constant processing times. They’ll depend on any number of factors: customer size, data skew, current system load, etc. Let’s look at what happens when we set the average processing time back to 1, but this time we distribute the times exponentially. As a reminder, the exponential distribution favors smaller values, but there’s a long tail of large ones. It looks like this:

This shows 100,000 random samplings of an exponential distribution. Thinking of it in terms of the number of requests that fall into the given processing time ranges, ~70,000 requests would be between 0 and 1 seconds, and ~90,000 would be between 0 and 2 seconds (90% of all requests). A relatively small number of requests would take more than 2 seconds, but we get requests all the way up to 12 seconds.

The average of all of these is still ~1 second. Let’s see what happens when these are the processing times instead of the constant 1 second, keeping the 1 / second arrival rate constant:

The max processing time looks to be around 5 seconds, and there are only a few requests that high. But the latency increases to over 10 seconds at parts, because there’s a big swell in the queue length at around request 150. I calculated some more metrics for this particular simulation run:

p99 latency: 4.26s
Average wait time: 3.34
Average queue length: 2.93

Even with an average processing time of 1 second, and rare long requests, there is still pretty constant queueing here.

The lesson here being: don’t only look at average request times, because there can be wildly different queueing characteristics for the same average value. The processing time distribution should always be considered.

In this particular case, where we only have one processor servicing the queue and a processing time distribution with a long tail, we can smooth out the queueing by adding an additional processor:

This is slightly surprising, because we know that we have requests up to and past 5 seconds. If multiple of those happen at the same time, even with two processors they should clog up the queue and introduce queueing. But, we know that those long requests are rare, so the odds of two of them getting processed at the same time is low. It still does happen, as we see by the queue length increasing at certain points, but the queue recovers quickly. Average wait time for this run was 0.08 seconds, and the average queue length was 0.04. So, any latency is due to the actual request processing time, which is ideal.

Code

Now for a little code, for those who are interested in running their own simulations (you’ll need to install simpy, matplotlib, and numpy via your favorite Python dependency manager):

import simpy
import itertools
import random
from dataclasses import dataclass
import matplotlib.pyplot as plt
import numpy as np

SIM_DURATION = 300

EXPONENTIAL_DIST = 'exponential'
GAUSSIAN_DIST = 'gaussian'
UNIFORM_DIST = 'uniform'
CONSTANT_DIST = 'constant'

MEAN_PROCESSING_TIME = 1
NUM_PROCESSORS = 1

@dataclass
class Monitor:
    wait_times: list[float]
    queue_lengths: list[int]
    latencies: list[float]
    processing_times: list[float]

class Queue:
    queue: simpy.Resource

    def __init__(self, env):
        self.env = env
        self.queue = simpy.Resource(env,  NUM_PROCESSORS)

def request(env, n, dist, monitor, queue):
    arrival = env.now
    monitor.queue_lengths.append(len(queue.queue.queue))
    with queue.queue.request() as req:
        yield req
        wait = env.now - arrival
        monitor.wait_times.append(wait)

        processing_start = env.now
        execution_time = 1
        mean_processing_time = MEAN_PROCESSING_TIME
        if dist == EXPONENTIAL_DIST:
            execution_time = random.expovariate(1 / mean_processing_time)
        elif dist == GAUSSIAN_DIST:
            execution_time = random.gauss(mean_processing_time, mean_processing_time / 4)
        elif dist == UNIFORM_DIST:
            delta = mean_processing_time * 0.5
            execution_time = random.uniform(mean_processing_time - delta, mean_processing_time + delta)
        elif dist == CONSTANT_DIST:
            execution_time = mean_processing_time

        yield env.timeout(execution_time)
        monitor.processing_times.append(env.now - processing_start)
        monitor.latencies.append(env.now - arrival)

def generate_load(env, latency_dist, monitor, queue):
    req_count = itertools.count()
    while True:
        yield env.timeout(1)
        env.process(request(env, next(req_count), latency_dist, monitor, queue))

def simulate_queue(latency_dist, monitor):
    env = simpy.Environment()
    q = Queue(env)
    env.process(generate_load(env, latency_dist, monitor, q))

    env.run(until=SIM_DURATION)

monitors = {}
for latency_dist in [CONSTANT_DIST, UNIFORM_DIST]:
    monitor = Monitor([], [], [], [])
    simulate_queue(latency_dist, monitor)
    monitors[latency_dist] = monitor

    print(f"Wait times: {monitor.wait_times}")
    print(f"Queue lengths: {monitor.queue_lengths}")
    print(f"Latencies: {monitor.latencies}")

    print()
    print(f"Average wait time: {sum(monitor.wait_times) / len(monitor.wait_times):.2f}")
    print(f"Average latency: {sum(monitor.latencies) / len(monitor.latencies):.2f}")
    print(f"p99 Latency: {np.percentile(np.array(monitor.latencies), 99):.2f}")
    print(f"Average queue length: {sum(monitor.queue_lengths) / len(monitor.queue_lengths):.2f}")
    print(f"Average processing time: {sum(monitor.processing_times) / len(monitor.processing_times):.2f}")

    print()

plot_monitors(monitors)

This is set up to run multiple different processing time distributions, which you can mix and match to compare. The main thing to know about the pattern that SimPy employs is that you yield events, which basically means “wait for this event to occur.” In generate_load we first yield env.timeout(1), which is how we say to send requests every 1 second. To be pedantic, this just sends it every one “time unit,” and we are just interpreting it as the unit being seconds.

After that timeout completes, we run the request function which interacts with the queue. SimPy has the concept of a Resource which is a thing that can only be accessed a finite number of times. A Resource with a capacity set to 1 is equivalent to a queue with 1 processor. We wait for the queue to be available with:

with queue.queue.request() as req:
    yield req
    ...

Then we pick a distribution and sample a value out of it, and wait for another timeout event with yield env.timeout(execution_time) which simulates the request processing time. We pass a Monitor object throughout which keeps track of the various raw pieces of data so we can plot them later.

Here’s the definition of plot_monitors for completeness:

def plot_monitors(monitors):
    for dist, monitor in monitors.items():
        plot_monitor(dist, monitor,)

def plot_monitor(dist, monitor):
    min_len = min(len(monitor.wait_times), len(monitor.latencies), len(monitor.queue_lengths), len(monitor.processing_times))

    x_values = range(min_len)
    y_values_list = [
        monitor.processing_times[:min_len],
        monitor.latencies[:min_len],
        monitor.wait_times[:min_len],
        monitor.queue_lengths[:min_len],
    ]
    y_labels = ["Processing Time (s)", "Latency (s)", "Wait Time (s)", "Queue Length"]
    titles = ["Processing Time", "Latency", "Wait Time", "Queue Length"]

    num_subplots = len(y_values_list)
    fig, axes = plt.subplots(num_subplots, 1, figsize=(8, 6))

    for i, ax in enumerate(axes):
        ax.plot(x_values, y_values_list[i], linestyle='-', label=y_labels[i])
        ax.set_title(titles[i], fontweight='bold')
        ax.set_xlabel("Request Number")
        ax.set_ylabel(y_labels[i])
        ax.grid(True, linestyle='--', alpha=0.7)
    
    plt.tight_layout()
    plt.show()

Controlling Nondeterminism in Model-Based Tests with Prophecy Variables

2024-12-23T00:00:00+00:00

We have to constantly wrestle with nondeterminism in tests. Model-based tests present unique challenges in dealing with it, since the model must support the implementation’s nondeterministic behavior without leading to flaky failures. In traditional example-based tests, nondeterminism is often controlled by adding stubs, but it’s not immediately clear how to apply this in a model-based context where tests are generated. We’ll look to the theory of refinement mappings for a solution.

In model-based testing, we construct a model of the system and use it as an executable specification in tests. One of the main benefits of doing this is that we end up with a highly-simplified description of the system’s behavior, bereft of low-level details like network protocols, serialization, concurrency, asynchronicity, disk drives, operating system processes, etc. The implementation, however, has all of these things, and is beholden to their semantics.

This generally means that model states are not equivalent to implementation states, and are thus not directly comparable. This is fine, because we can define a refinement mapping between them and carry on. Nondeterminism complicates this mapping though.

Let’s look at a concrete example. Here’s a model of an authentication system, that allows for the creation of new users:

type User = {
  id: number;
  username: string;
  password: string;
}

type CreateUser = {
  username: string;
  password: string;
}

type AuthError = 'username_exists';

class Auth {
  users: User[] = [];
  error: AuthError | null = null;

  createUser(toCreate: CreateUser) {
    if (this.users.some(u => u.username === toCreate.username)) {
      this.error = 'username_exists';
      return;
    }

    const user: User = {
      username: toCreate.username,
      password: toCreate.password
    }

    this.users.push(user);
  }
}

In this model, we have a set of Users that we can add to, or in doing so there might be an error if the username is already taken. This error is a domain error, related to the logic of authentication, so it’s essential to include in the model.

Not all errors are alike. In a real implementation, we’re going to have timeouts set on the web request as well as database statements. Timeouts are unrelated to the domain of authentication, and they also happen to be non-deterministic: for the same inputs, a timeout may or may not occur based on system load. It’s not obvious what to do about this, but if we do nothing, two problems arise:

A timeout in the test could lead to a flaky test failure.
We don’t sufficiently test the timeout-handling codepath.

These need to be addressed.

Handling Implementation-Level Errors in a Model

What does a timeout in the implementation mean in terms of the model? There’s two main interpretations:

It corresponds to a no-op in the model (aka a stutter step).
It maps to some separate error value in the model.

Either one isn’t more correct than the other, but be aware that allowing for stutter steps leads to potential false positive passing tests. If a timeout occurs in the createUser operation, no new users will be added to the set of all users, but the test will still pass because we chose to allow for equal initial and final states. Stutter steps are necessary in theory, but we should be careful when allowing for them in tests otherwise our test suite will pass on a run where 100% of calls to createUser time out.

There are ways of mitigating the risk of vacuously passing tests. For example, we could make a statistical correctness statement: the test only passes if no more than 10% of createUser operations time out. This is more of a statement about reliability though, and not a statement about functional behavior. I think it’s best to keep functional behavior tests in the domain of logical time, and to instead use observability tools for collecting reliability metrics.

For functional testing, there’s a better way that avoids statistical correctness statements. It just involves predicting the future.

Tests, Oracles, and Prophecy

A brief philosophical aside. Tests are almost entirely about seeing into the future. By simply writing down the expected outputs of an operation, that means that we know what they should be ahead of time. We are the so called test oracle. In model-based testing, we instead delegate this prediction to the model: the model is the oracle.

There’s a very well-known solution to the problem of predicting the future of a nondeterministic operation in a test: test doubles. Stubs in particular are commonly used to control things like timeouts. Say we have a client-server implementation of our Auth module. We’d likely make client-side network requests through an interface and use stubs in our tests to control the code path taken:

type User { ... }
type AuthSystemError = 'timeout';
type AuthError = 'username_exists';
type AuthServerResponse = User | AuthError | AuthSystemError;

interface AuthServer {
  createUser(toCreate: CreateUser): AuthServerResponse;
}

class AuthClient {
  users: User[] = [];
  server: AuthServer;
  error: string | null = null;

  constructor(server: AuthServer) {
    this.server = server;
  }

  createUser(toCreate: CreateUser) {
    const resp = this.server.createUser(toCreate);
    if (resp === 'timeout') {
      this.error = 'There was a problem creating the user. Please try again or contact support.';
    } else if (resp === 'username_exists') {
      this.error = 'That username is already taken. Please choose another.';
    } else {
      this.users.push(resp);
    }
  }
}

// test file:
class AuthServerTimeout implements AuthServer {
  createUser(toCreate: CreateUser): AuthServerResponse {
    return 'timeout';
  }
}

describe('Timeout behavior', () => {
  it('displays a timeout message when the request times out', () => {
    const auth = new AuthClient(new AuthServerTimeout());
    auth.createUser({ username: 'user', password: 'pass' });

    expect(auth.error).toEqual('There was a problem creating the user. Please try again or contact support.');
  });
});

This pattern is ingrained in our muscle memory, but it’s actually quite interesting from the perspective of oracles and predicting the future. The simple AuthClient has code paths that are not ergonomic to trigger in a test (we like to avoid the use of sleep anywhere in tests, and otherwise the timeout will be dependent on nondeterministic system load). So instead of triggering the scenario that leads to a timeout, we simply setup the code in a way that guarantees the timeout code path is taken. In effect, we tell the code under test what its own destiny is, and use that to be able to create a dependable, deterministic assertion in the test.

From the test-writers point of view, this is a simple technique, but from the code’s point of view, it’s as if we’re showing it a prophecy of its life ahead of time. We are an oracle indeed!

In model-based tests, we don’t create individual test cases, so we need a way to generate different stub configurations if we want to test a timeout code path. Once we put it that way, the answer is simple: just generate a variable that we can use to dynamically configure stubs. Because this variable predicts future execution, we call it a prophecy variable. For this, we can name it isTimeout, and go from there. First we extend the model to be aware of this variable:

class Auth {
  // ...
  error: AuthError | AuthSystemError | null = null;

  createUser(toCreate: CreateUser, isTimeout: boolean) {
    if (isTimeout) {
      this.error = 'timeout';
      return;
    }

    // ...
  }
}

This avoids the stutter-step issue from before. We elevate the system-level error to the model level, and we make it so that timeout error only occurs when isTimeout tells it to. This is how we can be sure that unintended timeouts aren’t happening in the tests. Then, the implementation:

class AuthServerImpl {
    users: User[] = [];

    createUser(toCreate: CreateUser): AuthServerResponse {
      // real networking / server impl
    }
}

class Client {
    users: User[] = [];
    error: AuthError | null = null;
    implError: AuthSystemError | null = null;
    
    server: AuthServer;

    constructor(server: AuthServer) {
      this.server = server;
    }

    createUser(toCreate: CreateUser) {
      const result = this.server.createUser(toCreate);
      if (result === 'timeout') {
        this.implError = result;
        return;
      }

      if (result === 'username_exists') {
        this.error = result;
        return;
      }

      this.users.push(result);
    }
}

And here’s what the model-based test would look like:

const genToCreate = () => fc.record({
  username: fc.string(),
  password: fc.string()
});

const genUser = () => fc.record({
  id: fc.integer(),
  username: fc.string(),
  password: fc.string()
});

const genUsers = () => fc.array(genUser());

const genProphecy = () => fc.boolean();

const externalAuthState = (auth: Auth): AuthState => {
  return {
    users: auth.users,
    error: auth.error
  }
}

const externalClientState = (client: Client): ClientState => {
  return {
    users: client.users,
    error: client.error,
    implError: client.implError,
  }
}

const refinementMapping = (isTimeout: boolean, implState: ClientState): AuthState => {
  return {
    users: implState.users,
    error: isTimeout? implState.implError : implState.error,
  }
}

describe('Prophecy-aware Auth test', () => {
  it('should correspond to the model', () => {
    fc.assert(
      fc.property(genUsers(), genToCreate(), genProphecy(), (users, toCreate, isTimeout) => {
        const auth = new Auth();
        auth.users = [...users];

        let server: AuthServer;
        if (isTimeout) {
          server = new AuthServerTimeout();
        } else {
          const realServer = new AuthServerImpl();
          server = realServer;
        }
        const client = new Client(server);
        client.users = [...users];

        auth.createUser(toCreate, isTimeout);
        client.createUser(toCreate);

        const authState = externalAuthState(auth);
        const mappedState = refinementMapping(isTimeout, externalClientState(client));
        expect(mappedState).toEqual(authState);
      }),
      { endOnFailure: true, numRuns: 10000}
    );
  });
  });

We use isTimeout to choose which AuthServer implementation to use, and we compare the now-prophecy-aware implementation to the model. To compare the different state values, we do a little bit of bookkeeping, first by projecting each object to an “external” state which omits any implementation details. We also create a refinementMapping function which maps implementation states to model states. The refinement mapping is also aware of the isTimeout variable, and uses that to make sure we only elevate the implementation error to the model when it is prophesied.

Now, we have a pattern for building property-based tests that can account for nondeterministic errors.

A Brief Note on The Theory of Prophecy Variables

Prophecy variables are much more powerful than simple stubs in example-based tests, but I can’t but help notice the practical similarity between them. They were introduced in the paper The Existence of Refinement Mappings to solve the theoretical problem of proving refinement between specifications with nondeterminism. The paper showed that there are programs where proving the refinement of their specification is impossible due to nondeterminism. Not only do prophecy variables solve that problem, they also lead to a complete solution to the problem. The main result in the paper is that we can find a suitable refinement mapping for any program to any specification, as long as we are able to add history and prophecy variables to the refinement mapping in a way that doesn’t alter the observable behavior of either the program or the spec.

That’s true of test doubles: they don’t alter the code under test, they just allow for specifying values ahead of time, which again is the key to dealing with nondeterminism in tests. Our usage of prophecy variables here differs slightly from the theoretical versions (we pass ours into the refinement mapping function rather than keeping the function as a pure mapping from implementation to model state, and we also use interfaces and stubs to modify the behavior rather than only limiting ourselves to state variables). Still, this departure is only surface-level, since we could map this all to the TLA+-style state framework if we wanted to. Using the idioms of the particular programming language we’re in makes for a more practical experience.

For more info, there’s a deeper dive into the theory of refinement mapping in Efficient and Flexible Model-Based Testing. There’s a whole paper dedicated to prophecy variables in Prophecy Made Simple. There’s also the fantastic blog post Linearizability! Refinement! Prophecy! that goes into a really detailed example of using prophecy variables to prove properties of nondeterministic queues.

This will be the most ambitious part of the post. It begins with a statement: we should design our dependencies to be prophecy-aware.

Dependencies are a double-edged sword, especially infrastructure dependencies like a database. On the one hand, we get an incredible amount of power and reliability that would be impossible to implement on our own. On the other, we lose control, and are beholden to extremely fine-grained semantics that, among other things, make holistic testing difficult. I greatly believe in integration testing, especially against something like a database, because of such semantics that our applications come to depend on. I wrote about this in Does Your Test Suite Account For Weak Transaction Isolation?. Things like transaction isolation ultimately affect the correctness of our applications, so their absence from most application test suites is an unideal blind spot.

This absence is totally understandable though: testing for it is a pain, precisely due to the inability to control nondeterminism. To systems and infrastructure developers: please account for the testing of nondeterministic functionality in the design of your tools. All nondeterministic choices should be able to be controllable by parameters. This allows nondeterminism to be used where necessary (and it often is necessary and not just a mistake, e.g. for performance or concurrency), while also being able to be controlled in tests. There’s definitely an upswing in projects thinking about this up front, notable examples being FoundationDB and TigerBeetle. I don’t want to make light of it, because it can radically alter the design of a system. But, having controllable determinism will always be a good thing in my book.

However, in the meantime, most of our dependencies are not prophecy-aware, so we do need an approach for handling them as-is. For this, I think our best bet is to create wrapper fakes which model a given dependency. These models will need to be nondeterministic, since the implementation is, however we can design them to also be prophecy-aware and thus controllable in tests as well. Because such models have this dual behavior, I think of this as “modal determinism.”

Let’s continue with the example of transaction isolation in Postgres. And let’s say that we first discovered weak transaction isolation and the Read Committed isolation level. We start to hone in on this being an issue, and we first write this test (against a real PG DB):

-- Create test schema
create table txn_iso (ival int);
insert into txn_iso (ival) values(1);

import * as fc from 'fast-check';
import { Database, DBModelNondet } from './database';
import { PoolClient } from 'pg';

type Tuple = {column: string; value: any}[];

class Database {
    pool: pg.Pool;

    constructor() {
      this.pool = new Pool(/* connection info */);
    }

    async selectInClient(client: pg.PoolClient): Promise<Tuple[]> {
      const res = await client.query(`SELECT * FROM txn_iso`);
      return res.rows.map((row) => {
        return [{ column: 'ival', value: row['ival'] }];
      });
    }

    async update(val: number)  {
      const client = await this.pool.connect();

      await this.updateInClient(client, val);

      client.release();
    }

    async updateInClient(client: pg.PoolClient, val: number) {
      return client.query(`UPDATE txn_iso SET ival = $1`, [val]);
    }
}

const genUpdateVal = () => fc.integer({ min: 0, max: 10 });

const genTxnOrder = () => fc.uniqueArray(fc.integer({ min: 0, max: 2 }), {minLength: 3, maxLength: 3});

const initialVal = 1;

describe('Database nondeterministic transaction reads', () => {
  it('should return consistent reads', async () => {
    let db: Database;
    let c1: PoolClient;
    let c2: PoolClient;
    await fc.assert(
      fc.asyncProperty(
        genUpdateVal(),
        genTxnOrder(),
        async (val, txnOrder) => {
          db  = new Database();

          c1 = await db.pool.connect();
          c2 = await db.pool.connect(); 

          await c1.query('BEGIN');
          await c2.query('BEGIN');
          const prevRead = await db.selectInClient(c2);
          await db.updateInClient(c1, val);

          const operations = [c1.query('COMMIT'), c2.query('COMMIT'), db.selectInClient(c2)];
          let orderedOperations = [];
          let readIdx = txnOrder[2];
          for (let i = 0; i < txnOrder.length; i++) {
            orderedOperations[txnOrder[i]] = operations[i];
          }

          const results = await Promise.allSettled(orderedOperations);
          const read = results[readIdx];
          
          if (read.status === 'fulfilled') {
            expect(read.value).toEqual(prevRead);
          } else {
            fail('Read failed');
          }
      }).afterEach(async () => {
        await db.update(initialVal);

        c1.release();
        c2.release();

        await db.pool.end();
      }),
      { endOnFailure: true, numRuns: 100}
    )
  });
});

This test creates two DB connections, one of which is updating a value in the txn_iso table, and another which reads it multiple times. We expect that the multiple reads return the same value, but they don’t. We also randomize the order of the commits of the transactions to exacerbate the issue, but even without that the test will fail nondeterminstically.

This is complex and surprising behavior, and we want to build a model of it so that we can deterministically control it in our application tests to get more realistic coverage. The key is recognizing that the model has to support this nondeterminism by returning multiple possible values for select statements instead of just a single one. We can then create a model-based test that allows for any of the possible values to be returned in the implementation. This draws inspiration from the nondeterministic seL4 specification which defines nondeterminism as transitioning between multiple allowable states.

We create the following model:

type Tuple = {column: string; value: any}[];
type Relation = { name: string, data: Tuple[] };
type Transaction = {id: number, isDirty: boolean, prev: Relation[], next: Relation[]};

type DBState = {
  relations: Relation[];
  transactions: Transaction[];
};

class DBModelNondet {
  state: DBState[] = [];

  select(txnId: number, relation: string): Tuple[][] {
    return this.state.map((s) => {
      const dirtyTxn = s.transactions.find((txn) => txn.id === txnId && txn.isDirty);
      if (dirtyTxn) {
        return dirtyTxn.next.find((rel) => rel.name === relation)?.data ?? [];
      }

      return s.relations.find((rel) => rel.name === relation)?.data ?? []
    });
  }    
}

We model the database (DBState) as a list of Relations, where each Relation is itself a list of Tuples. We also model transactions as having an id, a previous list of relations, a next list of relations, as well as an isDirty flag which signals whether or not the transaction has written any data at this point in time. The prev list of relations tracks the snapshot of the DB state when the transaction was started, and next tracks the current state including any transaction-local modifications that haven’t been committed yet.

We then store an array of these DBStates, not just a single one. Because the database hides a nondeterministic choice from us (the order of operations of when concurrent connections are scheduled), we have to support multiple initial starting states in the model. This allows us to handle both cases of the race condition here: where connection c1 is committed before and after the second read in c2.

Then, we write a simplified select model that executes within a specified transaction and returns all rows of a particular relation. For each current state, the select either returns tuples that have been modified in an in-progress transaction, or falls back to the committed state if the transaction hasn’t modified anything. Because there can be multiple states, select is also nondeterministic and returns a list of Tuple lists.

This surprisingly simple model allows us to accurately model non-repeatable reads. We can use it to ensure it supports the nondeterminism caught in the previous test:

describe('Database reads nondet model', () => {
  it('should return any of a set of allowable reads', async () => {
    let db: Database;
    let c1: PoolClient;
    let c2: PoolClient;

    await fc.assert(
      fc.asyncProperty(
        genUpdateVal(),
        genTxnOrder(),
        async (val, txnOrder) => {
          db  = new Database();
          const model = new DBModelNondet();

          c1 = await db.pool.connect();
          c2 = await db.pool.connect(); 

          await c1.query('BEGIN');
          await c2.query('BEGIN');
          await db.updateInClient(c1, val);

          model.state = [
            // State 1: write transaction has not been committed yet.
            {
              relations: [{ name: 'txn_iso', data: [[{ column: 'ival', value: initialVal }]] }],
              transactions: [
                { 
                  id: 1,
                  isDirty: false,
                  prev: [{ name: 'txn_iso', data: [[{ column: 'ival', value: initialVal }]] }],
                  next: [{ name: 'txn_iso', data: [[{ column: 'ival', value: val }]] }] 
                },
                {
                  id: 2,
                  isDirty: false,
                  prev: [{ name: 'txn_iso', data: [[{ column: 'ival', value: initialVal }]] }],
                  next: [{ name: 'txn_iso', data: [[{ column: 'ival', value: initialVal }]] }] 
                },
              ]
            },

            // State 2: write transaction has been committed
            {
              relations: [{ name: 'txn_iso', data: [[{ column: 'ival', value: val }]] }],
              transactions: [
                {
                  id: 2,
                  isDirty: false,
                  prev: [{ name: 'txn_iso', data: [[{ column: 'ival', value: initialVal }]] }],
                  next: [{ name: 'txn_iso', data: [[{ column: 'ival', value: initialVal }]] }]
                },
              ]
            }
          ];

          const operations = [c1.query('COMMIT'), c2.query('COMMIT'), db.selectInClient(c2)];
          let orderedOperations = [];
          let readIdx = txnOrder[2];
          for (let i = 0; i < txnOrder.length; i++) {
            orderedOperations[txnOrder[i]] = operations[i];
          }

          const results = await Promise.allSettled(orderedOperations);
          const modelResults = model.select(2, 'txn_iso');
          const read = results[readIdx];

          if (read.status === 'fulfilled') {
            // Check that DB state matches ANY model state
            expect(modelResults).toContainEqual(read.value);
          } else {
            fail('Read failed');
          }
      }).afterEach(async () => {
        await db.update(initialVal);

        c1.release();
        c2.release();

        await db.pool.end();
      }),
      { endOnFailure: true, numRuns: 100}
    )
  });
});

The race condition is explicitly modeled here in how we initialize model.states. Zooming in, the second state shows the state of the world after the write transaction has been committed: This manifests as the new written value (val) appearing in the committed relations state, and there only being one open transaction which hasn’t modified any data:

{
  relations: [{ name: 'txn_iso', data: [[{ column: 'ival', value: val }]] }],
  transactions: [
    {
      id: 2,
      isDirty: false,
      prev: [{ name: 'txn_iso', data: [[{ column: 'ival', value: initialVal }]] }],
      next: [{ name: 'txn_iso', data: [[{ column: 'ival', value: initialVal }]] }]
    },
  ]
}

The other state has both transactions open, and the new value has not yet been written. Running this test passes against the test PG instance. We’ve accurately modeled the nondeterminism.

This is great, but doesn’t help us in our application tests. To do that, we need to pick which value is correct. Because we know that select returns one result set for each nondeterministic initial state its configured with, we can accept a prophecy variable that picks a single one:

class DBModelProphecy {
    modelNondet: DBModelNondet = new DBModelNondet();

    select(txnId: number, relation: string, initialStateProphecy: number): Tuple[] {
        return this.modelNondet.select(txnId, relation)[initialStateProphecy];
    }
}

This allows a test to use the nondeterministic model in deterministic “mode,” which will make sure that the application either handles both cases correctly, or leads to an implementation change.

In Closing

Nondeterminism has been a major thorn in my side when writing model-based tests for real applications. I think prophecy variables as presented here provide a clear pattern for dealing with it. There’s a lot more to build out to have a production-grade model of a database like Postgres, but it’s encouraging to see that the idea does work in principle. It’s also really nice that the same technique can be applied to testing timeouts all the way to testing transaction isolation levels.

This all started from talking about the difficulty of property-based testing nondeterministic dependencies on lobste.rs with Stevan, the author of The sad state of propery-based testing libraries. I appreciate their views on the topic, you should read that post as well.

Does Your Test Suite Account For Weak Transaction Isolation?

2023-12-31T00:00:00+00:00

Transaction isolation is the kind of thing that you learn about and it fills you with fear. Specifically, there are weak transaction isolation levels which allow some fairly unexpected behavior. Tools like Jepsen are used to test the general isolation guarantees of databases, but it’s pretty uncommon to check the application layer for issues related to isolation anomalies. These anomalies can impact actual domain logic, so it’s important to understand them as well as how we can test them.

What is Weak Transaction Isolation?

Transaction isolation means that concurrent transactions against a database will be independent of one another. It’s the “I” in ACID. Unfortunately, “independence” in this context is a spectrum, and there are actually different isolation levels that are supported, each with subtly different behavior.

Here’s a quick example script which makes concurrent queries against a database (Postgres):

create table txn_iso (ival int);
insert into txn_iso (ival) values(1);

import { Pool, Transaction } from 'https://deno.land/x/postgres/mod.ts';

const pool = new Pool({
  user: 'postgres',
  hostname: 'localhost',
  database: 'postgres',
  port: 5433,
  password: 'test1234',
}, 10);

async function runQuery(
  txn: Transaction,
  query: string,
  args: (string | number)[],
  beforeMsg: string,
  afterMsg: (result: any) => string
) {
  console.log(beforeMsg);
  const result = await txn.queryObject(query, args);
  console.log(afterMsg(result));
}

async function readTransaction() {
  const query = 'select ival from txn_iso';
  const printResult = (result: any) => `Read result: ${result.rows[0].ival}`;

  const client = await pool.connect();
  const txn = client.createTransaction();

  await txn.begin();

  await runQuery(txn, query, [], 'Executing first read...',  printResult);

  // Wait for concurrent write to occur
  await new Promise(resolve => setTimeout(resolve, 2000));

  await runQuery(txn, query, [], 'Executing second read...', printResult);

  await txn.commit();

  await client.release();
}

async function writeTransaction() {
  await new Promise(resolve => setTimeout(resolve, 1000));
  const client = await pool.connect();
  const txn = client.createTransaction();

  await txn.begin();

  const updateVal = Math.floor(Math.random() * 1000);
  const updateMsgBefore = `Updating ival to ${updateVal}...`;
  const query = 'update txn_iso set ival = $1';
  await runQuery(txn, query, [updateVal], updateMsgBefore, () => 'ival updated');

  await txn.commit();

  await client.release();
}

await Promise.allSettled([readTransaction(), writeTransaction()]);

This script executes two transactions concurrently: one that reads the txn_iso.ival column two different times, and another which modifies the value of that column. There’s some sleeps sprinkled in so that the second read occurs after the write. The question is: do both reads return the same value?

In Postgres, with the default transaction isolation level set, the answer is surprisingly no. This is an example output of running the script:

Executing first read...
Read result: 839
Updating ival to 79...
ival updated
Executing second read...
Read result: 79

The first read will return the value of the column at the time that the read transaction begins, but the second read will return the value that was updated by the concurrent write transaction. That’s because the default level is Read Committed, which allows non-repeatable reads. A non-repeatable read means that in the span of the same transaction, queries to the same column may return different results! This isn’t unique to Postgres either - Read Committed is the default isolation level in Oracle and SQL Server as well¹.

This is surprising to a lot of people, and rightfully so, since it seems to go against the very definition of what a transaction is. But that’s because Read Committed is a weak transaction isolation level. Weak isolation means that transactions aren’t truly independent from one another, and the effects of one concurrent transaction can be seen in another. There’s 4 isolation levels defined by the ANSI SQL standard². All but Serializable, which is the strictest, are weak and allow some kind of interference between transactions.

Hopefully it’s clear why this is an issue. If you have an important column value, say a user’s account balance, you might query multiple different values in the same transaction which will surely result in a domain logic bug. Will our test suites catch such bugs? That depends how we set up the tests.

Simulating Concurrent Connections

The difficulty with coming up with tests that expose transaction isolation anomalies is that the test has to simulate multiple concurrent connections. Test cases almost always have the implicit assumption that they’re being executed by a single user, and isolation anomalies don’t show up in that scenario.

As an example, here’s some oversimplified code for making outbound transfers from an account with overdraft protection:

interface BalanceRepository {
  getBalance(txn: Transaction): Promise<number>;
  updateBalance(txn: Transaction, amount: number): Promise<void>;
}

async function checkOverdraftProtection(txn: Transaction, balanceRepo: BalanceRepository, amount: number) {
  const balance =  await balanceRepo.getBalance(txn);
  if (balance >= amount) {
    return;
  }

  balanceRepo.updateBalance(txn, balance + 100);
}

async function applyFundTransfer(txn: Transaction, balanceRepo: BalanceRepository, amount: number) {
  const balance = await balanceRepo.getBalance(txn);
  if (balance < amount) {
    console.error("Insufficient funds");
    return;
  }

  await balanceRepo.updateBalance(txn, balance - amount);
}

async function transferFunds(balanceRepo: BalanceRepository, amount: number) {
  const client = await pool.connect();
  const txn = client.createTransaction();

  await txn.begin();

  await checkOverdraftProtection(txn, balanceRepo, amount);
  await applyFundTransfer(txn, balanceRepo, amount);
  
  await txn.commit();

  await client.release();
}

const protectedBalanceRepo = {
  balance: 90,
  async getBalance(txn: Transaction) {
    return this.balance;
  },
  async updateBalance(txn: Transaction, amount: number) {
    this.balance = amount;
  }
}

await transferFunds(protectedBalanceRepo, 100);
console.assert(protectedBalanceRepo.balance === 90);

The main logic that we want to test is that overdraft protection adds additional funds when there’s not enough to cover a transfer, and that the final balance is correct. To test this, we’re placing all queries behind a BalanceRepository interface and creating a protectedBalanceRepo which starts out with insufficient funds but updates the balance based on overdraft protection.

This is the operation from the perspective of a single user and thus a single DB connection, so the insufficient funds error won’t get hit. As we saw with the Read Committed example though, another concurrent transaction can affect a value that’s read multiple times. So one way to simulate a concurrent transaction is to simply ignore the overdraft protection and specify a different balance result directly.

const concurrentWriteRepo = {
  currBalance: 0,
  balances: [100, 90],
  async getBalance(txn: Transaction) {
    return this.balances[this.currBalance++];
  },
  async updateBalance(txn: Transaction, amount: number) {
  }
}

...

await transferFunds(concurrentWriteRepo, 100);

This test double sets up two different balance results: it’ll first return 100, which will bypass overdraft protection, but the next balance check will return 90 which will result in an insufficient funds error. One way this would be possible in real life is if multiple people have access to the same account and initiate a transfer in close proximity to one another.

There’s a simple fix for this failure: just don’t read the balance multiple times, and instead pass in the sampled balance to any function that needs it:

async function transferFunds(balanceRepo: BalanceRepository, amount: number) {
  const client = await pool.connect();
  const txn = client.createTransaction();

  await txn.begin();

  const balance = await balanceRepo.getBalance(txn);
  const protectedBalance = await checkOverdraftProtection(txn, balance, balanceRepo, amount);
  await applyFundTransfer(txn, protectedBalance, balanceRepo, amount);
  
  await txn.commit();

  await client.release();
}

Now the logic of checkOverdraftProtection and applyFundTransfer can get changed to use a balance value instead of querying it. This also means that checkOverdraftProtection has to return the balance after protection is applied since applyFundTransfer used to get this value with the second balance query, and using the pre-protection balance will result in an insufficient funds error.

This solves the repeatable read anomaly by avoiding multiple reads, but there’s still a major issue: there’s a race condition between multiple concurrent transactions that can result in an incorrect balance.

Race Conditions and Serializability

To show the error we can execute two fund transfers concurrently against the actual DB, and we can introduce a write delay so that we can control which one writes last (we’d see errors even without this, but this reduces the non-determinism):

create table accounts (balance int);
insert into accounts (balance) values (100);

async function transferFunds(balanceRepo: BalanceRepository, amount: number, writeDelay?: number) {
  const client = await pool.connect();
  const txn = client.createTransaction();

  await txn.begin();

  const balance = await balanceRepo.getBalance(txn);

  if (writeDelay) {
    await new Promise(resolve => setTimeout(resolve, writeDelay));
  }

  const protectedBalance = await checkOverdraftProtection(txn, balance, balanceRepo, amount);
  await applyFundTransfer(txn, protectedBalance, balanceRepo, amount);
  
  await txn.commit();

  await client.release();
}

const postgresBalanceRepo = {
  async getBalance(txn: Transaction): Promise<number> {
    const result = await txn.queryObject('select balance from accounts');
    return result.rows[0].balance;
  },
  async updateBalance(txn: Transaction, amount: number) {
    await txn.queryObject('update accounts set balance = $1', [amount]);
  }
}

async function runInTransaction<T>(f: (txn: Transaction) => Promise<T>) {
  const conn = await pool.connect()
  const txn = conn.createTransaction()
  await txn.begin();

  const result = await f(txn);

  await txn.commit();
  await conn.release();

  return result
}

// Setup
await runInTransaction((txn) => {
  return postgresBalanceRepo.updateBalance(txn, 100);
});

// Run two concurrent fund transfers
await Promise.allSettled([transferFunds(postgresBalanceRepo, 80, 2000), transferFunds(postgresBalanceRepo, 60)]);

const balance = await runInTransaction((txn) => {
  return postgresBalanceRepo.getBalance(txn)
});

console.assert(balance === 60);

When this is run, the transfers will see the same initial balance (100), but the one with the write delay will overwrite the balance set in the other one. This also means that neither transfer will trigger overdraft protection, there will be no insufficient funds error, and the resulting balance will be an incorrect value of 20.

This is a serialization anomaly. The Postgres docs define this as:

The result of successfully committing a group of transactions is inconsistent with all possible orderings of running those transactions one at a time.

There are only two possible orderings of the two fund transfers here:

Transfer 80, then transfer 60
Transfer 60, then transfer 80

Since the starting balance is 100, both of these cases should trigger overdraft protection on the second transfer, and the resulting balance in both cases should be 60 (100 + 100 - 80 - 60). Race conditions can exist when transactions don’t adhere to serializability, and that’s what’s going on here - two fund transfers are initiated, but only one is accounted for because of a concurrent race. This is known as the “lost update” problem.

There’s a few different ways to fix this, but the simplest is to lock the row for the duration of the transaction with FOR UPDATE:

const postgresBalanceRepo = {
  async getBalance(txn: Transaction): Promise<number> {
    const result = await txn.queryObject('select balance from accounts FOR UPDATE');
    return result.rows[0].balance;
  },
  ...
}

Again from the Postgres docs:

FOR UPDATE causes the rows retrieved by the SELECT statement to be locked as though for update. This prevents them from being locked, modified or deleted by other transactions until the current transaction ends.

This means that each transaction will grab a lock on the accounts row and that will block all other transactions from modifying that row until it’s complete, i.e. the transactions will execute in a serializable fashion. It’s worth noting that this is now slower. Without the lock the transactions could truly operate concurrently, but now they have to wait in contention over balance updates to the same account. This is necessary for correct behavior, but it’s worth understanding the tradeoff.

Also of note, test doubles alone won’t help with this bug because the fix is in the real Postgres repository implementation. Test doubles are useful for testing application code independent of the database in many cases, but transaction isolation is a case where the different levels are so subtly different that it’s simpler to test against the real thing.

Transaction isolation has a major impact on an application, both in terms of performance as well as its influence on domain logic. It’s important to integration test against the real database to make sure that a weak transaction isolation level isn’t the cause of concurrency bugs. To expose such bugs, we have to execute at least two concurrent transactions in a test case. Unfortunately this can require some amount of time-based coordination which is never ideal, but is often necessary when tools like databases have opaque non-deterministic behavior that’s out of our control.

Still, it is something we can and should test for at the application level.

Interestingly, MySQL’s default isolation level is Repeatable Read, which avoids the bug presented here. Still, it’s very uncommon for any DB to have a default level of Serializable, so most databases are operating with weak isolation. A notable exception is FoundationDB, which does default to Serializable. ↩
Postgres actually only implements 3 out of 4, because Read Uncommitted conflicts with Postgres’ MVCC implementation. For more detail on isolation levels as Postgres implements them, see: https://www.postgresql.org/docs/current/transaction-iso.html ↩

Forward and Backward Reasoning in Proof Assistants

2023-10-01T00:00:00+00:00

Proof assistants are really fascinating tools, but the learning curve can be extremely steep. If you’re a programmer by trade and not a mathematician, this curve can be even steeper, because it’s not like programmers are doling out proofs left and right at work. One particular sticking point that I had trouble overcoming is the difference between forward vs. backward reasoning - proofs assistants support both.

Forward Reasoning

When thinking about logic, we generally think about forward arguments which get built up from one statement to the next, in sequence. For example, let’s make a logical argument about monitoring. We want to get an alert when our app goes down, and one way we know that the app is down is when a test user can’t login and see the home page. The way to express that in logic is to say that the home page not loading implies that the app is down:

\[HomePageDoesntLoad \implies AppIsDown\]

Implication is a useful thing to know, but it only tells us about the overall relationship and doesn’t tell us whether the app is down right now or not. We want to know the current state so we can determine if we should page someone, and for that we can use one of the oldest rules in all of logic: modus ponens.

Modus ponens is also known as “implication elimination,” which more accurately describes its behavior. It allows us to infer something about an implication, but the conclusion no longer contains one - the implication gets eliminated:

\[\dfrac{P~~~~~~~P \implies Q}{ Q }\]

This is written out as an inference rule, which in this case means that if we know P is true, and we know that P implies Q, then we can infer that Q is also true. On top of the bar are the premises, and on the bottom is the conclusion which we can infer if the premises are true. The reason that this rule is so old is that it’s just a formal description of common sense - if P implies Q, and we know P is true, of course Q is true. That’s what implies means.

In our monitoring context, we can take P to be “the home page doesn’t load” and Q to be “the app is down,” and by this rule we can conclude that the app is down if we actually observe the home page being unable to load. This is a forward argument - when an inference rule is taken from top to bottom.

Proof assistants almost always support forward reasoning. One way to do this in Isabelle is with the frule tactic:

lemma 
  assumes HomePageDoesntLoad 
    and "HomePageDoesntLoad ⟶ AppIsDown"
  shows "AppIsDown"
  using assms
  by (frule_tac P=HomePageDoesntLoad and Q="AppIsDown" in mp)

mp is the rule for modus ponens, which is defined like this¹:

lemma 
  assumes "P ⟶ Q"
    and P
  shows Q
  ...

frule_tac allows us to take a forward logical step if the premises are shown to be true. Since they’re assumed here, they are true, and we prove AppIsDown in one step.

Backward Reasoning

Proof assistants also allow us to work backwards from a goal.

Let’s take a look at a backward proof of our lemma:

lemma 
  assumes hp_load: HomePageDoesntLoad 
    and imp_appdown: "HomePageDoesntLoad ⟶ AppIsDown"
  shows "AppIsDown"
  apply(rule_tac P=HomePageDoesntLoad and Q=AppIsDown in mp)
  using imp_appdown
    apply(assumption)
  using hp_load
    apply(assumption)
  done

Here, instead of frule_tac, we use rule_tac, which applies a rule in a backward fashion. Instead of going from top to bottom in the rule, we replace the current proof goal with the premises in the top of the rule. This allows us to prove each one separately, which is one of the main benefits of backward rule application: we can more easily divide and conquer a complicated proof.

It works because an inference rule can be interpreted in two ways. As we said, the forward interpretation is: “we can conclude the bottom if the top premises are true.” The backward interpretation is: “to prove the bottom, it suffices to prove the top premises.” These are logically equivalent.

To dive in a bit more, we can look at the proof state after each step in the proof above. At the beginning of the proof, the goal is simply the final conclusion we want to show:

lemma 
  assumes hp_load: HomePageDoesntLoad 
    and imp_appdown: "HomePageDoesntLoad ⟶ AppIsDown"
  shows "AppIsDown"

goal (1 subgoal):
 1. AppIsDown

Now we apply modus ponens backwards:

apply(rule_tac P=HomePageDoesntLoad and Q=AppIsDown in mp)

goal (2 subgoals):
 1. HomePageDoesntLoad ⟶ AppIsDown
 2. HomePageDoesntLoad

Instead of having to show AppIsDown directly, we now just have to show that HomePageDoesntLoad ⟶ AppIsDown and HomePageDoesntLoad. In a real proof, we’d have to figure out how to prove these independently, but here both of these are true by assumption so the rest of the proof just pulls in the appropriate one and applies it.

Which One’s Better?

The unfortunate answer is that there’s no preferred direction, and we’ll often want to use both. We can also use higher-level and more powerful tactics anyway, which abstract the underlying reasoning. This monitoring example is very trivial, and can be proven in Isabelle with a variety of one liners, like:

lemma 
  assumes HomePageDoesntLoad 
    and "HomePageDoesntLoad ⟶ AppIsDown"
  shows "AppIsDown"
  by (auto simp: assms)

Backward reasoning seems more natural many times, but this is likely because of the history of proof assistants: they were pretty much designed around backward reasoning and interactivity from the start. The line gets blurred with more recent developments like Isar, which is an Isabelle sublanguage for defining structured proofs. In Isar, individual steps might be proven in a backwards fashion, but the proof proceeds in a structured and forward manner. Isar proofs are almost always preferred because they more closely resemble pen-and-paper proofs, and bring the very relevant intermediate proof state to the foreground.

Here’s one for the monitoring example:

lemma 
  assumes imp_appdown: "HomePageDoesntLoad ⟶ AppIsDown"
    and hp_load: HomePageDoesntLoad
  shows "AppIsDown"
proof (rule mp[where P=HomePageDoesntLoad and Q=AppIsDown])
  from imp_appdown show "HomePageDoesntLoad ⟶ AppIsDown" by assumption
  from hp_load show HomePageDoesntLoad by assumption
qed

This pretty closely mirrors the backward proof from before, and that’s because the structure of the proof is based on the backward application of mp by choosing rule and not frule in the proof command. But now the intermediate goals are visible, which gives the proof more structure. This is especially helpful for more complicated goals that can’t be proven in a single step because each goal can be respectively built up via intermediate steps.

All this to say: the logical direction often changes throughout a proof in a proof assistant, and the same rules can be used both forwards and backwards. Knowing which direction is being used is crucial for understanding our proofs.

It’s actually defined as an axiom, which means it’s implicitly taken to be true, and it also uses the older-style Isabelle syntax which lists assumptions in brackets: "⟦P ⟶ Q; P⟧ ⟹ Q". But this is equivalent to the assumes ... shows ... syntax being used here. ↩

Compiling a Test Suite

2023-08-23T00:00:00+00:00

When I first stumbled upon certifying compilation¹, I was absolutely awestruck. I thought a compiler was a very specific thing, a translator from source to target language. But a certifying compiler goes further: it also proves its own correctness. My motto has become “most tests should be generated”, so this immediately seemed like a promising approach to my goal of improving the generative testing of interactive applications. It wasn’t immediately clear how exactly to incorporate this into that context, but after a little experimentation I now have a prototype of what it might look like.

First, rather than describe the theory, let me show you what the workflow of certifying compilation looks like. Imagine invoking a command like this:

certc source.cc -o myprogram -p proof

certc compiles the source file into an executable, like every other compiler, but in addition it outputs this proof file. Imagine that you can open up this file, and from its contents be convinced that the compilation run contained zero bugs, and the output myprogram is a perfect translation of source.cc². The compilation run is certified by this proof. Such compilers are sometimes referred to as self-certifying for this reason - they produce their own proof of correctness.

We know that proofs are hard though, and for most of us tests are sufficient. So what if instead, we had this workflow:

certc source.cc -o myprogram -t test
./test

Instead of generating a proof, we now generate a test suite, and instead of opening it up to inspect it, we run it. If it passes, we’re still convinced that the compilation run was correct. Visually, certifying compilation just adds one more output artifact to a compilation run, which we can call a “checker,” and looks something like this:

From Programs to Applications

At this point, this doesn’t look very applicable to something like a web application, and I’m mostly interested in testing interactive distributed applications. The idea of compiling a source model into a full-fledged web app is farfetched to say the least. I actually tried going down that path for a bit, and I can confirm: it is hard. It’s definitely an interesting research area, but for now let me pitch an alternative workflow that’s still based on the mental model of certifying compilation.

What if we assume that our target application is something that we hand-modify out of band, and we just generate the checker for it, i.e.:

certc model -c test
./test

And visually:

In this workflow, we hand-develop the implementation application as we do normally, but we still generate the checker from a model. This puts us under the umbrella of model-based testing, but we’re going to look at the proof techniques that a certifying compiler uses as inspiration for how we should generate the correctness tests. Because of this difference, I’d call this paradigm “certifying specification.”

What’s nice about this is that it slots right in to existing workflows. We can even TDD with this if we’re so inclined, by first changing logic in the model and then generating the failing tests before implementing them. Workflow-wise, it’s simple enough to work.

Writing a Model

Since the checker generation depends on the existence of a model, we should first talk about how to write one. The first question to ask is: should we use an existing language or a new language to write models in? I really try to avoid thinking about or suggesting the introduction of new languages into the ecosystem. But, the question has to be asked, because using an existing language has a lot of tradeoffs with respect to specification:

Existing languages have no notion of system structure, i.e. how do we distinguish system state vs. local variables? How do we distinguish system actions vs. local mutation? How do we parse an arbitrary program and get relevant information out of it to help with test generation?
Programming languages are meant for programming. There are aspects of specification that require other language features, such as the ability to express logical properties and the ability to control aspects of test generation.
Programming languages have additional features that aren’t necessary in a modeling context. For example, a model has no need for filesystem operations or networking.

These can be overcome by creating an embedded DSL within an existing language to restrict the structure of models, but embedded DSLs have their own set of tradeoffs³.

One other option is to use an existing specification language, like TLA+. TLA+ in particular is too powerful for us here - we really want to limit models to be executable so that we can use their logic in the checker.

I think these are all viable approaches, but I also think that there are enough reasons to create a language that’s purpose-built for this use case. I’ve been experimenting with one that I call Sligh. Here’s a model of a counter application in Sligh:

record Counter:
  name: Id(String)
  value: Int
end

process CounterApp:
  counters: Set(Counter)
  favorites: Set(String)

  def GetCounters():
    counters
  end

  def CreateCounter(name: String):
    counters := counters.append(Counter.new(name, 0))
  end

  def Increment(name: String):
    def findCounter(counter: Counter):
      counter.name.equals(name)
    end

    def updateCounter(counter: Counter):
      Counter.new(counter.name, counter.value + 1)
    end

    counters := counters.update(findCounter, updateCounter)
  end

  def AddFavorite(name: String):
    favorites := favorites.append(name)
  end

  def DeleteFavorite(name: String):
    def findFavorite(favName: String):
      name.equals(favName)
    end

    favorites := favorites.delete(findFavorite)
  end
end

Sligh is not meant to be revolutionary in any way at the language level (in fact it aims to be much simpler than the average general purpose language), and hopefully the functionality is clear here. The main goal is that it supports enough analysis so that we can generate our model-based tests. The main notable syntactic features are the := operator and the structure of the process definition. The := operator denotes updates of the system state, distinguished from any modification of local variables. The CounterApp app has a set of counters and a set of favorites as system state. Local variables exist, but mutations to those are implementation details and don’t matter from the perspective of testing. Having a specific operator for the system state allows simple syntactic analysis to find state changes, which is essential for generating the certification test.

For example, in the Increment action, we know that the counters state variable is modified, and in the AddFavorite action the favorites state variable is modified. If no assignments occur on a state variable in the span of an action, then we know for sure that it’s not modified in that action. This becomes very important later when we can exploit this to generate the minimum amount of test data necessary for a given test iteration.

Sligh processes also support nested defs which define system actions. System actions are the atomic ways that the system state can change, like adding or incrementing counters. For those conceptual user operations, we have corresponding CreateCounter, and Increment actions. This is what Sligh uses to determine which operations to generate tests for.

These syntactic restrictions lead to a very powerful semantic model of a system that’s also statically analyzable - they effectively form a DSL for describing state machines.

Compiling the Test Suite

A Sligh model doesn’t get compiled into a test suite directly. To compile the above counter model, we’d run:

sligh counter.sl -w witness

which generates a “witness” file. This is a good time to talk a bit about the compiler internals and why that is.

It’s common for certifying compilers to decouple per-program generated output from a separate checker⁴ that’s written once. This makes the code generation phase of the compiler simpler, but also allows the checker to be written and audited independently. This is extra important since the checker is our definition of correctness for the whole application, and a misstatement there affects the guarantees our certification test gives us.

Here’s the current checker that’s in use:

export function makeTest(
    actionName: string,
    stateType: "read" | "write",
    stateGen: any,
    implSetup: any,
    dbSetup: any,
    model: any,
    modelArg: any,
    clientModelArg: any,
    runImpl: any,
    expectations: any,
  ) {
    test(`Test local action simulation: ${actionName}`, async () => {
      let impl: StoreApi<ClientState>;
  
      await fc.assert(fc.asyncProperty(stateGen, async (state) => {
        impl = makeStore();        
  
        const clientState = implSetup(state);

        // Initialize client state
        impl.setState(clientState);

        // Initialize DB state
        await impl.getState().setDBState(dbSetup(state));

        // Run implementation action
        await runImpl(impl.getState(), state);

        // Run model action and assert
        switch (stateType) {
          case "write": {
            const clientModelResult = model(clientModelArg(state));
            for (const expectation of expectations) {
              const { modelExpectation, implExpectation } = expectation(clientModelResult, impl.getState());
    
              expect(implExpectation).toEqual(modelExpectation);
            }
            break;
          }
          case "read": {
            let modelResult = model(modelArg(state));
            for (const expectation of expectations) {
              const { modelExpectation, implExpectation } = expectation(modelResult, impl.getState());
    
              expect(implExpectation).toEqual(modelExpectation);
            }
            break;
          }
        }
      }).afterEach(async () => {
        // Cleanup DB state
        await impl.getState().teardownDBState();
      }), { endOnFailure: true, numRuns: 25 });
    });
  }

This looks similar to other model-based tests we’ve built before in that it compares the output of the model and implementation for a given action at a given initial state. This test is parameterized though, and all of the input parameters for a given test come from the witness.

A “witness” in the certifying compilation world refers to data that’s extracted from the source program during compilation. Here’s the witness output for the CreateCounter action:

interface Counter {
  name: string;
  value: number;
}

interface CreateCounterDBState {
  counters: Array<Counter>;
}

interface CreateCounterType {
  counters: Array<Counter>;
  name: string;
  db: CreateCounterDBState;
}

interface CreateCounterModelIn {
  name: string;
  counters: Array<Counter>;
}

let CreateCounterModel = (params: CreateCounterModelIn) => {
  let name = params.name;
  let counters = params.counters;
  counters = (() => {
    let a = [...counters];
    a.push({ name: name, value: 0 });
    return a;
  })();
  return { counters: counters };
};

// ...

{
  name: "CreateCounter",
  type: "write",
  stateGen: fc.record({
    counters: fc.uniqueArray(
      fc.record({ name: fc.string(), value: fc.integer() }),
      {
        selector: (e: any) => {
          return e.name;
        },
      }
    ),
    name: fc.string(),
    db: fc.record({
      counters: fc.uniqueArray(
        fc.record({ name: fc.string(), value: fc.integer() }),
        {
          selector: (e: any) => {
            return e.name;
          },
        }
      ),
    }),
  }),
  implSetup: (state: CreateCounterType) => {
    return { counters: state.counters };
  },
  dbSetup: (state: CreateCounterType) => {
    return { counters: state.db.counters, name: state.name };
  },
  model: CreateCounterModel,
  modelArg: (state: CreateCounterType) => {
    return { counters: state.db.counters, name: state.name };
  },
  clientModelArg: (state: CreateCounterType) => {
    return { counters: state.counters, name: state.name };
  },
  runImpl: (impl: ClientState, state: CreateCounterType) => {
    return impl.CreateCounter(state.name);
  },
  expectations: [
    (modelResult: CreateCounterModelOut, implState: ClientState) => {
      return {
        modelExpectation: { counters: modelResult.counters },
        implExpectation: { counters: implState.counters },
      };
    },
  ],
},

// ...

The details here are likely to change over time, but the key thing to notice is that all of this information is generated from the definition of CreateCounter in the model. Here’s the CreateCounter definition again for reference:

def CreateCounter(name: String):
  counters := counters.append(Counter.new(name, 0))
end

This action takes a name string as input, but it also modifies the counters state variable (which Sligh is able to detect because of the presence of the := operator). From this, one of the things we generate is a type for all of the test’s input data, CreateCounterType:

interface CreateCounterType {
  counters: Array<Counter>;
  name: string;
  db: CreateCounterDBState;
}

And the stateGen property of the witness object gets a corresponding data generator for this type:

fc.record({
  counters: fc.uniqueArray(
    fc.record({ name: fc.string(), value: fc.integer() }),
    {
      selector: (e: any) => {
        return e.name;
      },
    }
  ),
  name: fc.string(),
  db: fc.record({
    counters: fc.uniqueArray(
      fc.record({ name: fc.string(), value: fc.integer() }),
      {
        selector: (e: any) => {
          return e.name;
        },
      }
    ),
  }),
})

Also, note what this excludes. The test doesn’t have to generate the favorites variable since it’s not referenced or modified in the span of this particular action. The test for each action only has to generate the bare minimum amount of data it needs to function. And most importantly, this means we totally avoid creating any global system states. I think this will be the key to testing a larger application in this way.

Other than that, the other params are similarly extracted from the CreateCounter signature and code, providing overall assistance to the checker. I expect to be able to hone these witness definitions over time, but this works for now.

At this point it should be apparent that the compiler and checker both have to know about some very important system details. They need to know what language the test is written in. They need to know the pattern for executing actions on both the implementation and the model (here the implementation interface is a Zustand store meant to be embedded in a React app). They need to know what testing libraries are being used - here we’re using vitest and fast-check. And they need to be able to set up the state of external dependencies like the database, done here with calls to impl.getState().setDBState and impl.getState().teardownDBState(), which means that the server has to be able to help out with initializing data states.

Still, lots of the functionality is independent of these concerns, and my hope is to make the compiler extensible to different infrastructure and architectures via compiler backends. For now, sticking with this single architecture has supported the development of the prototype of this workflow.

Finally, the test gets wired up together in a single file runnable by the test runner:

import { makeTest } from './maketest';
import { witness } from './witness';

for (const testCase of witness) {
    makeTest(
      testCase.name, 
      testCase.type as "read" | "write",
      testCase.stateGen, 
      testCase.implSetup, 
      testCase.dbSetup, 
      testCase.model, 
      testCase.modelArg, 
      testCase.clientModelArg,
      testCase.runImpl, 
      testCase.expectations
    );
  }

Outro

Ok, I went into a lot of details about the internals of the Sligh compiler. But to reiterate, the developer workflow is just:

sligh counter.sl -w witness
./test

I’m using this on a working Next.js application, and workflow-wise it feels great. I’m excited to see what other challenges come up as the application grows.

I can’t rightfully end the post without talking about a few tradeoffs. I can probably write a whole separate post about that, since this one is already quite long, but two big ones are worth mentioning now. First, because we’re testing single state transitions, a test failure won’t tell you how to actually reproduce the failure. It might take a series of very particular action invocations to arrive at the starting state of the simulation test, and it’s not always clear if the specific state is likely or even legitimately possible in regular application usage. I have ideas there - similar to property-based testing failure minimization, it should be possible to search for action sequences that result in the failing initial state.

The second tradeoff is that data generation for property tests of a full application is non-trivial. Sligh is currently doing the bare minimum here, which is use type definitions to create data generators. I’m hoping the language can help out here though, and more intelligent generators might be able to be extracted from the model logic.

And lastly, I have to call out the awesome Cogent project one last time. So many of these ideas were inspired by the many publications from that project. Specifically, check out this paper: The Cogent Case for Property-Based Testing.

I first heard about certifying compilation through a talk on YouTube and a corresponding paper (by Liam O’Connor, Zilin Chen, Christine Rizkallah, Sidney Amani, Japheth Lim, Toby Murray, Yutaka Nagashima, Thomas Sewell, and Gerwin Klein). These are about the Cogent language, which compiles from itself to C, but also generates a proof of its correctness in Isabelle. ↩
Any compiler-writer will tell you, compilers are just as buggy as other programs. This is why certifying compilation exists in the first place - to provide higher assurance about the correctness of a compiler. ↩
I once read an interesting take about building embedded DSLs inside of an existing language that influenced my thinking here. The takeaway: eDSLs are often not worth it. ↩
Certifying Algorithms by R. M. McConnella, K. Mehlhornb, S. Näherc, P. Schweitzer ↩

Most Tests Should Be Generated

2023-07-02T00:00:00+00:00

Traditional testing wisdom eventually invokes the test pyramid, which is a guide to the proportion of tests to write along the isolation / integration spectrum. There’s an eternal debate about what the best proportion should be at each level, but interestingly it’s always presented with the assumption that test cases are hand-written. We should also think about test generation as a dimension, and if I were to draw a pyramid about it I’d place generated tests on the bottom and hand-written scenarios on top, i.e. most tests should be generated.

Correctness is What We Want

What are we even trying to do with testing? The end goal is to show correctness. We do this for two main reasons: to show that new functionality does what’s expected before release, and to ensure that existing functionality is not broken between releases. Tests are a means to this end, nothing more. Importantly, they also can only ever show approximate correctness. To understand that fully, let’s define correctness precisely. Here’s a paraphrasing of Kedar Namjoshi’s definition from Designing a Self-Certifying Compiler.

First we have to define what a program is. The simplest representation is just a function from values in X to values in Y. This may look oversimplified, but an interactive program can even be modeled this way by assuming the program function is invoked in response to each user interaction in a loop. So a program P is:

\[P: X \rightarrow Y\]

Correctness requires a specification to check against. This might be surprising, since one rarely exists, but think of traditional test suites as simply defining this specification point-wise. A specification S can be a function of the same type:

\[S: X \rightarrow Y\]

We can express correctness with the following property:

\[\forall x \in X: P(x) = S(x)\]

In English: for every x value in X, evaluating P(x) yields the same value as evaluating S(x).

Point being, we want to check that the implementation program does the same thing as the specification, always. Notice how achieving 100% branch coverage in a test suite doesn’t get us here by the way, since that doesn’t account for all inputs in X.

Let’s look at how scenarios and generated tests differ with how they show correctness.

Testing for Correctness with Scenarios

As I mentioned, the traditional test pyramid is talking about hand-written test scenarios, aka examples / test cases etc. Correctness is pretty simple to express as a logical property, but it’s very difficult to test for. The first thing we run into is the test oracle problem - how do we actually get the value of S(x) to check against? Executable specifications rarely exist (though I am a proponent of using them for this reason), so normally what happens is that the test writer interprets an informal specification and hard codes the expected value of S(x) for a specific x as the test assertion. The informal specification is what the team talks about when deciding to build the feature, and the test writer is the test oracle. Sometimes some details are written down, sometimes not, but the burden of coming up with the expected test value is always on the test writer, and it’s a completely manual process.

The next issue is the number of values in the input domain X. Each test case needs to specify a single input value from X, but testing for all values from X is not feasible in any way. This is not an exaggeration - if X is the set of single 64-bit integers, we’d have to check 18,446,744,073,709,551,616 test cases. This multiplies for each additional integer, and how many integers do you think are in the entire state of a realistic program? We said earlier that a test suite only approximates correctness, but this makes it more formal. A test suite actually represents this property:

\[TX \subseteq X \land \forall tx \in TX: P(tx) = S(tx)\]

How effective a test suite is boils down to how confident we are that testing the input values that we chose implies that the correctness will hold for all of the input values, i.e.

\[\forall tx \in TX: P(tx) = S(tx) \implies \forall x \in X: P(x) = S(x)\]

This is probably true sometimes, but we have no guarantee of it in general. How can we ever know that the values that we pick out of X are “good enough”?

So test scenarios have an informal and manual test oracle process, and are pretty quantitatively incomplete in terms of how much of the input domain they can possibly cover. That doesn’t mean they’re not useful! Testing via scenarios is unreasonably effective in practice. There are two main benefits to them. First, they’re easy to write. This is likely because they require very literal and linear reasoning, since we just need to assert on the actual output of the program. If we really want, we can just run the program and observe what it outputs and record that as a test assertion. People do this all the time, and there’s even a strategy that takes this to the extreme called “golden testing” or “snapshot testing.”

The next benefit, somewhat obviously, is that they’re specific. If we have a test case in our head that we know is really important to check, why not just write it out? When we do this, we also get more local error messaging when the test fails, which can point us in a very specific direction. This is always cited as one of the main benefits of unit testing, and it really is helpful to have a specific area of the code to look at vs. trying to track down a weird error in a million lines of code.

Now let’s look at generated tests.

Generating Tests for Properties

Our correctness statement from earlier is expressed as a property: P(x) = S(x) is a property that’s either true or not for all of the program inputs. Now, we know that we can’t actually check every single input in a test, but what we can do is generate lots and lots of inputs and check if the property holds. With property-based testing these inputs are usually generated randomly, but there are other data generation strategies as well. So here, we’re talking about property-based testing more generally, and it has a couple of subtly different problems than testing with scenarios.

When checking for properties, the test oracle problem also presents itself immediately. We can always evaluate P(x), since that’s our implementation that we obviously control, but how do we know what the expected value S(x) is? And, furthermore, we have a chicken-and-egg problem: how do we know what S(x) is when code is generating x and we don’t know what it is ahead of time?

The answer is to define S(x) with logic that we can actually execute in the test, i.e. an executable specification. This often sounds weird to people at first, but looking at our correctness statement this is the more natural way to test. Instead of implicitly defining the specification via a bunch of individual test cases, we just define S(x) and call it during testing. This can take the form of simple functions that represent invariants of the code, all the way up to entire models of the functional behavior.

The input space issue is also still present with property-based testing, but in a different way: generating data is hard. Like, really hard. One of the main challenges is logical constraints, e.g. “this number must be less than 100”. These constraints can get very complicated in real-world domains, and sometimes that even leads to performance issues where you have to discard generated inputs until the constraint is met.

Property-based testing has an absolute killer feature though: it discovers failure cases for you, i.e. it actually finds unknown unknowns. This is worth more than gold. With scenarios, you have to know the failure ahead of time, but isn’t every bug in production a result of a failure that you didn’t even think of before deploying? Rather than check cases that we know ahead of time, we generate tests that search for interesting failures. This simply can’t be done with ahead-of-time test scenarios.

The Test Generation Pyramid

We looked at some of the pros and cons of scenarios vs. generated tests, so which should we prefer? I definitely think we should write both kinds, but overall most tests should be generated. Test strategies have to be represented as a triangle, so here is this idea in triangle form:

Why should we prefer generated tests? It all boils down to the fact that they find failures for us, which means that they naturally bring us closer to correctness. Unfortunately, no matter how perfect our selected test scenarios are they leave the vast majority of the input space uncovered, and there is no way to know which uncovered inputs are important and which are redundant. By having a suite of generated tests that are constantly looking for new inputs, we put ourselves in the best position to find edge cases that we just aren’t considering at the moment.

It’s like having a robot exploratory tester that we can deploy at will, which opens up a whole new mode of testing. We can run generated tests in CI before merging, sure, but we can also run them around the clock since generated tests search for failures vs. checking predetermined scenarios. More testing time means more of the input domain being searched, so to check more inputs we simply run each generated test for longer and run more test processes in parallel.

This doesn’t mean that we stop writing scenarios. That’s why there’s two sections in the pyramid. All of the proposed values of test scenarios are valid - we get specific error messages, free executable documentation, and a guarantee that important cases are checked. But generated tests are fundamentally stronger than scenarios, since the generated tests will often find the same inputs that we use in our scenarios in addition to ones we haven’t thought about.

Since the ultimate goal of testing is correctness, not documentation and local error messages, it’s in our best interest to supplement our scenarios with lots and lots of generated tests.

Logical Time and Deterministic Execution

2023-02-28T00:00:00+00:00

Recently, Tomorrow Corporation released this video of their in-house tech stack doing some truly awesome time-travel debugging of a production-quality game. You should watch this video, even if you don’t read this post, because the workflow that they’ve created is really inspiring. The creator kept bringing up the fact that the reason their tools can do this is that they have determinism baked into them at the very foundational levels. You simply can’t bolt this on at higher levels in the stack.

This got me thinking - not only do we rarely have this level of control in our projects, but I think it’s rare to even understand how determinism is possible in modern systems that are interactive, concurrent, and distributed. If we don’t understand this, we can’t ever move our tools toward determinism, which I think is a very good idea. It turns out that even if we can’t predict exactly how a program will execute in a specific run, we can still model and reason about it deterministically. This is a prerequisite for most formal methods, and while I understand that formal methods aren’t everyone’s cup of tea, this is the number one thing that I wish more people understood. So today, we won’t be talking about testing or verifying anything, we’ll just be looking to better understand software in general by diving into logical time and how it enables deterministic reasoning.

User Interaction and Non-Deterministic Choice

Talk of non-determinism can get very abstract very quickly, but there is a practical manifestation that we’ve all observed even if we didn’t know the term: non-deterministic choice. An application with a user interface is a classic example of a system with non-deterministic choice - no one can predict the order that a user will click through the interface, and the user is free to make any choice that’s visible and enabled.

We’ll introduce an example to get more specific, and it’s important to always use TodoMVC as the interactive application example¹ (here’s one of the implementations if you want to click around). In TodoMVC, we can add new named to-do items and then mark them as completed. We can also remove a to-do without marking it as completed. Like all interactive applications, we can do this in any order though, and these are all valid sequences of actions:

Add to-do named “t1”
Mark “t1” as completed

Add to-do named “t1”
Remove “t1”
Add to-do named “t2”
Mark “t1” as completed

Add to-do named “t1”
Mark “t1” as completed
Add to-do named “t2”
Mark “t2” as completed
Add to-do named “t3”
Add to-do named “t4”
Add to-do named “t5”
Remove “t3
Add to-do named “t6”
Add to-do named “t7”
Remove “t4”
Add to-do named “t8”
Add to-do named “t9”
Mark “t6” as completed

We can visualize this non-determinism with a state graph:

A non-deterministic choice exists when more than one transition arrow flows away from a given state. It means that all of them are valid choices that can occur in separate executions, but one has to somehow be chosen to proceed through the state graph. An interactive application lets the user decide via the UI, but as we’ll see later, there are other things that can make choices. Functionally, it doesn’t matter who does the choosing.

A quick aside: this is the complete behavior up to a bound of 2 to-dos. Physical space constraints aside, the full state graph of TodoMVC is theoretically infinite, because you can always add a to-do with a new name. Visualizing infinite bubbles is painful for everyone involved, so we place a constraint on the model along the lines of “there are only two to-dos in the entire universe.” This is a silly constraint, but it helps us visualize the state space in a manageable way. Bounded models also help with making properties checkable, but we’re not talking about that today because we’re not actually doing formal methods!

Let’s look at an example run through the program by picking specific choices. We’ll start at the gray initial state, add two to-dos named “t1” and “t2”, and then we’ll complete them both. Here’s that path in red:

We can get to the same final state a different way, by adding to-do “t2”, completing it, then adding to-do “t1” and completing it:

We all know how software works intuitively, but seeing these runs against the full state graph hints at a couple of precise definitions: software behavior is simply a sequence of states, and a program is a set of allowable behaviors. It also gives us our first step towards determinism. When a non-deterministic choice exists, we don’t know which path will be taken in a specific program run, but we do know what all of the possible runs are. Each of those runs is a totally deterministic behavior.

Said another way, a non-deterministic choice becomes deterministic when we pick one.

For fun, here’s the state graph of TodoMVC with 5 to-dos:

Determinism isn’t necessarily easy.

Concurrency

Concurrency is another notorious source of non-determinism, but let’s define why. Imagine we have N network requests that start in an idle state, begin fetching some data, and eventually complete. Continuing to keep our bounds small, let’s start with N = 2:

In every state, we can either initiate an uninitiated request or an in-progress request can complete. It’s possible for different requests to complete in different orders too, e.g. request 0 can complete first:

And request 1 can also complete first, even if request 0 was initiated before it:

The order that requests complete is a non-deterministic choice, which we’ve already seen, but there’s a major difference from the TodoMVC example: the OS or language runtime determines the choice, not a user. This is one reason why concurrency is a constant thorn in the side, and feels much more complex than the non-determinism of user interfaces. We literally don’t have control over the order of operations.

In the same way as the choices in the user interface, though, we just have to account for all of their combinations, and then we can know which orders of execution are possible. Another way to think about this is that if a race is possible, both sides of the race will always eventually occur, and we have to plan for both cases.

Because N = 2 is no fun, here’s N = 5 (i.e. 5 concurrent requests) which has 639 distinct states:

I’m sure a mutex will make this more manageable.

Logical Time, Time-Travel, and Beyond

Both state graphs show the set of all behaviors for the given system, and they do this by showing logical time, in contrast to physical time. A user might wait 17 years before selecting a transition in a UI, or an OS scheduler might pick one thread to execute while another waits for I/O. The real-world execution of a program runs in physical time, but our state graphs are only concerned with abstract states and transitions between them. And good thing for that - it would be awkward to have to wait 17 years to understand the possible behaviors of TodoMVC.

Beyond helping us understand the complete picture of all of the different interleavings of transitions, logical time is also what enables time-travel debugging. We can’t logically move through a system until it’s been properly decomposed into states and the steps between them. This in itself is a design space - how much of the system state do we store vs. derive? How much additional state do we add to make things possible like searching for states by timestamp?

All we need for logical time are states and transitions between them, i.e. logical time is inherently tied to state machines / transition systems. In fact, a time-travel debugger can pretty much be seen as a user interface for a state machine. But most importantly, this mental model allows us to have a totally deterministic view of the behavior of a complex system. That in turn enables powerful features like time-travel debugging.

To take advantage of logical time, this model has to be built into an application somehow. Because our tools generally don’t have any notion of determinism, you often see this with language-layer patterns like Redux or the Elm Architecture, or architecture-level patterns like event sourcing. All of those patterns reduce nicely down to the sequential state machine model presented here, but they’re up to the application developer to implement. The question that the Tomorrow Corporation demo asks is: what do we get if our tools did this for us without any up-front effort?

Imagine not needing to have to add sleeps / retries to tests of asynchronous behavior. Or imagine a tool that identified concurrent code and showed us the different interleavings that we might have otherwise been unaware of, and allowed us to step through and try each of them out. I’m not a Nix user (yet), but others are already imagining a world with deterministic package management. Non-determinism is fundamentally at odds with human brains it seems like, so I for one would love to see more determinism in any tool that I use.

To get there, we’ll have to understand and implement logical time.

Outro

I have no idea how the tools at Tomorrow Corporation are implemented, but I respect their commitment to determinism. Non-determinism is a part of life, but to have full control over a system it’s essential to view it through the deterministic lens of logical time. Because of things like concurrency which often rely on OS or language features that we can’t directly interact with, this can be difficult, but that video shows that there’s tremendous value in baking determinism further down into our foundational tools.

The main thing I wanted to share in this post was a specific mental model. Sequential state machines are a tried and true model with deterministic properties, and they’ve legitimately changed how I look at software. In this model, a program is a set of behaviors, where each behavior is a sequence of states. It’s hard to imagine reducing programming down to a simpler explanation than that, and that clarity is necessary for wrangling complexity.

The images in this post were generated from TLA+ specs, which I won’t really explain, but hopefully they show that it doesn’t take a ton of effort to write simple models. TLA+ is a logic and tool which has this mental model at its foundation. I can’t recommend learning and using it enough. Its companion model checker makes the act of modeling tactile, and you can get machine feedback on your models vs. getting stuck in state-machine quicksand. The state graph visualizer is also very handy sometimes, though as was shown here is more useful when the bounds of the model are small.

Here’s the spec for TodoMVC:

------------------------------ MODULE TodoMVC ------------------------------
VARIABLES todos, completedTodos

Todos == {"t1", "t2"}

Init == todos = {} /\ completedTodos = {}

RemainingTodos == Todos \ todos

IncompleteTodos == todos \ completedTodos

AddTodo == \E t \in RemainingTodos: todos' = todos \union {t} /\ UNCHANGED completedTodos

CompleteTodo == \E t \in IncompleteTodos: completedTodos' = completedTodos \union {t} /\ UNCHANGED todos

RemoveTodo == \E t \in todos: todos' = todos \ {t} /\ completedTodos' = completedTodos \ {t}

Next == AddTodo \/ CompleteTodo \/ RemoveTodo

=============================================================================

And here’s the spec for the concurrency example:

---------------------------- MODULE Concurrency ----------------------------
EXTENDS Integers

VARIABLES requests

Requests == 0..2

Init == requests = [r \in Requests |-> "idle"]

SendRequest(r) == requests' = [requests EXCEPT ![r] = "fetching"]

RecvResponse(r) == requests' = [requests EXCEPT ![r] = "done"]

SendReq == \E r \in Requests: requests[r] = "idle" /\ SendRequest(r)

RecvResp == \E r \in Requests: requests[r] = "fetching" /\ RecvResponse(r)

Terminate == \A r \in Requests: requests[r] = "done" /\ UNCHANGED requests

Next == SendReq \/ RecvResp

=============================================================================

Even if you never use TLA+, the mental model presented here can help understand software at a more fundamental level. Kudos to the Tomorrow Corporation team for an inspiring set of tools that I hope pushes people to think about determinism more.

\s, but it actually is a good learning tool and proxy for most interactive applications ↩

Efficient and Flexible Model-Based Testing

2023-01-31T00:00:00+00:00

In Property-Based Testing Against a Model of a Web Application, we built a web application and tested it against an executable reference model. The model-based test in that post checks sequences of actions against a global system state, which is simple to explain and implement, but is unsuitable for testing practical applications in their entirety. To test the diverse applications that arise in practice, as well as test more surface area of a single application, we’ll need a more efficient and flexible approach.

In that post, I promised that we’d dive deeper into the theory of model-based testing. To upgrade our testing strategy, we’ll look at the theoretical concepts of refinement mappings¹ and auxiliary variables², and add in a couple of tweaks based on the specific context of testing. All of this will get applied to a real test of a full-stack application.

A Quick Recap of Actions

Understanding the notion of “action” is essential for building our upgraded model-based testing strategy. When we say “action,” we mean something very specific: a transition in a state machine / state transition system, whichever name you prefer. It might be helpful to think of it from a code perspective:

class Counter {
  count: number = 0;

  constructor(count: number) {
    this.count = count;
  }

  increment() {
    this.count += 1;
  }

  decrement() {
    this.count -= 1;
  }
}

let counter = new Counter();
counter.increment();
counter.decrement();

count is the state variable, and increment and decrement are actions which transition the variable to a new state. Imagine the value of count after each of these actions.

The presence of a class has nothing to do with this being an object-oriented concept by the way, it’s just that classes are a convenient wrapper around a set of stateful variables and operations on them, and thus they are a good representation of a state machine. We could just as easily write:

let count = 0;

function increment(count: number): number {
  return count + 1;
}

function decrement(count: number): number {
  return count - 1;
}

count = increment(count);
count = decrement(count);

These are behaviorally equivalent, which we can convince ourselves of by again imagining the value of the count state variable after each action. The pattern that we use to talk about state machines is superficial, and has nothing to do with how to structure programs in the large. Don’t let the pattern get in the way of the underlying concepts: all we need are states and transitions between them, and we call these transitions “actions.”

In an interactive application, actions are generally initiated by the user by clicking on or tapping UI elements. The system itself can trigger actions, for example via cron jobs. Even external systems can trigger actions in the system by calling web APIs.

Actions are what allow an application to move through different states over time.

A Preview of Our Destination

The end goal is to convert our existing model-based test into one that’s more efficient and allows us to check more interesting properties. To do that, we’re going to end up with something that looks like this:

type DeleteRecurringTransactionState = {
  recurringTransactions: RecurringTransaction[];
  id: number;
  db: DBState;
}

class Impl {
  db: DBState;
  client: Client;

  aux: AuxiliaryVariables;

  constructor(db: DBState, client: Client, aux: AuxiliaryVariables) {
    this.db = db;
    this.client = client;
    this.aux = aux;
  }

  async deleteRecurringTransaction(id: number) {
    await this.client.deleteRecurringTransaction(id);
    this.aux.clientModel.deleteRecurringTransaction(id);
  }

  ...
}

type AuxiliaryVariables = {
  clientModel: Budget;
}

function refinementMapping(impl: Impl): Budget {
  let budget = new Budget();
  budget.error = impl.client.error;

  budget.recurringTransactions = [...impl.db.recurring_transactions];
  budget.scheduledTransactions = [...impl.client.scheduledTransactions];

  return budget;
}

Deno.test("deleteRecurringTransaction", async (t) => {  
  let state = /*<generate test state>*/;

  await fc.assert(
    fc.asyncProperty(state, async (state: DeleteRecurringTransactionState) => {
      let client = new Client();
      client.recurringTransactions = state.recurringTransactions;

      let clientModel = new Budget();
      clientModel.recurringTransactions = state.recurringTransactions;

      let impl = new Impl(state.db, client, { clientModel });
      let model = refinementMapping(impl);

      const cresp = await client.setup(state.db);
      await cresp.arrayBuffer();

      await impl.deleteRecurringTransaction(state.id);
      model.deleteRecurringTransaction(state.id);

      impl.db.recurring_transactions = await client.dbstate();

      let mappedModel = refinementMapping(impl);

      await checkRefinementMapping(mappedModel, model, t);
      await checkImplActionProperties(impl, t);

      await client.teardown();
    }),
    { numRuns: 10, endOnFailure: true }
  );
});

There’s no way to evaluate if this is a good test or even what exactly it’s testing for without understanding some theory. But all of this theory is in service of testing a real, functional single-page web application.

Correctness as Equivalent Behavior of Action Sequences

We have to start all the way at the beginning and define what it really means for an implementation to be correct with respect to a model. Action sequences are a good choice for this, because they’re simple to understand. Using our increment and decrement functions from above, an example action sequence would be:

type Action = "increment" | "decrement";

// Combine individual actions into a single top-level action
function counterAction(counter: number, action: Action): number {
  switch (action) {
    case "increment":
      return increment(counter);
    case "decrement":
      return decrement(counter);
  }
}

type ActionFunc<S, A> = (state: S, action: A) => S;

// Generic action sequence evaluation function
function execute<S, A>(actionFunc: ActionFunc<S, A>, init: S, actions: A[]): S {
  let result = init;
  for (const action of actions) {
    result = actionFunc(result, action);
  }

  return result;
}

let counter = 0;
execute(counterAction, counter, ["increment", "increment", "decrement", "increment"]);

An action sequence is one particular path through a system. Here, we incremented the counter twice, decremented once, and ended with another increment. These are some more valid action sequences:

[“increment”]
[]
[“increment”, “decrement”, “decrement”, “decrement”]
[“decrement”]
[“decrement”, “increment”, “increment”, “decrement” “decrement”, “increment”, “increment”]

How many possible sequences of actions are there for our simple counter system? 1,000? 500,000,000? Unfortunately, the answer is infinity, and that’s true of all interactive systems. That’s one reason why testing and verification is hard.

Even though they are infinite, it’s very natural to express the correctness of a model-based system in terms of action sequences using universal quantification, aka “for all” statements:

** Holistic correctness statement **:

For all initial states 's',
  all sequences of actions 'acts',
  a top-level action function 'impl',
  and a top-level action function 'model':
  
  execute(impl, s, acts) = execute(model, s, acts)

Less formally: no matter what sequence of actions you take in the implementation, nor what state it starts in, it should always agree with the model. The key words being “no matter what” and “always” - this should be true of all actions, in any order, from any starting state, ever. In other words, this statement is complete, and we’ll refer to it as “the holistic correctness statement.” It’s important to keep this statement in mind, since this is our definition of correctness and our end goal, and any optimization that we do always has to tie back to it. (Note: this is also a classic way of expressing refinement).

As we hinted at in the introduction, there are some very unfortunate things about this holistic correctness statement in a practical testing context. First is the actions variable. A real application accepts an infinite stream of actions. Even though we limit our test to finite sequences, combinatorics is just not on our side, with the number of k-length sequences of n actions equaling n^k - a dreadful exponential growth curve. That means that as the number of actions in the systems grows, and as we test longer sequences, the number of possible interleavings of actions grows exponentially. Whatever subset of sequences our test generates is an infinitesimal portion of them all.

Next is the s variable. This is the entire state of the system, and unless we’re building a counter application with a single integer variable it’s way too much data to generate in a test.

A third problem is that s is used in both of the model and implementation, which means that they both have to have the same state type. This very rarely works, because the whole point of separating the model and implementation is that the implementation is complex and will have additional state to deal with that. States are often incompatible in practice.

The last straw is that sometimes, you don’t even have the state variables that you need to check for correctness. This sounds weird, but it’s well known that specifications often have to be augmented with “invisible” variables so that certain properties can be shown to hold.

Each of these problems eventually arises when you try to use model-based testing, and we need some extra machinery to solve them.

Refinement mappings solve problems 1 and 3, and somewhat magically still also imply the truth of the holistic correctness statement. Meaning that, if we test for a proper refinement mapping, then it’s also true that the implementation correctly implements the model in all possible usage scenarios.

A refinement mapping is just a function with a couple of special rules, some of which are out of scope for this post. The first rule is that the function is from the implementation state to the model state, e.g. in our preview of the budget app test we can see that the refinement mapping maps the Impl implementation state type to the Budget model type:

function refinementMapping(impl: Impl): Budget {
  ...
}

The goal here is to be able to compare the implementation to the model, and if they have different state types we need to translate states in the implementation’s state space to ones in the model’s. On top of this, the most relevant other rule for a valid refinement mapping is that, for all implementation states and actions, the action is equivalent to the model action with the refinement mapping applied in the appropriate places. In logic pseudocode:

** Correctness via Refinement Mapping ** 
For all implementation states 's',
  all implementation actions 'impl',
  all model actions 'model'
  and a refinement mapping 'rm':

  rm(impl(s)) = model(rm(s))

The intuition for why it works is that, if every single-step action in the implementation agrees with the same action taken in the model, then chaining multiple actions into sequences should preserve that equivalence. This is an example of an inductive argument. The refinement mapping function can be defined in many different ways depending on how we want to relate the two state types, which gives our new correctness statement an important caveat: we consider the system correct under the provided refinement mapping. This is the price we pay for dealing with state incompatibilities.

In our budget app test, the refinement mapping is defined as follows:

function refinementMapping(impl: Impl): Budget {
  let budget = new Budget();
  budget.error = impl.client.error;

  budget.recurringTransactions = [...impl.db.recurring_transactions];
  budget.scheduledTransactions = [...impl.client.scheduledTransactions];

  return budget;
}

The Impl implementation type has both database (impl.db) and client states (impl.client), reflecting the independent states in a client-server application. In this system, only recurring transactions are persisted, and scheduled transactions are derived data. Because of this, the implementation’s recurring transactions in the database map to the model’s recurring transactions, whereas the implementation’s scheduled transactions in the client map to the model’s scheduled transactions. Any error in the client maps to an error in the model. Notably, this is talking about system errors, i.e. errors / results in the domain logic. The model has no notion of networking, so networking errors can be stored separately, but they don’t map to any model state³.

The meat of the test is where we compare single actions, and in order to do this we make the states compatible by applying the refinement mapping:

...

let impl = new Impl(state.db, client, { clientModel });
let model = refinementMapping(impl);

...

// Run the action in the implementation and the model
await impl.deleteRecurringTransaction(state.id);
model.deleteRecurringTransaction(state.id);

...

let mappedModel = refinementMapping(impl);

await checkRefinementMapping(mappedModel, model, t);

The combination of comparing single transitions and converting between implementation and model state types is an efficiency and flexibility win. We’ve gone from potentially long sequences of actions to comparing simple function calls, we only need to generate a single state value per test iteration, and we can compare the states of the implementation and model even if they aren’t the same type.

It’s great progress, but we can do even better.

From Global to Local State

The s variable in our new iteration of the correctness statement is still the global state, but an observation comes to mind: how much of the global state is necessary for each action? There’s no equation which answers this question directly, but intuitively, an action will only ever operate on a small subset of the global state, leaving the rest unchanged. We can then just ignore that superfluous state and think of the action as operating on its own, local state. This is not related to refinement mapping, or any other theory that I know of (though it might relate to one that I don’t know of), but ends up being a very useful optimization in practice.

For example, consider an oddly-specific system for point translation:

type Point = {
  x: number;
  y: number;
}

function translateX(point: Point, delta: number): Point {
  const result = { ...point };
  result.x += delta;

  return result;
}

function translateY(point: Point, delta: number): Point {
  const result = { ...point };
  result.y += delta;

  return result;
}

translateX and translateY are actions which operate on a Point type, but each only modifies a single part of the state - only x or y of the Point, but never both. Why, then, do we need to generate a full Point type in our test for comparing them? We can instead construct a new action function, say translateOnlyX, which only operates on the data that it actually modifies:

function translateOnlyX(x: number, delta: number): number {
  return x + delta;
}

In the model-based testing context, instead of comparing the functions at the global state level (Point in this case), we can compare the actions at the local level:

** Local Refinement Mapping Correctness Statement **

For all action functions 'impl',
  all action functions 'model',
  all local states 'ls',
  and a refinement mapping 'rm':
  
  rm(impl(ls)) = model(rm(ls))

Breaking out the action implementation in this way has no behavioral effect on the global-level translateX function, since translateX can easily be implemented in terms of translateOnlyX:

function translateX(point: Point, delta: number): Point {
  const result = { ...point };
  result.x = translateOnlyX(result.x, delta);

  return result;
}

And this is exactly what’s going on in our upgraded budget test. In our excerpt, we’re only focusing on the deleteRecurringTransaction action, and we generate a test state specific to this action:

type DeleteRecurringTransactionState = {
  recurringTransactions: RecurringTransaction[];
  id: number;
  db: DBState;
}

Deleting a recurring transaction doesn’t interact in any way with the scheduledTransactions state variable in that application, so we can leave that out of the test state for this particular action.

The end result of this is that we can get global guarantees for the cost of local checking, i.e. we can use local states to still show the holistic correctness statement.

One More Wrinkle

One last wrinkle presents itself for now - the notorious problem number 4. It may sound counterintuitive, but there are both refinement mappings and properties of our systems that are not expressible with the state variables of the system itself. Even if they are, they may be more naturally expressed by adding auxiliary variables. Auxiliary variables are additional variables that are added to a program (usually the implementation) that don’t affect the behavior of the program, but can be used to state properties or aid in a refinement mapping to a model.

Auxiliary variables provide one solution to a problem in the budget app test, and for tests for client-server applications in general. Our implementation is both the state component of a single-page application, and the corresponding server and database. One implication of that is that the client and database state can become out of sync. Consider the following action sequence:

The database starts with these recurring transactions: [rt1, rt2, rt3].
User 1 loads the home page - its client holds [rt1, rt2, rt3]
User 2 loads the home page - its client holds [rt1, rt2, rt3]
User 2 deletes rt2 - its client now holds [rt1, rt3], and the database holds [rt1, rt3]
User 1 adds a new recurring transaction, rt4 - its client holds [rt1, rt2, rt3, rt4] and the database holds [rt1, rt3, rt4].

At the end of these actions, the system has the following state:

User 1’s client: [rt1, rt2, rt3, rt4]
User 2’s client: [rt1, rt3]
The database: [rt1, rt3, rt4]

Again, there are a few different ways to approach either allowing or disallowing this behavior. One option is to just forbid differences in client values, but this would require a web socket to update all clients on each data write. While some applications actually do this (like chat applications), I would say that most don’t. Instead, we have to allow diverging client states, but we still want to do that in a controlled manner.

Well, one solution to that is to add a separate model instance as an auxiliary variable to the implementation which tracks the source of truth of the state of the client alone. Then, whenever a write occurs, we double-write to the implementation and this client model. Again, there are many patterns for doing this, but I like wrapping the implementation (Client here) in a new class with the same interface that forwards actions to the relevant members, this way the structure of the test doesn’t have to change and we keep all of the auxiliary variables in test-specific code:

class Impl {
  db: DBState;
  client: Client;

  aux: AuxiliaryVariables;

  constructor(db: DBState, client: Client, aux: AuxiliaryVariables) {
    this.db = db;
    this.client = client;
    this.aux = aux;
  }

  async deleteRecurringTransaction(id: number) {
    await this.client.deleteRecurringTransaction(id);
    this.aux.clientModel.deleteRecurringTransaction(id);
  }

  ...
}

In the test excerpt, we see another assertion named checkImplActionProperties⁴, and its defintion will now make sense:

async function checkImplActionProperties(impl: Impl, t: Deno.TestContext) {
  await t.step("loading is complete", () => assertEquals(impl.client.loading, false));

  await t.step("write-through cache: client state reflects client model",
    () => assertEquals(impl.client.recurringTransactions, impl.aux.clientModel.recurringTransactions)
  );
}

After each action has been invoked, we check that the actual state of the client matches the state of the client model, not the system model which is only aware of the database state. We also check that the loading variable in the client is false for good measure, ensuring that any spinners or other loading UI are hidden at the end of every action.

The key here is that, as long as they don’t affect the behavior of the implementation, we can add any auxiliary variables we want for tracking additional information. Once we have them, we can use them for test assertions, totally independent of the implementation that runs in production. They’re test-only code.

I’m going to be honest - I can have too much fun with auxiliary variables, and that means that we should be careful with them. They are basically a cheat code, and can be used as an escape hatch to get out of all kinds of situations. That being said, they’re sometimes the most elegant solution to a problem, and they’re a key piece in making our test flexible enough to handle the many scenarios that arise in practice. If anything becomes difficult to assert on or express as a property, we can try and make them easier by adding new auxiliary variables.

Recap

Alrighty. We went over four main problems and solutions to them:

Action sequences
Global state
State incompatibility
Expression inability

We introduced refinement mappings, which are functions from the implementation state to the model state, and which require that single-transitions in the implementation and model must be equivalent under this mapping. This overcomes both state incompatibility and the need for action sequences. We showed that by using action-local state we can avoid ever constructing global system state in the test. And we showed that if we ever have the inability to express a property about our system, we can always add auxiliary variables which don’t affect the system behavior but track additional information that we can use in test assertions.

What we ended up with is a framework for writing model-based tests that is both efficient and flexible, and applicable to real-world systems like database-backed web applications.

The linked papers have plenty more theoretical background and examples for deeper dives on these topics.

Thanks

Big thanks to Hillel Wayne for having an in depth conversation about refinement with me, which influenced my thinking about how to best define the system state for a client-server application.

I recommend reading this paper to get a handle on refinement mappings. Another name for this technique is simulation, which you can see an example of in how seL4 proves that the implementation implements its functional specification. Both are the same ultimate idea - prove that one program implements another by showing that all single transitions in each implement each other. ↩
We’ll expand on what auxiliary variables are throughout the post, but you can read more about them here and here. ↩
Errors that can be present in the implementation but not the model are an interesting topic. For example, if a network error in a request during the course of an action in the implementation, then it certainly won’t complete the action in a way that implements the model. One option is to be liberal, and simply avoid comparing the model and implementation in this case. We didn’t cover stuttering here, but models are allowed to stutter (transition to the current state) during implemenation steps, so an implementation error could be interpreted as a model stutter. The issue is, if the network error happens on every single action invocation, the implementation will never match the non-stuttering step of the model. The other option is to be harsh, and require that there are no network errors in tests, but still plan for them and allow them in production. This current version of this test chooses to be harsh. I’ll let you know how that goes. ↩
Action properties are a subset of temporal properties. They allow you to assert things about state transitions, that you couldn’t assert about individual states. They’re very useful. ↩

Concerning Quality

Bug Bash 2025 Conference Experience

Challenge the Status Quo

Test End-to-End

Formal Methods Push The Limits of Testing

Branch Coverage Won’t Prove The Collatz Conjecture

Simulating Some Queues

The Unit Queue

Discrete Event Simulation

Processing Distributions

Code

Controlling Nondeterminism in Model-Based Tests with Prophecy Variables

Handling Implementation-Level Errors in a Model

Tests, Oracles, and Prophecy

A Brief Note on The Theory of Prophecy Variables

Prophecy-Aware Dependencies and Modal Determinism

In Closing

Does Your Test Suite Account For Weak Transaction Isolation?

What is Weak Transaction Isolation?

Simulating Concurrent Connections

Race Conditions and Serializability

Forward and Backward Reasoning in Proof Assistants

Forward Reasoning

Backward Reasoning

Which One’s Better?

Compiling a Test Suite

From Programs to Applications

Writing a Model

Compiling the Test Suite

Outro

Most Tests Should Be Generated

Correctness is What We Want

Testing for Correctness with Scenarios

Generating Tests for Properties

The Test Generation Pyramid

Logical Time and Deterministic Execution

User Interaction and Non-Deterministic Choice

Concurrency

Logical Time, Time-Travel, and Beyond

Outro

Efficient and Flexible Model-Based Testing

A Quick Recap of Actions

A Preview of Our Destination

Correctness as Equivalent Behavior of Action Sequences

Single Transitions and Compatible States with Refinement Mappings

From Global to Local State

One More Wrinkle

Recap

Thanks