Concurrency bugs are perhaps the most insideous kind of bugs¹. They’re sneaky, intermittent, hard to reproduce, and only like to announce themselves when and where you least expect (usually in production). However, that only makes them all the more interesting! Add to that the feeling of victory you get when you catch & solve one. So let’s see how I did that ;).

The Hunt

It all began when our sneaky concurrency bug kindly decided to reveal itself on CI, although somewhat coyly, while I was working on Methanol’s redis HTTP cache storage backend, through some HttpCacheTest failure. The failed test looked more or less like this:

@StoreParameterizedTest
void cacheGetWithMaxAge(Store store) throws Exception {
  setUpCache(store);
  
  server.enqueue(
      new MockResponse()
        .setBody("Pikachu")
        .setHeader("Cache-Control", "max-age=2"));
  verifyThat(send(serverUri))
      .isCacheMiss()
      .hasBody("Pikachu");

  clock.advance(Duration.ofSeconds(1));
  verifyThat(send(serverUri)) 
    .isCacheHit()
    .hasBody("Pikachu");
}

(yup, I use Pokémon names when testing, makes it fun!)

We’re testing basic cache behavior here. On first encounter, the cache forwards the request to network, and saves the response as it arrives. This results in an initial cache miss. Later on, we expected the response to be cached within its allowed time window (here it’s 2 seconds).

The above test failed with:

HttpCacheTest > cacheGetWithMaxAge(Store) > cacheGetWithMaxAge(Store)[3]: com.github.mizosoft.methanol.internal.cache.DiskStore@24018c8b FAILED
    org.opentest4j.AssertionFailedError: 
    expected: "Pikachu"
    but was: ""
... omitted stacktrace ...

Hmm, we’re getting an empty response body from cache. Looks like we’re either not saving the body properly or truncating it while reading. The cache has been around for more than a year, and I’ve never come across such a failure. Indeed, lamely staving this off as yet-another-flaky-test is attractive. However, I was determinted to find the cause, at least out of pure interest! This kind of bugs, however, are tough because you don’t exactly know where to look. Additionally, debuggers won’t help much². However, we can start with pinning down potential cultprits:

• DiskStore: This handles low-level storage of response data on disk.
• CacheWritingPublisher: Intercepts body flow from network and writes it as it’s consumed.
• CacheReadingPublisher: Reads the response body from cache on cache hits.
• CacheInterceptor: Decides, amongst other things, whether to forward the request to network or return a cached response if appropriate.

The trick is to know where to look first. We can overlook DiskStore for now as it basically wraps file IO using a simple layout (one file per entry). The interceptor has little to do with how data is actually read or written. CacheWritingPublisher & CacheReadingPublisher are the main suspects, particulary because they have involved concurrent implementations, so there’s a larger surface area for bugs. The former is less probable to be the cause than the latter, however, as reading the response body from network succeeds (as you can see from the first part in the test), which makes it likely that it has been fully written. CacheReadingPublisher is our main suspect! Let’s see how it’s structured.

Reading From Cache

CacheReadingPublisher is basically a producer and a consumer bundled together. The producer actively reads data from the store and dumps it into a queue till end of file is reached (with cool-downs if the consumer is slow). The consumer polls data from the queue and passes it downstream as requested³. As the consumer can be faster than the producer, it can perceive an empty queue while data is being read. That’s why there’s a volatile boolean exhausted that’s set to true by the producer when it’s done, so the consumer knows for sure no more data is expected and can go on to complete downstream when it sees an empty queue.

The producer and consumer are interleaved. There’s no syncyhronization between them aside from the implicit synchronization enforced by the queue (a happens-before between adding an item and polling it), and sharing exhausted. However, actions in each happen sequentually, in a quasi-single-threaded manner⁴.

The relevant consumer code looks(ed) like this:

@Override
protected long emit(Subscriber<? super ByteBuffer> downstream, long emit) {
  ByteBuffer next;
  long submitted = 0;
  while (true) {
    if (queue.isEmpty() && exhausted) {
      cancelOnComplete(downstream);
      return submitted;
    } else if (submitted >= emit || (next = queue.poll()) == null) {
      return submitted; // Exhausted demand or items.
    } else if (submitOnNext(downstream, next)) {
      submitted++;
    } else {
      return 0;
    }
  }
}

And this is producer’s callback after an asynchronous read:

private void onReadCompletion(
    ByteBuffer buffer, @Nullable Integer read, @Nullable Throwable exception) {
  assert read != null ^ exception != null;

  // The subscription could've been cancelled while this read was in progress.
  if (state == State.DONE) {
    return;
  }

  if (exception != null) {
    state = State.DONE;
    listener.onReadFailure(exception);
    fireOrKeepAliveOnError(exception);
  } else if (read < 0) { // End of file.
    state = State.DONE;
    exhausted = true;
    listener.onReadSuccess();
    fireOrKeepAlive();
  } else {
    readQueue.offer(buffer.flip().asReadOnlyBuffer());
    if (!tryScheduleRead(true) && STATE.compareAndSet(this, State.READING, State.IDLE)) {
      // There might've been missed signals just before CASing to IDLE.
      tryScheduleRead(false);
    }
    fireOrKeepAliveOnNext();
  }
}

There’s a number of things happening here, but the relevant parts are setting exhausted = true and readQueue.offer(...), followed by fireOrKeepAlive[OnNext](). The latter basically ensures the consumer is up and running to see what we’ve produced. Now give yourself a moment to find the bug! (hint: the culprit is the consumer).

The Bug

I like thinking about thread safety issues because most of the time it’s like a guessing game. You’re basically trying to work out scenarios that break your code. Indeed, there are more formal ways for proving correctness, but let’s now rely on our imagination and picture this scenario:

• The queue is now empty, and there’s an ongoing read.
• The consumer gets called and executes the if (queue.isEmpty() && exhausted) part.
• queue.isEmpty() returns true, and just before testing (reading) exhausted, the consumer thread gets paused⁵.
• The read completes, triggering another one that also completes (reads are triggered one after another upto a certain prefetch limit).
• The first read results in pushing a data buffer into the queue. The second read observes end of file and sets exhausted to true.
• The consumer thread gets unpaused, and goes on testing exhausted, which has just become true.
• The test succeeds and the consumer completes downstream.
• Boom! A data buffer gets completely ignored.

Because our response in the test was small, this happened at the beginning, so we got an empty response body. Now give yourself a moment to come up with a simple fix! (hint: one hacky fix is to change only one statement in consumer code).

The Hacky Fix

Well, we can basically switch the testing order from if (queue.isEmpty() && exhausted) to if (exhausted && queue.isEmpty())⁶. This works because once exhausted is set to true, no more buffers are expected. That is, once exhausted is true, we’re bloody sure we’ve exhausted all buffers, and if we see an empty queue henceforth, that’ll be its final state, and it doesn’t matter whether, when, or where the thread gets paused.

Note that our main problem is Java’s lack of a closeable queue, which implies 3 states (non-empty, empty, closed) instead of two (non-empty, empty). In Go, for instance, we would’ve simply used a channel, which is basically a closeable BlockingQueue⁷.

A Better Fix

Having correctness rely on testing order is fragile. Interestingly, we only care about closing from one side: the producer⁸. We can use this fact to come up with a better fix using what’s fancily called sentinel values⁹. A sentinel value is basically a made-up constant that marks a certain event. We can create an empty buffer as a sentinel and add it to the queue when end of file is reached. This modifies producer’s code as follows:

private static final ByteBuffer SENTINEL = ByteBuffer.allocate(0);

private void onReadCompletion(
    ByteBuffer buffer, @Nullable Integer read, @Nullable Throwable exception) {
  
  ...

  if (exception != null) {
    ...
  } else if (read < 0) { // End of file.
    state = State.DONE;
    queue.offer(SENTINEL);
    listener.onReadSuccess();
    fireOrKeepAlive();
  } else {
    ...
  }
}

And the consumer can confidently complete downstream when it sees that sentinel:

private @Nullable ByteBuffer lastNext;

@Override
protected long emit(Subscriber<? super ByteBuffer> downstream, long emit) {
  // Pick up from where we left off.
  var next = lastNext;
  lastNext = null;

  long submitted = 0;
  while (true) {
    // Poll prematurely to complete regardless of demand.
    if (next == null) { 
      next = queue.poll();
    }

    if (next == SENTINEL) {
      cancelOnComplete(downstream);
      return submitted.
    } else if (submitted >= emit || next == null) {
      lastNext = next; // Save the last polled buffer, which might be non-null.
      return submitted; // Exhausted demand or items.
    } else if (submitOnNext(downstream, next)) {
      next = null; // Consume.
      submitted++;
    } else {
      return 0;
    }
  }
}

This’ll work because seeing SENTINEL strictly happens after seeing all the buffers pushed by the producer (due to the FIFO nature of the queue), so we won’t miss anything. Note that we had to store the last polled buffer¹⁰ as, in order to check for completion, we poll before checking downstream demand. So we might end up polling a buffer that hasn’t yet been requested.

It was an enormous, round, flaming object that I saw approaching through the mist of the bright blue sky. Vaguely perceivable as a partially visible, gibbous moon on fire, with its semi-molten crust almost retaining its dark grey color despite the blaze, the figure seemed to get larger although in a manner that could only be felt rather than perceived. Gazing at it through one of the two narrow, vertically oblong windows in my room, it came to my awareness, as if by some form of recollection or latent realization, that the object is aiming for somewhere near, somewhere not very far behind where I stand. The colossal hellish figure gave the unequivocal impression that the havoc it is destined to cause wasn’t only the local destruction of where it strikes, but that of the planet in its entirety. It was our extinction, gently penetrating the atmosphere. That gentleness, however, couldn’t alleviate the acute sense of impending doom coarsely spreading through the observer’s awareness, most notably, my awareness. A sense that could only be intensified by the immense, jarring form of the world destroyer and its fierce, volcanic flames. Helpless as I am, as anyone was, the last few moments could only be spent absorbing the striking, apocalyptic scenery in paralyzing awe, all while appreciating the heavenly reddish-yellow, elegantly hazed aura surrounding the rampant burning moon.

At the outset of these moments, however, as soon as one’s doom was unmistakably ascertained, a certain kind of illusion was suddenly unveiled. It was one of these illusions that collectively and cooperatively continue to make the tragedy of existence vaguely tolerable. One simply couldn’t bear without, willingly or unwillingly, yielding to such illusions. And it was precisely that said illusion that made any foresighted action not only possible but also desirable. What illusion would that be? Precisely an illusion that death was only a prospect. An illusion of permanence, of eternal continuance. That the future was a real place, or at least as real as the present. That the present could be sacrificed. That such sacrifice is warranted. And, simultaneously and paradoxically, that one’s responsibilities and obligations could be safely staved off for eternity. Such an illusion is so necessary that it could only be evaporated by an incident of such gravity. Only in moments like these that you knew death is a certainty. An indisputable truth. As all-too-real as it could be. You could taste it in your mouth. You could feel it in your stomach. All while it’s slowly taking possession of your awareness, and not planning to give it back.

At first, it didn’t matter how much time was left. At any rate, I knew I was only as dead as I could ever be. However, death then wasn’t only a knowable fact. It was an internalized realization. The kind of realization that you didn’t only know, but immediately had to act upon. A question remains: what could any action do? It is precisely that kind of absolute appreciation of inescapable death that makes inaction the only logical action.

Nonetheless, the fact that I still had time, even if mere seconds, somehow struck my consciousness as relevant. I thought, as naively as I could, the approaching cosmic spheroid of death would spare me these seconds before it strikes, and before the wave of destruction its impact causes spreads till ultimately swallowing me. And with that, it only became self-evident that there was something I ought to do before my inevitable dissolution into nothingness. But what could ever be done that would make a difference?

Insufferable dread suddenly took possession of my soon-to-be-star-dust-again body. It wasn’t exactly death that was dreadful. It self-evidently transpired that when death became a palpable reality, you could cease the need to fear it. The fear wasn’t of death, as per se; it was of what death entailed, of its implications on your existence, or lack thereof. Now that death became an actuality, it was another prospect that was ought to be feared. After all, what is fear if not an all-too-crushing realization of what could be?

By the dictates of what seemed like some form of swift realization that was in fact instinct in disguise, deciding to manifest itself as intuition, I started enacting what seemed to be the only thing worthy of enacting. Suddenly, I struck the ground headlong, with my forehead laying flat, and my legs tucked between my torso and the ground. I started uttering hardly distinctive words, in a manner that indicated I knew exactly who I was talking to, with voice that couldn’t help but tremble out of urgency and fear. As my quivering voice got harsher, murmurs gave way to bellows. Of the top of my lung, words were roared at a rate and an intensity that made them indistinguishable from screams. No matter how much I roared, however, it never seemed I’d roared enough. Never have I had so much to say, with so little time to say it, and an irresistible desire to roar it all out.

Before I could grasp my shouts, which were supposed to resemble uttered words, an insuppressible sense of futility pervaded. I knew, at the very depths of my awareness, or what was left of it, that my indistinctive, rushed, frenetic maneuver, whichever purpose it served, was all in vain. It was simply too late. It was something that wasn’t done when it was due and now must be compressed into seconds of roaring, all to ultimately serve no purpose whatsoever, but perhaps to retain, or rather lose, the crumbs of dignity one was left with. And with that I could finally, if vaguely, grasp my words, and know precisely what my actions represented. It was a feeble little attempt at redeeming my unredeemable self. A futile, unworthy repentance. One last plead to the infinite, crushing force of absolute and ultimate judgment. A force that was only then realized, and as a result, became too much to bear that it made you instantly crumble appealing for absolution. One last vain effort at doing all the things I’ve always known I must’ve done, but willingly decided not to, manifested in what seemed most as so then. One last prayer.

In a manner that seemed abrupt, although one that couldn’t be more seamless and elegant, resembling a transition that one blacked out right when it occurred, I found myself drifting upwards through an unfamiliar medium, right after the roaring rose to its climax. It almost felt as if I was floating, but there was no water. It was this weightless, bodyless, appendageless sensation that I could most recognize. The entirety of my bodyless singularity seemed to be concentrated right behind my eyes, or whichever that which gave me sight. What I saw wasn’t of the world that once was, that I once knew. Spinning patterns that vaguely resembled colorful, although greyish, interweaving, smooth television noise were morphing into other alternating noise patterns, all while maintaining the same overall pattern. Despite my self-awareness, I couldn’t quite tell whether I was an actor or a spectator. My self-conscious, drifting soul seemed to unmistakably know it was no more. I knew I was dead. I knew death wasn’t the end. I knew precisely I was experiencing what seemed to be my post-corporeal continuance. All dread was gone. Whichever made me an agent of my dread-induced impulses seemed to magically evaporate. I couldn’t do but endlessly and continually transcend upward this unearthly, colorful, dominantly greyish, noisy, interminable void.

In that same manner of seamless abruptness, I could feel the familiar heft of my familiar body. Simultaneously, I could feel the weight of my firmly shut eyelids. Slightly perplexed, I endeavored to open them. All in a split of a second, I succeeded. I could see my head resting on the familiar pillow, all on top of the familiar bedsheet, and, less than a meter from my feet, the familiar two narrow, oblong windows. A silent sigh of relief echoed throughout my reclaimed awareness. It was yet another wicked act of my spiteful unconscious. A vehement protest of the psyche against its owner. Perhaps it was a warning. A violent call towards the path of salvation. One last trick up my unconscious’ sleeve to call forth what my consciousness miserably failed to. Perhaps it was a cry for help. An elaborate exclamation. Perhaps it was a call towards a necessary initiation one ought to undergo before it’s too late. Perhaps, considering the sense of futility that hasn’t yet worn off, it was an affirmation that it is indeed too late, but one could at least retain his dignity, or humanity, by making one last earnest attempt, even though it’s ineffectual. Am I to answer to such a call? Perhaps so. Perhaps I have no other choice. But perhaps that could be safely staved off. Perhaps I could go off to retain my usual 12-hour streak of sleeping. Perhaps all matters shall be resolved tomorrow, or whenever I really wake up. After all, I have all the time in the world. After all, I am eternal. I am permanent. After all, death is only a prospect.