We ran many iterations on
@resonatehqio's Distributed Async Await to make durable execution dead simple. Despite its simplicity, the protocol hides edge cases.
Here is one.
Every promise p has a timeout t. If a request arrives after t but before the server performed a timeout sweep, the request handler must time p out itself.
As an optimization we did this lazily: return a timedout promise to the caller without writing to storage, let the sweep write the promise later.
Single server: No issues. Multi server: Death by clock skew.
Two servers, two clocks. A's clock says t has passed, returns promise timed out. B's clock says t has not passed, settles the promise.
A promise goes from timedout to settled, a violation of promise semantics.
The fix: A handler that witnesses a time-out first must write to storage before returning a timedout promise.
Now the timeout is durable: any other handler reads the settled-as-timed-out state and is fenced from writing, regardless of its own clock.
Even the simplest distributed protocols explode into intractable state spaces.
Death by a thousand edge cases 🍎