The two previous posts shared a comfortable assumption: doing something again is
free. In the post about the
queue, a stuck job that was fetching a file list simply went back to PENDING
and ran again — one more read hurts nobody. Translating twice gives the same
text, too.
Delivering is different. Creating a GitHub Release twice, posting to Slack twice, or emailing a thousand people twice is not free. And when a worker stops in the middle of a call, it cannot tell whether the provider acted.
ADR-0020 recorded exactly that question before ReleaseFlow had its first delivery feature. This post, the last in the series, is about the answer.
A question nobody can answer
Picture ReleaseFlow sending a release’s note to Slack. The request goes out, then the connection drops before a response arrives. Did Slack post the message?
Maybe, maybe not. Nobody knows — ReleaseFlow included. Record that delivery as
FAILED and retry it automatically, and the Slack channel may get two messages.
Record it as SUCCEEDED, and nobody may get anything without anyone noticing.
So ReleaseFlow picks neither. "We don’t know" is recorded as exactly that:
/**
* How one delivery ended. {@code UNKNOWN} means the provider may have accepted it:
* nothing may repeat such an Action without a person saying so.
*/
record ActionResult(ExecutionStatus outcome, String externalReference, String errorCode) {
// ...
}
A delivery ends in one of six statuses: PENDING, RUNNING, SUCCEEDED,
FAILED, UNKNOWN and CANCELLED. UNKNOWN is not a kind of failure. It is a
status of its own, with a rule of its own: nothing may send it again on its own.
FAILED or UNKNOWN: where the line is
The question that decides is simple: is it certain the provider did nothing?
This is the response handling in SlackActionExecutor:
} catch (RestClientResponseException exception) {
int status = exception.getStatusCode().value();
if (exception.getStatusCode().is5xxServerError()) {
// Slack may have posted the message before it failed to answer.
log.warn("Slack answered HTTP {} for the note of release {}.", status, command.releaseId());
return ActionResult.unknown(ActionResult.OUTCOME_UNKNOWN);
}
log.warn("Slack refused the note of release {} with HTTP {}.", command.releaseId(), status);
return ActionResult.failed(REJECTED);
} catch (RestClientException exception) {
// Slack may have posted the message before the connection failed.
log.warn("Could not reach Slack with the note of release {}.", command.releaseId());
return ActionResult.unknown(ActionResult.OUTCOME_UNKNOWN);
}
-
A 4xx means Slack read the request and refused it — a wrong URL, a revoked webhook. Nothing was posted for sure, so it is
FAILED. -
A 5xx means something went wrong on Slack’s side, but it may have gone wrong after the message was posted. That is
UNKNOWN. -
A dropped connection — a timeout, a network failure — is
UNKNOWNtoo, because the request may have arrived.
ReleaseFlow does not follow redirects when delivering. A redirect response means
the note went nowhere, so it is FAILED, not UNKNOWN.
The same question draws slightly different lines for each provider:
-
For email, a failure while composing the message is
FAILED— nothing has left ReleaseFlow yet. But when SMTP does not confirm the send, it isUNKNOWN, because the server may have accepted the message for some of the recipients, or all of them. -
Zendesk takes two calls: get an access token, then create the article. The first call never ends
UNKNOWN— nothing has been published, so trying again later cannot create a duplicate. The second call can. Even when Zendesk answers 2xx but the response names no article, the result is stillUNKNOWN: nobody can say for sure whether an article exists.
A dead worker becomes UNKNOWN, not PENDING
The delivery queue uses the same FOR UPDATE SKIP LOCKED pattern as the second
post. But when it recovers stuck jobs, it does the opposite.
In the second post, a file-fetching job stuck for more than ten minutes went back
to PENDING. Here, a delivery stuck for more than five minutes becomes
UNKNOWN:
/**
* Deliveries cannot be repeated safely, so an Action whose worker stopped is never
* released back to PENDING: it becomes UNKNOWN and waits for a person.
*/
@Query(value = """
SELECT id FROM automation_action_runs
WHERE status = 'RUNNING' AND claimed_at < :staleBefore
FOR UPDATE SKIP LOCKED
""", nativeQuery = true)
List<UUID> findStaleRunningIds(@Param("staleBefore") Instant staleBefore);
The worker calls markUnknown on every delivery it finds, and settles the status
of the whole run again. Same queue, same way of claiming jobs, opposite decisions
— because the cost of doing it again is different. To me, this is the biggest
lesson of the whole series: no retry policy is right for everything. You have to
ask, for each kind of work: what does doing it again cost?
One more rule: the actions of a rule run in order, and the first action that
ends FAILED or UNKNOWN stops every action behind it. Nobody wants the note
posted to Slack while it is unclear whether the GitHub Release exists.
Only a person decides to send again
A FAILED run can be retried freely: nothing was sent, for sure. An UNKNOWN
run needs a person to confirm that sending twice is acceptable:
public AutomationRunView retry(UUID organizationId, UUID runId, boolean confirmUnknown) {
AutomationRun run = find(organizationId, runId);
if (run.getStatus() == ExecutionStatus.UNKNOWN && !confirmUnknown) {
throw AutomationConflictException.unknownNeedsConfirmation();
}
// ...
}
Without the confirmation, the API answers
409 automation_unknown_needs_confirmation. In the interface, the Run again
button comes with a box that must be ticked: "This may deliver twice".
A retry resets only the first action that stopped, along with the actions behind it that were waiting on it. An action that succeeded is never repeated.
Before pressing the button, the operator can do what ReleaseFlow cannot: open the Slack channel or the Help Center and check whether the message or article is already there.
When the provider lets you ask: GitHub Release
For a GitHub Release, ReleaseFlow can answer "did the last attempt get through?" itself, because a release can be read back by its tag. Every delivery writes a marker carrying its own ID at the end of the body:
private static String marker(UUID actionRunId) {
return "<!-- releaseflow-action:" + actionRunId + " -->";
}
Before creating anything, the executor reads the release by tag:
// A release this Action Run wrote is the delivery; anything else is somebody's work.
private static ActionResult reconcile(GitHubRelease release, String marker) {
String body = release.body() == null ? "" : release.body();
return body.contains(marker)
? ActionResult.succeeded(release.htmlUrl())
: ActionResult.failed(RELEASE_CONFLICT);
}
-
No release with that tag yet: create one, with the marker.
-
One exists, and its body carries this delivery’s marker: that was the previous attempt, and it got through. The result is
SUCCEEDED, and nothing more is created. -
One exists, but it is somebody else’s:
FAILEDwithgithub_release_conflict. ReleaseFlow never overwrites a release somebody else made. -
Somebody creates that exact tag between the read and the create: read again and decide as above.
So retrying an UNKNOWN GitHub Release action is safe: if the previous attempt
arrived, this one just recognises it.
Not every provider allows that. A Confluence page also carries an HTML comment
naming the run that wrote it, but ReleaseFlow does not read the page back. So a
Notion or Confluence delivery that was never confirmed stays UNKNOWN until a
person accepts a second page.
Delivery never blocks a release
There is one more place where things can be sent twice — or worse — and it sits right at publication. The project’s old version created its runs inside the publishing transaction. A rule pointing at an audience with no note threw an exception, and the publication rolled back with it. As ADR-0020 puts it, a release that is ready to publish is not the place to discover that somebody mistyped a Slack URL.
ReleaseFlow now writes exactly one outbox row when publishing:
@EventListener
void onReleasePublished(ReleasePublished event) {
if (jobRepository.existsByReleaseId(event.releaseId())) {
return;
}
jobRepository.save(new AutomationPublishJob(
UUID.randomUUID(),
event.organizationId(),
event.projectId(),
event.releaseId(),
clock.instant().truncatedTo(ChronoUnit.MICROS)
));
}
This listener matches no rule, reads no note, and calls nothing outside. Only
after the transaction commits does AutomationWorker turn that row into runs. A
rule that cannot be carried out becomes a FAILED action in the history — for
example release_note_missing — rather than a release that will not publish.
Stopping duplicates on the way in
Duplicate deliveries do not only come from retries. They also come from being triggered twice. Two unique indexes in migration V22 stop that:
-- A publication fires each matching Rule once, and a repeated manual request is the
-- same Run rather than a second delivery.
CREATE UNIQUE INDEX automation_runs_published_once
ON automation_runs (rule_id, release_id)
WHERE trigger_type = 'RELEASE_PUBLISHED';
CREATE UNIQUE INDEX automation_runs_request_once
ON automation_runs (rule_id, request_id)
WHERE request_id IS NOT NULL;
Each rule fires once per publication. The manual run form carries a request ID,
so pressing the button twice or the browser resubmitting the form both return
the same run. The same goes for other systems calling in through a signed
webhook: the X-ReleaseFlow-Delivery header is a UUID that stays the same on a
retry, so a webhook delivered again does not create a second run.
The trade-offs, plainly
-
UNKNOWNneeds a person. A network incident can leave a few runs waiting for an operator to open them, check, and decide. That is extra work, and I accept it. -
I chose "a delivery may be missing, but someone will know" over "a delivery may happen twice, and nobody will know". A missing release note can be sent late. An email that reached a thousand inboxes twice cannot be taken back.
-
The run history is terse. Errors are stored only as fixed codes, such as
execution_outcome_unknownorgithub_release_conflict, never as a message from a provider or an exception — so nothing unexpected leaks into the interface. To learn more, for example the HTTP status Slack returned, an operator has to read the application log.
Closing the series
Four posts, three technical decisions, and looking back they share one idea:
-
A job queue in PostgreSQL — the machine repeats what can be repeated, and never silently skips work;
-
AI suggests, people decide — the machine drafts, and the rules and people keep the decisions;
-
and this post — when the machine is not sure, it says so, and lets a person decide.
That is also the point of the Reddit comment from the first post: automation for the repetitive parts, AI for the writing, and people to review and decide.
Thank you for reading this far. ReleaseFlow’s code is in the
GitHub repository, the
documentation is on the
docs site, and the decisions
in this post are recorded in ADR-0020. If you try it and run into an UNKNOWN
delivery — or anything else — open a
Discussion on
GitHub.