Hoang Luong Tran

Delivering without delivering twice: why UNKNOWN is not FAILED

· 9 min read releaseflow java spring-boot postgresql

The two previous posts shared a comfortable assumption: doing something again is free. In the post about the queue, a stuck job that was fetching a file list simply went back to PENDING and ran again — one more read hurts nobody. Translating twice gives the same text, too.

Delivering is different. Creating a GitHub Release twice, posting to Slack twice, or emailing a thousand people twice is not free. And when a worker stops in the middle of a call, it cannot tell whether the provider acted.

ADR-0020 recorded exactly that question before ReleaseFlow had its first delivery feature. This post, the last in the series, is about the answer.

A question nobody can answer

Picture ReleaseFlow sending a release’s note to Slack. The request goes out, then the connection drops before a response arrives. Did Slack post the message?

Maybe, maybe not. Nobody knows — ReleaseFlow included. Record that delivery as FAILED and retry it automatically, and the Slack channel may get two messages. Record it as SUCCEEDED, and nobody may get anything without anyone noticing.

So ReleaseFlow picks neither. "We don’t know" is recorded as exactly that:

/**
 * How one delivery ended. {@code UNKNOWN} means the provider may have accepted it:
 * nothing may repeat such an Action without a person saying so.
 */
record ActionResult(ExecutionStatus outcome, String externalReference, String errorCode) {
    // ...
}

A delivery ends in one of six statuses: PENDING, RUNNING, SUCCEEDED, FAILED, UNKNOWN and CANCELLED. UNKNOWN is not a kind of failure. It is a status of its own, with a rule of its own: nothing may send it again on its own.

FAILED or UNKNOWN: where the line is

The question that decides is simple: is it certain the provider did nothing?

Diagram of one delivery splitting into three outcomes: SUCCEEDED when the provider confirms, FAILED when the provider refuses and a retry is safe, UNKNOWN when the provider may have accepted it and only a person may decide to send it again
Figure 1. How a delivery ends: only a provider that certainly refused makes it FAILED; whenever the provider may have accepted it, it is UNKNOWN.

This is the response handling in SlackActionExecutor:

} catch (RestClientResponseException exception) {
    int status = exception.getStatusCode().value();
    if (exception.getStatusCode().is5xxServerError()) {
        // Slack may have posted the message before it failed to answer.
        log.warn("Slack answered HTTP {} for the note of release {}.", status, command.releaseId());
        return ActionResult.unknown(ActionResult.OUTCOME_UNKNOWN);
    }
    log.warn("Slack refused the note of release {} with HTTP {}.", command.releaseId(), status);
    return ActionResult.failed(REJECTED);
} catch (RestClientException exception) {
    // Slack may have posted the message before the connection failed.
    log.warn("Could not reach Slack with the note of release {}.", command.releaseId());
    return ActionResult.unknown(ActionResult.OUTCOME_UNKNOWN);
}
  • A 4xx means Slack read the request and refused it — a wrong URL, a revoked webhook. Nothing was posted for sure, so it is FAILED.

  • A 5xx means something went wrong on Slack’s side, but it may have gone wrong after the message was posted. That is UNKNOWN.

  • A dropped connection — a timeout, a network failure — is UNKNOWN too, because the request may have arrived.

ReleaseFlow does not follow redirects when delivering. A redirect response means the note went nowhere, so it is FAILED, not UNKNOWN.

The same question draws slightly different lines for each provider:

  • For email, a failure while composing the message is FAILED — nothing has left ReleaseFlow yet. But when SMTP does not confirm the send, it is UNKNOWN, because the server may have accepted the message for some of the recipients, or all of them.

  • Zendesk takes two calls: get an access token, then create the article. The first call never ends UNKNOWN — nothing has been published, so trying again later cannot create a duplicate. The second call can. Even when Zendesk answers 2xx but the response names no article, the result is still UNKNOWN: nobody can say for sure whether an article exists.

A dead worker becomes UNKNOWN, not PENDING

The delivery queue uses the same FOR UPDATE SKIP LOCKED pattern as the second post. But when it recovers stuck jobs, it does the opposite.

In the second post, a file-fetching job stuck for more than ten minutes went back to PENDING. Here, a delivery stuck for more than five minutes becomes UNKNOWN:

/**
 * Deliveries cannot be repeated safely, so an Action whose worker stopped is never
 * released back to PENDING: it becomes UNKNOWN and waits for a person.
 */
@Query(value = """
        SELECT id FROM automation_action_runs
        WHERE status = 'RUNNING' AND claimed_at < :staleBefore
        FOR UPDATE SKIP LOCKED
        """, nativeQuery = true)
List<UUID> findStaleRunningIds(@Param("staleBefore") Instant staleBefore);

The worker calls markUnknown on every delivery it finds, and settles the status of the whole run again. Same queue, same way of claiming jobs, opposite decisions — because the cost of doing it again is different. To me, this is the biggest lesson of the whole series: no retry policy is right for everything. You have to ask, for each kind of work: what does doing it again cost?

One more rule: the actions of a rule run in order, and the first action that ends FAILED or UNKNOWN stops every action behind it. Nobody wants the note posted to Slack while it is unclear whether the GitHub Release exists.

Only a person decides to send again

A FAILED run can be retried freely: nothing was sent, for sure. An UNKNOWN run needs a person to confirm that sending twice is acceptable:

public AutomationRunView retry(UUID organizationId, UUID runId, boolean confirmUnknown) {
    AutomationRun run = find(organizationId, runId);
    if (run.getStatus() == ExecutionStatus.UNKNOWN && !confirmUnknown) {
        throw AutomationConflictException.unknownNeedsConfirmation();
    }
    // ...
}

Without the confirmation, the API answers 409 automation_unknown_needs_confirmation. In the interface, the Run again button comes with a box that must be ticked: "This may deliver twice".

A run with status Outcome unknown: the Zendesk action carries the code execution_outcome_unknown, next to a This may deliver twice box with Run again and Cancel
Figure 2. A run whose Zendesk action has an unknown outcome: running it again requires confirming it may deliver twice.

A retry resets only the first action that stopped, along with the actions behind it that were waiting on it. An action that succeeded is never repeated.

Before pressing the button, the operator can do what ReleaseFlow cannot: open the Slack channel or the Help Center and check whether the message or article is already there.

When the provider lets you ask: GitHub Release

For a GitHub Release, ReleaseFlow can answer "did the last attempt get through?" itself, because a release can be read back by its tag. Every delivery writes a marker carrying its own ID at the end of the body:

private static String marker(UUID actionRunId) {
    return "<!-- releaseflow-action:" + actionRunId + " -->";
}

Before creating anything, the executor reads the release by tag:

// A release this Action Run wrote is the delivery; anything else is somebody's work.
private static ActionResult reconcile(GitHubRelease release, String marker) {
    String body = release.body() == null ? "" : release.body();
    return body.contains(marker)
            ? ActionResult.succeeded(release.htmlUrl())
            : ActionResult.failed(RELEASE_CONFLICT);
}
  • No release with that tag yet: create one, with the marker.

  • One exists, and its body carries this delivery’s marker: that was the previous attempt, and it got through. The result is SUCCEEDED, and nothing more is created.

  • One exists, but it is somebody else’s: FAILED with github_release_conflict. ReleaseFlow never overwrites a release somebody else made.

  • Somebody creates that exact tag between the read and the create: read again and decide as above.

So retrying an UNKNOWN GitHub Release action is safe: if the previous attempt arrived, this one just recognises it.

Not every provider allows that. A Confluence page also carries an HTML comment naming the run that wrote it, but ReleaseFlow does not read the page back. So a Notion or Confluence delivery that was never confirmed stays UNKNOWN until a person accepts a second page.

Delivery never blocks a release

There is one more place where things can be sent twice — or worse — and it sits right at publication. The project’s old version created its runs inside the publishing transaction. A rule pointing at an audience with no note threw an exception, and the publication rolled back with it. As ADR-0020 puts it, a release that is ready to publish is not the place to discover that somebody mistyped a Slack URL.

ReleaseFlow now writes exactly one outbox row when publishing:

@EventListener
void onReleasePublished(ReleasePublished event) {
    if (jobRepository.existsByReleaseId(event.releaseId())) {
        return;
    }
    jobRepository.save(new AutomationPublishJob(
            UUID.randomUUID(),
            event.organizationId(),
            event.projectId(),
            event.releaseId(),
            clock.instant().truncatedTo(ChronoUnit.MICROS)
    ));
}

This listener matches no rule, reads no note, and calls nothing outside. Only after the transaction commits does AutomationWorker turn that row into runs. A rule that cannot be carried out becomes a FAILED action in the history — for example release_note_missing — rather than a release that will not publish.

Stopping duplicates on the way in

Duplicate deliveries do not only come from retries. They also come from being triggered twice. Two unique indexes in migration V22 stop that:

-- A publication fires each matching Rule once, and a repeated manual request is the
-- same Run rather than a second delivery.
CREATE UNIQUE INDEX automation_runs_published_once
    ON automation_runs (rule_id, release_id)
    WHERE trigger_type = 'RELEASE_PUBLISHED';
CREATE UNIQUE INDEX automation_runs_request_once
    ON automation_runs (rule_id, request_id)
    WHERE request_id IS NOT NULL;

Each rule fires once per publication. The manual run form carries a request ID, so pressing the button twice or the browser resubmitting the form both return the same run. The same goes for other systems calling in through a signed webhook: the X-ReleaseFlow-Delivery header is a UUID that stays the same on a retry, so a webhook delivered again does not create a second run.

The trade-offs, plainly

  • UNKNOWN needs a person. A network incident can leave a few runs waiting for an operator to open them, check, and decide. That is extra work, and I accept it.

  • I chose "a delivery may be missing, but someone will know" over "a delivery may happen twice, and nobody will know". A missing release note can be sent late. An email that reached a thousand inboxes twice cannot be taken back.

  • The run history is terse. Errors are stored only as fixed codes, such as execution_outcome_unknown or github_release_conflict, never as a message from a provider or an exception — so nothing unexpected leaks into the interface. To learn more, for example the HTTP status Slack returned, an operator has to read the application log.

Closing the series

Four posts, three technical decisions, and looking back they share one idea:

  • A job queue in PostgreSQL — the machine repeats what can be repeated, and never silently skips work;

  • AI suggests, people decide — the machine drafts, and the rules and people keep the decisions;

  • and this post — when the machine is not sure, it says so, and lets a person decide.

That is also the point of the Reddit comment from the first post: automation for the repetitive parts, AI for the writing, and people to review and decide.

Thank you for reading this far. ReleaseFlow’s code is in the GitHub repository, the documentation is on the docs site, and the decisions in this post are recorded in ADR-0020. If you try it and run into an UNKNOWN delivery — or anything else — open a Discussion on GitHub.