Time-Based and Proactive Agent Loops: Scheduling, Events, Deduplication, and Idempotency

A scheduled agent that opens two pull requests for the same dependency bump is not a model problem. It is a delivery-semantics problem wearing a model costume.

The trigger is the easy part. A cron expression, a webhook, a queue consumer — none of that is hard. The hard part is what the loop does when the same trigger arrives twice, when it arrives late, when two workers pick it up at once, or when the process dies halfway through a side effect. That is the design object this article is about.

I assume you already know the basic loop anatomy: state, action, observation, feedback, termination. What changes when the loop becomes proactive is that a machine decides when to start it, and machines are worse than humans at knowing whether they already asked.

Why Triggers Are Not Function Calls

The weak mental model is that a trigger is a function call. Something fires, your handler runs once, it returns. This model is comfortable and almost always wrong.

Timers fire twice across a leader election. Queues redeliver when an acknowledgment is lost. Webhooks retry with backoff when your endpoint is slow, and a 200 response is a promise you have to keep. The realistic default for every trigger source is at-least-once delivery. Exactly-once is not something you receive from infrastructure; it is a property you construct inside your own loop.

Four hazards follow from that, and they are the ones worth designing against explicitly:

• Duplicate delivery. The same logical event arrives more than once.
• Missed or late events. A webhook is dropped, a scheduler skips a window, a message sits in a backlog for an hour.
• Concurrent execution. Two workers process the same entity at the same time.
• Partial failure mid-loop. The model call succeeds, the side effect half-lands, the process dies.

The invariants that make these survivable are short enough to memorize:

Duplicate deliveries produce at most one committed local decision. External effects are exactly-once only when the effect system supports an idempotency key or transactional coupling; otherwise the loop provides at-least-once attempts plus reconciliation. Every run terminates under an explicit budget.

The single most common source of bugs here is conflating trigger identity with run identity. Trigger identity is what happened in the world: this repository got a new issue, this hour's reconciliation window closed. Run identity is what your agent did about it. A redelivered message has the same trigger identity and a different message id. If you key your deduplication on the message id, you will deduplicate nothing.

The Identity Map: Five Names for Five Jobs

Before the architecture, name the identifiers. Most deduplication bugs come from using one identifier for two jobs.

Identifier	Scope	Stable across retries?	Purpose
Logical event key	The real-world event	Yes	Deduplication and effect identity
Delivery ID	One message delivery	No	Tracing, ack bookkeeping
Run ID	One execution attempt	No	Logs, budgets, terminal reason
Effect key	One external side effect	Yes	Idempotency key for the effect system
Lease epoch	One claim on an entity	No	Fencing stale workers

The logical event key is the one that must survive redelivery. Everything else is per-attempt bookkeeping. When you find yourself asking "should this be idempotent?", you are really asking "which of these five am I holding?"

Anatomy of a Proactive Loop: Trigger, Gate, Run, Commit

I find it useful to split a proactive loop into four stages, each owning a different piece of state and absorbing a different failure.

Trigger. A timer, cron entry, queue message, webhook, or internal signal. It carries an event key and a payload. Nothing more. It does not decide anything.

Gate. Deduplication, rate limiting, budget checks, and concurrency admission all happen here, before any model call. This stage must be cheap and deterministic. A key comparison costs microseconds; a model call costs seconds and money. Never use the model as your deduplicator — a regex or a hash lookup is fast, deterministic, and does not hallucinate a false positive.

Run. The bounded agent loop itself, with its own turn and cost budget, writing progress to durable state as it goes.

Commit. The single point where side effects become visible. Everything before it must be replayable.

Stage	Owns	Idempotency responsibility	Observable signal	Failure if skipped
Trigger	Event key, payload	None	Delivery count, lag	—
Gate	Dedup key, lease, budget	Reject duplicates before work	Rejected/duplicate counter	Duplicate model calls, duplicate effects
Run	Loop state, checkpoints	Resumable or discardable	Turns, tokens, cost	Wasted spend, half-finished work
Commit	Effect ledger	At-most-once effect per key	Committed effect count	Duplicate or lost side effects

The gate is where most of the safety lives, and it is the stage people skip because it feels like ceremony. It is not ceremony. It is the difference between an agent you can leave running and an agent you have to watch.

Idempotency Keys and Effect Ledgers

An idempotency key is the identity of a logical event, expressed so that two deliveries of the same event produce the same key. Derive it from the event, not the delivery:

key = hash(source, entity_id, event_type, time_bucket)

source is the system that produced the event. entity_id is the thing being acted on. event_type is what happened. time_bucket is a coarse window — an hour, a day — that collapses repeated notifications about the same underlying change. Do not use the message id, the delivery timestamp, or the attempt number. All three change on redelivery.

The effect ledger is a durable record of (key, status, result, lease_epoch). The second delivery reads the ledger and returns the stored result instead of re-running the work. The write order matters, and there is no free lunch:

• Side effect, then ledger write. A crash between the two produces a duplicate effect on retry.
• Ledger write, then side effect. A crash between the two produces a lost effect — the ledger says done, the world says otherwise.

Pick per effect. For an upsert or a deterministic file write, the first order is fine because the effect is idempotent by construction. For a payment or an outbound send, you want the ledger write first plus a reconciliation step that can detect and repair the lost-effect window.

Some effects are idempotent by construction: upserts keyed on a natural key, deterministic file writes, creating a pull request keyed by branch name. Others need a compensating action: sends, payments, calls to third-party APIs you do not control. Classify every effect before you write the loop.

A Minimal State Machine, Not a Wrapper

The smallest useful implementation is not a claim-then-execute wrapper. It is a three-state ledger with an expiry, because a crashed worker must not permanently suppress recovery. The states are in_progress, done, and failed_retryable. A duplicate delivery does something different in each:

Ledger state	Duplicate delivery behavior
in_progress (lease valid)	Return "already running"; do not start a second run
in_progress (lease expired)	Reclaim with a new lease epoch; retry
done	Return the stored result; no work
failed_retryable	Retry if budget remains; otherwise escalate

def claim(key: str, owner: str, ttl_seconds: int) -> tuple[str, dict | None]:
    now = time.time()
    row = db.fetchone(
        "SELECT status, result, lease_epoch, lease_expires_at FROM effect_ledger WHERE key = ?",
        (key,),
    )
    if row is None:
        db.execute(
            "INSERT INTO effect_ledger (key, status, lease_epoch, lease_owner, lease_expires_at) "
            "VALUES (?, 'in_progress', 1, ?, ?)",
            (key, owner, now + ttl_seconds),
        )
        return "claimed", None
    if row["status"] == "done":
        return "done", row["result"]
    if row["status"] == "in_progress" and row["lease_expires_at"] > now:
        return "busy", None
    # expired or failed_retryable: reclaim with a new epoch
    new_epoch = row["lease_epoch"] + 1
    db.execute(
        "UPDATE effect_ledger SET status = 'in_progress', lease_epoch = ?, "
        "lease_owner = ?, lease_expires_at = ? WHERE key = ?",
        (new_epoch, owner, now + ttl_seconds, key),
    )
    return "claimed", None

def commit(key: str, epoch: int, result: dict) -> bool:
    rows = db.execute(
        "UPDATE effect_ledger SET status = 'done', result = ? "
        "WHERE key = ? AND lease_epoch = ? AND status = 'in_progress'",
        (json.dumps(result), key, epoch),
    )
    return rows == 1

The unique constraint on key does the deduplication. The lease expiry makes abandoned claims recoverable. The lease_epoch check in commit is what makes a stale worker's write fail. This is the whole mechanism, and it fails in three predictable ways: effects on systems you do not control, non-idempotent third-party APIs, and keys that are too coarse (dropping legitimate distinct events) or too fine (never deduplicating).

One Failure Trace, End to End

Walk the crash window once with the fields above, because this is where the design either holds or leaks.

1. Delivery A arrives. Gate calls claim(key, owner=w1, ttl=60). Ledger row: in_progress, epoch 1, owner w1.
2. Worker w1 calls the model, gets a result, then crashes before commit.
3. Lease expires. Delivery B arrives (redelivery of the same logical event). Gate sees in_progress with an expired lease, reclaims with epoch 2, owner w2.
4. Worker w2 runs the loop, calls commit(key, epoch=2, result). The lease_epoch = 2 predicate matches. Row becomes done.
5. Worker w1 restarts (or its process was merely paused) and calls commit(key, epoch=1, result). The predicate fails — zero rows updated. w1 discards its result.
6. Delivery C arrives. Gate sees done and returns the stored result. No second effect.

Now the variant where the crash happens after commit but before the message is acknowledged. Delivery B arrives, the gate finds done, and returns the stored result. The acknowledgment is lost, but the effect is not duplicated. That is the whole point of the ledger.

Scheduling Semantics: Catch-Up, Drift, and Missed Windows

Scheduling looks like a solved problem until you ask what happens when a run takes longer than the interval.

Fixed-rate scheduling fires on a wall-clock cadence. If a run exceeds the interval, the next fire overlaps it. Fixed-delay scheduling waits a fixed gap after the previous run finishes. It never overlaps, but it silently stretches the period — a job you thought ran hourly drifts to every ninety minutes under load. Neither is wrong; you just have to know which one you picked.

Catch-up policy is a product decision, not a scheduler default. When the scheduler was down for three hours, do you run every missed window, run only the latest, or skip and record the gap? The decision rule I use:

If the work is a reconciliation of current state, catch up once. If the work is a per-interval record, catch up all windows.

Reconciling a repository against its desired state does not care that you missed three windows — the current state is the current state. Writing an hourly metrics row does care, because the missing hours are the data.

Overlap policy is the same kind of decision. Skip-if-running is the safe default. Queueing is correct when every trigger carries distinct work. Cancel-and-restart is correct when only the latest state matters. State which one your agent uses, in the code, so the next person does not have to infer it from behavior.

Two more traps. Time zones and DST transitions mean the same cron expression can fire twice or zero times on a transition day — schedule in UTC and convert at the edges. And clock skew across workers means two machines can disagree about whether the window has closed. A schedule that fires every five minutes with a ten-minute worst-case run is a concurrency bug waiting for a busy day.

Event-Driven Triggers: Ordering, Bursts, and Backpressure

Events add hazards that timers do not have.

Out-of-order delivery. Two events for the same entity can arrive reversed. The agent then acts on stale state — it "fixes" a problem that a later event already resolved. Use per-entity ordering keys or version checks so the loop can detect that it is looking at an old view.

Bursts and coalescing. A busy entity can generate fifty events in a minute. Collapse them into one run within a window, and record which events were absorbed so the run's inputs are auditable. Coalescing is deduplication with a wider aperture.

Backpressure. Bounded queues, admission control, and an explicit shed policy. The failure mode of an unbounded queue is not a crash; it is a growing lag that nobody notices until the agent is processing yesterday's events today. Decide what you drop and say so.

Webhooks deserve their own paragraph. They retry with backoff, they should be signature-verified, and a 200 response is a promise that you have durably accepted the event. If you return 200 before the event is persisted, you have lied to the sender, and the retry you were trying to avoid will not come — the event is simply gone. Durable acceptance means the event is in storage before the response leaves your process.

Fan-in and fan-out both need a deduplication scope decision. One event triggering many agents means each agent needs its own key namespace. Many events triggering one agent means the agent needs to know which events belong to the same logical unit of work.

Event-driven is not always the right answer. Low-frequency, human-paced work is usually simpler as a scheduled poll. If you are checking a dashboard once an hour, a poll is less machinery than a webhook pipeline, and it fails more visibly.

Concurrency, Leases, and Crash Recovery

Two workers processing the same entity is the failure that produces the strangest bugs, because both runs look correct in isolation.

Use leases with expiry, not locks. A crashed worker must not hold an entity forever. The lease is renewable, has a deadline, and is observable — you can query which entities are currently leased and by whom.

Fencing is what makes leases safe. A worker that lost its lease must not commit. The lease_epoch predicate in commit is the fence: a stale worker's write is rejected even if it finishes its work after the lease expired. If the update affects zero rows, the worker lost its lease and must discard its result. This is the check that turns "usually fine" into "safe under concurrency."

Checkpoint loop state so a resumed run continues rather than restarts, and decide which steps are safe to redo. Heartbeats distinguish a slow run from a dead one before you reclaim its lease — reclaiming too eagerly is how you get two workers on the same entity.

Observability that actually helps is per-run and includes the trigger key, lease id, attempt count, budget consumed, and terminal reason. Without those five fields, debugging a duplicate effect is archaeology.

Termination and Budgets for Loops That Never Stop

A proactive loop has no human present to stop it, so stopping conditions have to be explicit.

Two budget layers are needed. A per-run budget caps turns, tokens, wall clock, and cost for a single execution. A per-period budget caps runs per hour and spend per day across the fleet. The first prevents a runaway loop; the second prevents a thousand well-behaved loops from collectively burning the month's budget.

Terminal reasons must be recorded and distinguishable: success, budget exhausted, no-op, blocked, escalated, cancelled. The no-op case matters more than people expect. A proactive agent that finds nothing to do should exit cheaply and say so, not spin looking for work. If your traces cannot tell a no-op from a success, you cannot tell whether the agent is doing anything at all.

Escalation is a termination mode, not a failure. Hand off to a human with preserved state rather than retrying forever. And build the kill switch and drain path before you need them — stopping a fleet of scheduled agents without leaving half-committed effects is a design property, not an operational improvisation.

Finally, the honest boundary: if the work is user-initiated and low-frequency, a turn-based loop with a manual trigger is less machinery for the same outcome. Proactive loops earn their complexity when the work is recurring, machine-paced, and would otherwise wait on a human to notice.

Build Order and the One Question That Gates Everything

Build in this order. Pick one recurring job. Give it an idempotency key derived from the logical event. Add an effect ledger with a unique constraint and a three-state machine. Add a lease with an epoch fence. Only then add the schedule. Each step is testable on its own, and the schedule is last because it is the least interesting part.

The decision rule that gates the whole design:

If you cannot name the key that makes a duplicate delivery harmless, you are not ready to put the agent on a timer.

The next experiment is small and worth running before you trust anything. Split it into two tests, because they prove different guarantees.

Test 1 — side-effect-free run. Replay the same trigger twice against a staging ledger and confirm the second run is a no-op that returns the stored result. Then kill the worker between the model call and commit, and confirm the next delivery reclaims the expired lease, re-runs, and commits exactly once. Success here proves the local ledger and lease protocol.

Test 2 — external effect. Point the loop at a provider that supports an idempotency key (a payment API, a send API with a client-supplied key). Pass the effect key through. Kill the worker after the provider call but before commit, then let the retry run. Success means the provider deduplicated the second call. If the provider has no idempotency key, the test cannot prove exactly-once — the correct outcome is an explicit reconciliation check that detects the ambiguous window and either repairs or escalates. Treating a passing local ledger test as proof of external-effect safety is the mistake this split is designed to prevent.

If both tests pass, you have a proactive loop. If either fails, you have a demo.

The adjacent concept to study next is how evaluators and feedback contracts decide whether a completed run actually succeeded — because a loop that terminates cleanly is not the same as a loop that did the right thing.