The first version of our credential handling was the version everyone writes first: try to log in, and if it fails, grab another credential from the pool and try again. It worked fine at small scale. At a few thousand accounts running continuously, it turned into a machine for getting accounts banned.
Here's why the naive version breaks, and what I replaced it with.
Why "Just Retry With a Different One" Falls Apart
The naive approach treats every credential as interchangeable and every failure as the same kind of failure. Neither of those is true at scale.
Not every failure means "this credential is bad." A login failure could mean the password actually changed, or it could mean the target's login page hiccuped, or it could mean you're mid-CAPTCHA-challenge and the account is completely fine. Treat all three the same way — burn the credential and grab the next one — and you end up cycling through your entire pool for problems that had nothing to do with the credentials themselves.
Worse, with no memory of how recently or how often a credential was used, "just grab another one" tends to concentrate load on whichever accounts happen to be fastest to check out of the pool. Those accounts get hammered, they trip the target's abuse detection, and they get flagged or banned — which shrinks your usable pool, which increases load on what's left, which bans more accounts. It's a feedback loop that quietly kills your account inventory from the inside, and by the time someone notices the pool is 40% smaller than last month, the damage compounds for weeks before anyone looks at usage patterns instead of just the failure logs.
What was actually needed was persistent memory of what state each credential is in, and a policy for moving between those states — which is another way of saying: a state machine.
Modeling Credentials as a State Machine
A state machine forces you to answer two questions honestly, upfront: what are all the states a credential can actually be in, and what event is allowed to move it from one state to another. Once you write those down, most of the "why did this account get banned" postmortems turn out to be a transition nobody had actually defined — the code just did something ad hoc instead.
// credential-states.ts
export type CredentialState =
| 'ACTIVE' // healthy, eligible for use right now
| 'IN_USE' // checked out, currently running a task
| 'COOLING_DOWN' // used recently, resting before eligible again
| 'CHALLENGE' // hit a CAPTCHA or step-up auth, needs attention
| 'LOCKED_OUT' // target rejected credentials, needs rotation
| 'DEAD'; // permanently unusable — banned or retired
export type CredentialEvent =
| { type: 'CHECK_OUT' }
| { type: 'TASK_SUCCEEDED' }
| { type: 'TASK_FAILED'; reason: 'timeout' | 'network' }
| { type: 'AUTH_REJECTED' }
| { type: 'CHALLENGE_DETECTED' }
| { type: 'CHALLENGE_RESOLVED' }
| { type: 'COOLDOWN_ELAPSED' }
| { type: 'MARKED_DEAD' };
The transition function is the whole policy in one place — the part that's genuinely worth arguing about in a design review, instead of being buried across a dozen call sites:
// credential-transitions.ts
export function transition(
current: CredentialState,
event: CredentialEvent
): CredentialState {
switch (current) {
case 'ACTIVE':
if (event.type === 'CHECK_OUT') return 'IN_USE';
return current;
case 'IN_USE':
switch (event.type) {
case 'TASK_SUCCEEDED': return 'COOLING_DOWN';
case 'TASK_FAILED':
// a transient failure is not the credential's fault —
// it goes back to cooling down, not to lockout
return 'COOLING_DOWN';
case 'AUTH_REJECTED': return 'LOCKED_OUT';
case 'CHALLENGE_DETECTED': return 'CHALLENGE';
default: return current;
}
case 'COOLING_DOWN':
if (event.type === 'COOLDOWN_ELAPSED') return 'ACTIVE';
return current;
case 'CHALLENGE':
if (event.type === 'CHALLENGE_RESOLVED') return 'COOLING_DOWN';
if (event.type === 'MARKED_DEAD') return 'DEAD';
return current;
case 'LOCKED_OUT':
// requires an external rotation event — a human or a rotation
// job supplying a new credential — not an automatic transition
if (event.type === 'MARKED_DEAD') return 'DEAD';
return current;
case 'DEAD':
return 'DEAD'; // terminal, no way back
}
}
Two decisions in there are doing most of the work. First, TASK_FAILED doesn't jump straight to LOCKED_OUT — a timeout or a flaky network call says nothing about whether the credential itself is good, so it goes back through COOLING_DOWN instead of getting punished for the target's infrastructure having a bad moment. Second, LOCKED_OUT is a dead end with no self-recovery. Getting out of it requires an explicit external event, because an automated retry loop that can talk itself out of a lockout state is exactly the kind of thing that gets an account permanently banned instead of temporarily flagged.
Persisting State Without Losing Your Mind
The state machine is only useful if the state actually survives process restarts and is visible across every worker pulling from the same pool. That means a database row per credential, not an in-memory map:
CREATE TABLE bot_credentials (
id UUID PRIMARY KEY,
target_id TEXT NOT NULL,
state TEXT NOT NULL DEFAULT 'ACTIVE',
last_used_at TIMESTAMPTZ,
cooldown_until TIMESTAMPTZ,
failure_count INT NOT NULL DEFAULT 0,
secret_ref TEXT NOT NULL, -- pointer into the secrets store, never the secret itself
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX idx_bot_credentials_selection
ON bot_credentials (target_id, state, last_used_at);
secret_ref is deliberate — it's a reference into a proper secrets manager, not the password or token itself. The credentials table tracks lifecycle, not secrets. Those are two different concerns with two very different access-control requirements, and collapsing them into one table is how a database backup ends up being a plaintext credential dump.
Selecting the next credential to use is just a query against that index — pull from ACTIVE, order by last_used_at ascending so the least-recently-used account goes first:
SELECT id, secret_ref
FROM bot_credentials
WHERE target_id = $1 AND state = 'ACTIVE'
ORDER BY last_used_at ASC NULLS FIRST
LIMIT 1
FOR UPDATE SKIP LOCKED;
FOR UPDATE SKIP LOCKED is the piece that makes this safe under real concurrency — two workers racing for a credential at the same moment won't both grab the same row; the second one just skips past it to the next best candidate instead of blocking or double-checking-out the same account. Round-robin by recency, enforced at the query level, is what actually prevents the "same five accounts get all the traffic" problem instead of just hoping the application code behaves.
Security Hygiene That's Easy to Skip Under Deadline Pressure
A few rules that are cheap to follow and expensive to violate:
Never log the secret, ever — not even at debug level. Log the credential's id and target_id, never the password or token. It's tempting to leave a console.log(credential) in during a debugging session and forget to take it out. Structured loggers that redact known secret fields by name are worth setting up once, so a stray log line can't leak a live credential into a log aggregator that a dozen people have read access to.
Rotate secrets through the secrets manager, not through your own code. The credentials table stores a reference; the actual rotation — generating a new value, updating it at the source, invalidating the old one — is a job for a real secrets manager with its own audit trail. Writing your own rotation logic on top of a plain database column is how you end up with secrets that are "rotated" in name but still valid for the old value for an indeterminate window.
Treat LOCKED_OUT as a signal, not just a state. A credential landing in LOCKED_OUT is worth alerting on above a certain rate, because a spike usually means something upstream changed — a target rotated its detection, a shared IP got flagged, a whole batch of credentials went stale at once. The state machine tells you what happened to each credential individually; someone still has to watch the aggregate rate of ACTIVE → LOCKED_OUT transitions to catch the systemic version of the problem.
The Actual Payoff
None of this is exotic engineering — it's a state machine, a database table, and a query with a locking hint. What it bought us was boring in the best way: credential health became something you could query and graph instead of something you inferred from a spike in failed jobs three hours after the fact. Bans dropped because load spread evenly across the pool instead of concentrating on whoever happened to be fastest to check out. And when something did go wrong, the state history on the row told you exactly what happened to that credential and when, instead of someone reconstructing it from scattered log lines across three services.
The naive version wasn't wrong because retrying is a bad idea. It was wrong because it had no memory. A state machine is just retrying with memory — and at a few thousand accounts, memory is the whole difference between a system that self-heals and one that quietly eats its own inventory.
