Payment Gateway & Ledger Service

The naive version of a payments table has a status column that gets overwritten. That design loses history, permits illegal jumps, and makes disputes unanswerable. Instead a payment moves through explicit states, and each transition is appended to an audit trail: who triggered it, when, and what the previous state was.

The practical benefit is that illegal transitions become impossible rather than unlikely. A settled payment cannot be captured again, because the state machine has no such edge.

This is the defining problem of a payments API. The client has no idea whether its first request succeeded — the response may have been lost after the charge committed — so it retries. Without protection, the retry is a second charge, and the customer finds out on their statement.

What I built

A servlet filter in front of the payment endpoints, backed by Redis, that treats the client's idempotency key as the unit of truth:

• On arrival, the filter attempts to acquire a lock on the key. If it can't, a request with that key is already in flight and the duplicate is rejected rather than queued.
• If the key has a stored response, that response is replayed verbatim — same body, same status. The second click gets the first charge's answer.
• If the key is new, the request proceeds, and the response is written back against the key before the lock is released.

The important detail is that the lock is taken before any work happens, not after. An idempotency check that runs after the charge is a race with a friendlier name.

Once a payment is captured, other things must happen: a ledger entry, a receipt, a webhook to the merchant. The obvious implementation publishes to Kafka right after the database commit. The obvious implementation is a dual-write bug.

If the service dies between the commit and the publish, the payment exists and the event does not. Nothing retries it, because nothing knows it was owed. Move the publish inside the transaction instead and you get the mirror-image failure: the event is sent for a transaction that then rolls back.

The transactional outbox

The fix is to make the event part of the same commit. The payment row and an outbox row are written in one transaction, so they succeed or fail together. A separate relay polls the outbox, publishes to Kafka, and marks rows dispatched — retrying until the broker acknowledges.

This buys at-least-once delivery, which means consumers must tolerate duplicates. That's a fair trade: duplicate delivery is a solvable problem on the consumer side, whereas a lost event is unrecoverable.

I wanted the card-handling path to reflect how a real gateway is constrained, so I built a PCI-inspired vault rather than storing anything usable. Card details are encrypted with AES-GCM under envelope encryption — a per-record data key, itself encrypted by a master key — and the rest of the system only ever sees a token.

Because AES-GCM is authenticated, tampering with ciphertext fails verification instead of decrypting into garbage. Rate limiting on the sensitive endpoints runs as a Redis Lua script, so the check-and-increment happens atomically inside Redis rather than as two round trips a burst of traffic can slip between.

What I'd do next

• Replace outbox polling with logical decoding off the WAL, so delivery latency stops depending on poll interval.
• Add a reconciliation job that walks the ledger against the payment states and reports drift, rather than trusting that it can't happen.
• Move key management to a real KMS, since a master key in application config is the weakest link in the current design.