| # Idempotency |
|
|
| ## Context and Problem Statement |
|
|
| Event-driven ingestion requires idempotency to handle message retries, ensuring reliable processing. |
| Key considerations are: |
|
|
| - Distributed systems often guarantee least-once delivery. |
| - Event replays may be necessary for failure and recovery scenarios. |
| - CloudEvents are unique by ID + Source properties. |
|
|
| Neither Kafka nor ksqlDB natively supports message idempotency. |
|
|
| ## Considered Options |
|
|
| To achieve idempotency we discovered event deduplication options both on producer (API Server) and consumer (ksqlDB) side: |
|
|
| 1. Redis with key expiration |
| 1. Redis with bloom filter |
| 1. Deduplication in ksqlDB via windowed tables |
|
|
| ## Decision Outcome |
|
|
| To maintain strong consistency and avoid complex multi-phase commits and state management, we chose deduplication via ksqlDB. |
|
|
| - Idempotency via deduplication |
| - Deduplication in ksqlDB via windowed tables |
| - Events are unique by ID + Source |
| - 32 days default deduplication window |
|
|
| ### Consequences |
|
|
| - Pros: No need to run Redis. |
| - Pros: Strong consistency can be guaranteed. |
| - Cons: Increased Kafka storage for deduplication window. |
| - Cons: Longer retention window may impact backup and recovery time. |
|
|
| ## Pros and Cons of the Options |
|
|
| ### Redis with key expiration |
|
|
| - Pros: Simple implementation. |
| - Cons: Inconsistent with Kafka Producer. |
| - Cons: Requires Redis. |
|
|
| ### Redis with bloom filter |
|
|
| - Same tradeoffs as Redis with key expiration. |
| - Pros: Efficient storage. |
| - Cons: Possible false positives leading to dropped events. |
|
|
| ### Deduplication in ksqlDB via windowed tables |
|
|
| Deduplicating Kafka messages with ksqlDB requires an extra table where we count how many times we have seen a specific event. Then we only include events in streaming processing at their first occurrence. |
|
|
| See [How to find distinct values in a stream of events](https://developer.confluent.io/tutorials/finding-distinct-events/ksql.html) for more details. |
|
|
| - Pros: No separate system required. |
| - Pros: Guaranteed consistency. |
| - Cons: Increased Kafka storage for deduplication window. |
| - Cons: Longer retention window may impact backup and recovery time. |
|
|
