So I was thinking about the architecture for this new service. It needs to handle about 10,000 requests per second and we probably want to use a message queue for the async processing. RabbitMQ seems like overkill, maybe Redis Streams would work. The main concern is that we need exactly-once processing semantics, and Redis Streams support consumer groups which could help. We decided to go with Redis Streams for the MVP and potentially migrate to Kafka later if we need better durability guarantees. The TODO list is: set up Redis Streams consumer, implement the worker process, add dead letter queue handling, and write integration tests. One open question is whether we should use a separate Redis instance for the queue or share the caching Redis instance.
