Celery
- It allows you to run tasks asynchronously in the background, separate from the main application process. This is useful for tasks that take a long time to complete, such as sending email, processing images, or scraping websites, as it allows the main application process to continue handling other requests while the task is being executed.
- To work with Celery, we also need to install RabbitMQ because Celery requires an external solution to send and receive messages. Those solutions are called message brokers. Currently, Celery supports RabbitMQ, Redis, and Amazon SQS as message broker solutions
- Also supporting task scheduling.
- More
Kafka
- Distributed, fault tolerant, high throughput pub-sub messaging system.
RabbitMQ
- Message broker, better for low latency message delivery, while Kafka is good for Big Data.
- More commonly, a client sends a request to a service, the service then sends a message to the queue, and another service picks up that message and processes it. This allows for proper decoupling and separation of concerns between different parts of the system.
Redis
- In-memory data structure store, used as a database, cache, and message broker. RabbitMQ is better as a MQ, as it guarantees message delivery with the help of message durability and acknowledgments.