We evaluated its codebase again and quickly realized that it’d be a great platform to build upon.
Before starting the project from scratch, we rediscovered Bee-Queue, having first seen it a couple years ago. With this in mind, we decided to pursue building our own lightweight queue based on Redis that did only what we needed it to, in order to make performance a priority. We’re big fans of Redis at Mixmax so everyone on the team has experience working with it. Redis also gives us the benefit of keeping our traffic inside our network VPC – a core security requirement for all our sensitive data. With Redis, we can easily scale our Redis cluster to meet our ever-growing demand, now at a tens of million of jobs an hour. We decided to stick with Redis over other messaging frameworks (such as RabbitMQ) because of how easy it is to work with and how much performance we could get for a low cost. We were faced with the decision to move entirely to a different queue such as RabbitMQ or start over and write our own simple high-performance Redis-based queue. That race condition was fixed in Bull v3, but unfortunately v3 also experienced a significant performance regression over v2. Bull served us well for a year, but there was a race condition that resulted in some jobs being double processed, causing major problems at scale. So, why another job queue in the Node.js ecosystem? Up until now, Mixmax had been using a similar queue, Bull. Here is how Bee-Queue v1.0 compares to other Redis-based job queues in the Node ecosystem (including its own prior v0.x release): Scaling is as simple as running more workers, and Bee-Queue even has a useful interactive dashboard that we use to visually monitor job processing. Mixmax is using Bee-Queue in production to process tens of millions of jobs each day.īee-Queue is meant to power a distributed worker pool and was built with short, real-time jobs in mind. With the help of Bee-Queue’s original author, Lewis Ellis, we revived the project to make it the fastest and most robust Redis-based distributed queue in the Node ecosystem. Original post by Eli Skeggs excited to announce the v1.0 release of Bee-Queue : a fast, lightweight, robust Redis-backed job queue for Node.js.
0 kommentar(er)
