The word around the water cooler is that a queue has yet to be created that I don’t like. Whether it’s RabbitMQ, AWS SNS/SQS, or Google Cloud Pub-Sub, regardless of the implementation, I love queues to death, gobble, gobble...I’ll eat ‘em up. I mean, what’s not to like?
Not too long ago at a due diligence review, I was presenting my idea for a mission-critical enterprise architecture. The Pub-Sub pattern played a critical role in my thinking. So I did my dog and pony presentation and things seemed to have gone swimmingly well, then later that day one of the attendees at the presentation stopped by my desk and told me, “I like your thinking, but I gotta tell you, I hate queues. I think they suck.”
I was dumbstruck. My world shook. I felt as if I were a five year old and someone had just told me there was no Santa Claus, and I could not imagine a world with no Santa Claus.
My immediate reaction was to flip the bozo bit and dismiss his comment as one made by a guy who had no idea as to what he was talking about, but, I knew the background of the person. He was no dope and he had a boatload of experience. He’s worked in telecom for a very long time, on very large systems. Given his background and expertise, I’d be dumb not to consider his position. Going against every impulse had to defend my ego, I said, “Oh, why?”
And he told me.
Using a Queue is Lazy
“Basically using a message queue to facilitate interservice communication is lazy,” he said. “You should just have one service send a HTTP POST to the other service that wants the information. For a little more work, you get a lot more bang. Let me show you on the whiteboard.”
Figure 1 shows what he drew.
Figure 1: A typical Pub-Sub pattern using a message queue subscribed to a topic.
“In a typical Pub-Sub pattern you have a service that sends a message to a topic (1). If there are no queues subscribed to the topic, the messages accumulate, eating storage resource. Yes, you can configure a topic to delete a message after a time, but still, the topic is responsible to store the message.
“Luckily, in this case I’ve diagrammed, we have a queue subscribed to the topic. The topic could be a list that's populating quickly such as real-time stock transactions for a stock brokerage firm. The topic will send a copy of the message to the subscribers it knows about (2). For example, the brokerage's accounting system as well as another system belonging to the brokerage's official auditor. Then, once all subscribers get the message, the topic will flush the message.
“Now we have the message sitting in the queue waiting for the Service bound to the queue to pull the message (3). That’s a lot of work. Not only do we have to devote resources to getting the message from the publishing service to the ultimate consuming service(s), but the consuming service has to create the queue and then subscribe the queue to the topic.