The story of Publisher/Subscriber pattern....

Hello Folks,

After a long time, we are back with another engineering topic: publisher/subscriber.

So now the first question arises why do even we need a publisher/subscriber when we can do all the things via our native request/response architecture?

Let's understand this with an example:-

Suppose you have an API and its route like /users having GET method. Let's see the request/response architecture

Below is the flow of how things will be working:-

1. The user will make a request to the Web Server

2. The Web server going to invoke the particular API route i.e.; /users: GET

3. The route containing special business logic will do some Database operations

4. After doing the database operation it gonna return the response to the Web server

5. And the web server gonna return to the user client

The above request/response pattern looks very simple and has no problem at all now let's do some complicated operations with request/response architecture with microservices.

Below is the flow of how things will be working:-

1. The user will make a request to upload images to the web server.

2. The request will first go to the API gateway

3. The API gateway will decide which service to invoke on the following endpoint /upload/image: POST

4. Now firstly the user gonna upload his zip folder containing a bunch of images

5. Once this service uploaded the zip successfully now it will call the unzipping of the uploaded zip file service

6. Once this service is completed it's gonna call the thumbnail service which will be creating the thumbnail of all the images present in the unzipped folder.

7. The most important point here is to consider that while the above 6 steps are going on the user needs to wait and see the annoying loader coming up on the screen.

8. After the thumbnail service has executed successfully now it will be invoking the images/thumbnail service

9. Once it's done it's gonna save the uploaded image URL and its metadata into the database.

10. Finally! Finally! We are done with our image uploading process and done with making users annoying completely :-(

If you see the above process its have the following cons:-

1. Taking too much time to process

2. The user client needs to wait

3. And what if any service got failed so how are we gonna handle the failure or rollback?

In order to handle all the above problems let's call our savior!

Well, it's not superman;-P! But the publisher/subscriber pattern!

The publisher/subscriber pattern can include both queues and pub/sub-methods just like SNS (pub/sub) and SQS(queue). In our case let's say we will be gonna use apache Kafka or RabbitMQ

"The publisher-subscriber simply says that there will be some topics on which you gonna publish your metadata. Once the data has been successfully published then it's gonna invoke the associated subscriber"

But before implementing the publisher/subscriber pattern we need to optimize our service flow so that the user didn't need to wait so long!

As per our use case, the zip files can be very small varying to very large. So in this case we can take help with the background uploading process or asynchronous uploading process.

Let's see how

Below is the flow

1. If you see in the step-1 block the user gonna simply submit the request to the API gateway that he is gonna upload a zip

2. And once the uploading has been done successfully the API Gateway gonna immediately respond to the user client stating "Your file is in uploading progress"

3. Now after this the step-2 block will be executed in the background

4. And all the processes which we discussed earlier will be done

But if you see here still we are lacking in retry methodology now for handling this case we can use publisher/subscriber. Let's see the below flow

1. If you see in the step-1 block the user gonna simply submit the request to the API gateway that he is gonna upload a zip

2. And once the uploading has been done successfully the API Gateway gonna immediately respond to the user client stating "Your file is in uploading progress"

3. Once the above two steps are done it's gonna publish to a topic name as "unzipping upload image topic"

4. Note:- On each above topics we have a dedicated subscriber

5. Once we have published the date to "unzipping upload image topic" then it's gonna invoke the "unzipping uploading service" which will be working in an async fashion.

6. Once it's done we gonna publish to another topic which is "Create thumbnail topic" which gonna invoke the "create thumbnail service"

7. And suppose if "create thumbnail service" got failed due to some reason then we can again publish the data "Create thumbnail topic" and retry for the same

8. and so on for other topics too the above steps will be applicable

9. At last, once all the topics have been invoked and executed successfully then we gonna publish to the "Notification topic" which gonna invoke a dedicated "Notification service" which gonna send a notification to the user stating "woohoo! Your image has been uploaded successfully"

So the summary of the above is

"Whenever you are having a long-running process or having a lot of service calls in a single service then try to use publisher/subscriber pattern and try to process things in an async manner"

That's all for this blog

Happy coding!