architecture - how do you scale a system from 1 request/sec to thousands of requests/sec

Question

General System design question: There is an application that queries DB with lat,long for address and makes an API call to format the address. Currently these requests are processed at a speed of 1 request per second, how do you scale up our application to 1000 requests/second.

question from:https://stackoverflow.com/questions/65931955/how-do-you-scale-a-system-from-1-request-sec-to-thousands-of-requests-sec

Answer 1

You need to make your system distributed. You embed your app in a microservice, and then you horizontally scale it out to reach the traffic levels you need. More specifically:

• The application will have a limited number requests that it can handle (let's say 100 requests/second), so you need to add 10 replicas of your app in order to handle 1000 requests / second.
• You also need a load balancer in front of your app, to distribute the load evenly across replicas
• Once you scale up even more, the DB will become a bottleneck, so you might need to think about optimizations in that area, for example:
  • Design the DB on a NoSQL model, because the flat model of lat/long data maps really well on a NoSQL storage format, which comes with a lot of advantages in terms of scalability and distributivity.
• Also, the API you use to format the address will also become a bottleneck once you scale the app, so if it's a third party you need to check the throughput limits and consider them during the scale-up, or if it's in your control, you can follow a similar procedure to scale it up.