Metrics Driven Development – What I did to reduce AWS EC2 costs to 27% and improve 25% in latency

Recently, I did some work related to auto-scaling and performance tuning. As a result, the costs reduced to 27% and service latency improved 25%.

Overall Instance Count And Service Latency Change
Overall Instance Count And Service Latency Change

Takeaways

  • React Server Side Render performs not good under Nodejs Cluster, consider using a reverse proxy, e.g. Nginx
  • React V16 Server Side Render performs much faster than V15, 40% in our case
  • Use smaller instances to get better scaling granularity if possible, e.g. change C4.2xLarge to C4.Large
  • AWS t2.large performs 3 times slower than C4.large on React Server Side Render
  • AWS Lambda performs 3 times slower than C4.large on React Server Side Render
  • There’s a race condition in Nginx http upstream keepalive module which generates 502 Bad Gateway errors (104 connection reset by peer)

Background

Here’s the background of the service before optimization:

Improve performance of React Server Side Render by warming up service

Background

We’re using ElasticBeanstalk Blue/Green deployment by swapping DNS, and in front of the deployed services, there’s a company level Nginx to forward requests to different services.
The TTL of DNS entry set to 1 minute, and Nginx will cache the resolved name entries until it expired. Each time after we deploy, all the requests hit the new environment after the DNS cache expired in Nginx.

React Server Render time without warming up

The response time increases a lot in the following a couple of minutes and becomes stable after 5 minutes. Because the response time impacted by upstream services, it’s better to analyze and improve react server render time which is a sync method call and not involve IO operations.

Here’s the initial reactServerRender time for Home Page and Search Result Page:


For the Home page, it took 2-3 minutes for the reactRender time reduced from 450 – 550 ms to 120 ms

Continue reading “Improve performance of React Server Side Render by warming up service”