Category: Linux

Linux

Metrics Driven Development – What I did to reduce AWS EC2 costs to 27% and improve 25% in latency

Recently, I did some work related to auto-scaling and performance tuning. As a result, the costs reduced to 27% and service latency improved 25%.

Overall Instance Count And Service Latency Change
Overall Instance Count And Service Latency Change

Takeaways

  • React Server Side Render performs not good under Nodejs Cluster, consider using a reverse proxy, e.g. Nginx
  • React V16 Server Side Render performs much faster than V15, 40% in our case
  • Use smaller instances to get better scaling granularity if possible, e.g. change C4.2xLarge to C4.Large
  • AWS t2.large performs 3 times slower than C4.large on React Server Side Render
  • AWS Lambda performs 3 times slower than C4.large on React Server Side Render
  • There’s a race condition in Nginx http upstream keepalive module which generates 502 Bad Gateway errors (104 connection reset by peer)

Background

Here’s the background of the service before optimization:

Analyze ‘Connection reset’ error in Nginx upstream with keep-alive enabled

What? Connection reset by peer?

We are running Node.js web services behind AWS Classic Load Balancer. I noticed that many 502 errors after I migrate AWS Classic Load Balancer to Application Load Balancer. In order to understand what happened, I added Nginx in front of the Node.js web server, and then found that there are more than 100 ‘connection reset’ errors everyday in Nginx logs.

Here are some example logs:

2017/11/12 06:11:15 [error] 7#7: *2904 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 172.18.0.1, server: localhost, request: "GET /_healthcheck HTTP/1.1", upstream: "http://172.18.0.2:8000/_healthcheck", host: "localhost"
2017/11/12 06:11:27 [error] 7#7: *2950 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 172.18.0.1, server: localhost, request: "GET /_healthcheck HTTP/1.1", upstream: "http://172.18.0.2:8000/_healthcheck", host: "localhost"
2017/11/12 06:11:31 [error] 7#7: *2962 upstream prematurely closed connection while reading response header from upstream, client: 172.18.0.1, server: localhost, request: "GET /_healthcheck HTTP/1.1", upstream: "http://172.18.0.2:8000/_healthcheck", host: "localhost"
2017/11/12 06:11:44 [error] 7#7: *3005 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 172.18.0.1, server: localhost, request: "GET /_healthcheck HTTP/1.1", upstream: "http://172.18.0.2:8000/_healthcheck", host: "localhost"
2017/11/12 06:11:47 [error] 7#7: *3012 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 172.18.0.1, server: localhost, request: "GET /_healthcheck HTTP/1.1", upstream: "http://172.18.0.2:8000/_healthcheck", host: "localhost"

Analyzing the errors

The number of errors was increased after I migrate Classic LB to Application LB, and one of the differences between them is Classic LB is using pre-connected connections, and Application LB only using Http/1.1 Keep-Alive feature.

From the documentation of AWS Load Balancer:

Possible causes:

  • The load balancer received a TCP RST from the target when attempting to establish a connection.
  • The target closed the connection with a TCP RST or a TCP FIN while the load balancer had an outstanding request to the target.
  • The target response is malformed or contains HTTP headers that are not valid.
  • A new target group was used but no targets have passed an initial health check yet. A target must pass one health check to be considered healthy

Continue reading “Analyze ‘Connection reset’ error in Nginx upstream with keep-alive enabled”

Backup WordPress to Version Control Automatically

I migrated my WordPress blog to a VPS hosting recently, and after that, the first thing that came to my mind was: Backup.

There were a lot of similar posts on the Internet, but what I found was not good enough for me, so I wrote what I did in this article to help people who want to do the similar things. It would be great if you have better ideas and please feel free to let me know.

The following scripts had been tested on: Ubuntu 13.04 and MySQL 5.5. The directory and scripts may need to be changed for different Linux distributions.

The Problem

First, we need to answer the questions: what do we want to backup, what do not, and where to save the backup.

What do we want to backup? everything not from the setup, which includes:

  • Posts, Pages and Comments. They are the most important things that we want to backup.
  • Uploaded Media Files. They are the same important with the posts/pages.
  • The Installed Themes and Plugins. I don’t want to search, install, and customize them again, especially the colors.

Ok, that’s sounds reasonable, but is there anything you don’t want to backup? Continue reading “Backup WordPress to Version Control Automatically”

Nginx的DNS解析过程分析

Nginx怎么做域名解析？怎么在你自己开发的模块里面使用Nginx提供的方法解析域名？它内部实现是什么样的？

本文以Nginx 1.5.1为例，从nginx_mail_smtp模块如何进行域名解析出发，分析Nginx进行域名解析的过程。为了简化流程，突出重点，在示例代码中省掉了一些异常部分的处理，比如内存分配失败等。DNS查询分为两种：根据域名查询地址和根据地址查询域名，在代码结构上这两种方式非常相似，这里只介绍根据域名查询地址这一种方式。本文将从以下几个方面进行介绍：

  1. 域名查询的函数接口介绍
  2. 域名解析流程分析
  3. 查询场景分析及实现介绍

一、域名查询的函数接口介绍

在使用同步IO的情况下，调用gethostbyname()或者gethostbyname_r()就可以根据域名查询到对应的IP地址, 但因为可能会通过网络进行远程查询，所以需要的时间比较长。

为了不阻塞当前线程，Nginx采用了异步的方式进行域名查询。整个查询过程主要分为三个步骤， Continue reading “Nginx的DNS解析过程分析”

Linux下打开文件后没有关闭的后果分析

这两天测试文件操作的性能，发现了有的地方打开文件后没有关闭。不关闭文件的后果是比较严重的，尤其是对服务器端程序来说更为严重。那这样会有什么问题呢？
1.不能再打开新文件.
打开大量文件并且不关闭, 很快会达到进程最大允许打开的文件数限制，这样就不能再打开文件。
在Linux上，可以通过ulimit -n 来查看和更改当前session的限制数，比如在我的机器上是：

$ ulimit -n
7168
$ ulimit -n 10000
10000

也可以通过修改/etc/security/limits.conf来永久性的修改限制数

2. 硬盘空间被占满。
如果文件被打开后，再被删除，在文件不被关闭的情况下， Continue reading “Linux下打开文件后没有关闭的后果分析”

Notes for playing with ptrace on 64 bits Ubuntu 12.10

This blog is the notes during I learning the “Playing with ptrace”(http://www.linuxjournal.com/article/6100).

The original examples was using 32 bits machine, which doesn’t work on my 64 bits Ubuntu 12.10.

Let’s start from the first ptrace example:

#include <sys/ptrace.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <unistd.h>
#include <linux/user.h>   /* For constants
                                   ORIG_EAX etc */
int main()
{   pid_t child;
    long orig_eax;
    child = fork();
    if(child == 0) {
        ptrace(PTRACE_TRACEME, 0, NULL, NULL);
        execl("/bin/ls", "ls", NULL);
    }
    else {
        wait(NULL);
        orig_eax = ptrace(PTRACE_PEEKUSER,
                          child, 4 * ORIG_EAX,
                          NULL);
        printf("The child made a "
               "system call %ldn", orig_eax);
        ptrace(PTRACE_CONT, child, NULL, NULL);
    }
    return 0;
}

The compiler shows the following error:

fatal error: 'linux/user.h' file not found
#include <linux/user.h>

Something need to change because of:

  1. The ‘linux/user.h’ no longer exists
  2. The 64 bits register is R*X, so EAX changed to RAX

There are two solutions to fix this: Continue reading “Notes for playing with ptrace on 64 bits Ubuntu 12.10”

HOW TO: Create ssh tunnel at boot time under Ubuntu

Create ssh tunnel

The simplest command to create a ssh tunnel is:

#The following command will create a sock5 proxy on port 7070, and then you can use it in your browser
ssh -ND 7070 HOSTNAME

Use Autossh instead of ssh

I prefer to use autossh instead of ssh because it will auto reconnect if the connection lost, Continue reading “HOW TO: Create ssh tunnel at boot time under Ubuntu”

Could not compile libxml2: /bin/rm: cannot remove `libtoolT’: No such file or directory

When I compile libxml2-2.7.8 under Ubuntu 11.04, it always complain that:

/bin/rm: cannot remove `libtoolT’: No such file or directory.

I did a search on google, but none of them works.

From the console output, I found that it failed because removing a file which does not exists.

So I edit the configure file, find the line which contains $RM “$cfgfile”, and replace it with $RM -f “$cfgfile”, then everything works.

(‘rm -f’ means do not stop even if the file doesn’t exists)