How To: Create Subscription Filter in CloudWatch using serverless

Recently, I worked on a task which need to collect all CloudWatch logs to a Kinesis stream. The project is using Serverless for deployment. There are some plugins to create CloudWatch Log subscription filter, but none of them using Kinesis as the destination.

Then by using the serverless-scriptable-plugin, I’m able to do this very easily. The following code find out all CloudWatch LogGroups, and create a SubscriptionFilter for each of them.

Create a file at build/serverless/add-log-subscriptions.js Continue reading “How To: Create Subscription Filter in CloudWatch using serverless”

Troubleshooting of blocked requests when fetching messages from hundreds SQS queues

I’m working on a project which needs to fetch messages from hundreds of SQS queues. We’re using SQS long polling to reduce the number of empty responses. It was very quick to get response at first when there are only dozen queues. As we added more and more queues, the performance getting worse and worse. It takes 60 seconds to get the response when there’s 300 queues and WaitTimeSeconds set to 10 seconds.

We are using Node.js in single thread mode, and I believe that it could handle 10 thousands connections without any problem because most of the tasks are IO processing. We also created an AWS support case, but nobody clearly answered the question.

Using AWS SDK to reproduce the issue

I start to troubleshoot the problem, the first step is reproduce the issue using a simple code which makes it easier to find out the issue. Continue reading “Troubleshooting of blocked requests when fetching messages from hundreds SQS queues”

How to: convert asciidoc book to epub/mobi formats

Many open source books are written in asciidoc, in order to read the book in kindle, I have to convert it to mobi file. Here’s a quick note on how to convert files.

  1. Use asciidoctor to convert asciidoc book to docbook
  2. Use pandoc to convert to epub
  3. Use Calibre to convert epub to mobi
asciidoctor -d book -b docbook5 book.asciidoc -o output.docbook
pandoc -f docbook -t epub out.docbook -o book.epub

Then use Calibre to convert epub to mobi files

Aws Lambda retry behaviours on stream-based event sources

From the documentation, AWS Lambda will retry failed function on stream-based events sources.

By using Node.js, we can fail the function by many different ways, e.g. using callback to return error, throw exception directly, throw exception inside Promise, using Promise.reject. Then the questions is, what’s the proper way to let AWS Lambda know it needs a retry?

I did an quick test on following scenarios by setting up DynamoDB Stream and event mappings. It’s fun to have a guess which one will be retried and which one won’t.

Different ways to end the function

  • On Exception
module.exports.throwException = (event, context) => {
  console.log(JSON.stringify(event));
  throw new Error('something wrong');
};

Continue reading “Aws Lambda retry behaviours on stream-based event sources”

How to clean up branches in local Git repository

The git branches in local repository will grow rapidly if you are using branch development, e.g. always create a new branch for any new feature/story/bug-fix.

The branch becomes useless after it merged to master, here’s some commands to clean up branches in local repository.

Remove remote branches which have been deleted on remote repository

Using any of the following commands to remove branches which have been deleted on remote repository, e.g. branch deleted after merge to master

  git fetch --prune
  git pull --prune

If you are on git 1.8.5+ you can set this action to happen during each pull

git config fetch.prune true

Remove local branches which not following remote branch

List local branches which not following remote branch

git branch -vv | grep '\[origin/[^ ]*: gone\]' \
  | tr -s ' ' | cut -d' ' -f2 | grep -v master

Remove the above branches

git branch -vv | grep '\[origin/[^ ]*: gone\]' \
  | tr -s ' ' | cut -d' ' -f2 | grep -v master \
  | xargs git branch -d 

Capture console output when using child_process.execSync in node.js

I’m working on a nodejs project recently, and need to execute a command by using child_process.execSync().

The ideal solution should:

  • Support color output from command, which generally means the stdout/stderr should be a tty
  • Be testable, which means I can get the command output and verify

From the node.js documentation, I can use options.stdio to change stdio fds in child process.

For convenience, options.stdio may be one of the following strings:

‘pipe’ – equivalent to [‘pipe’, ‘pipe’, ‘pipe’] (the default)
‘ignore’ – equivalent to [‘ignore’, ‘ignore’, ‘ignore’]
‘inherit’ – equivalent to [process.stdin, process.stdout, process.stderr] or [0,1,2]

I started from the default behaviour – using pipe, which returns the console output after child process finishes. But it doesn’t support colours by default because piped stdout/stderr is not a tty. Continue reading “Capture console output when using child_process.execSync in node.js”

Backup WordPress to Version Control Automatically

I migrated my WordPress blog to a VPS hosting recently, and after that, the first thing that came to my mind was: Backup.

There were a lot of similar posts on the Internet, but what I found was not good enough for me, so I wrote what I did in this article to help people who want to do the similar things. It would be great if you have better ideas and please feel free to let me know.

The following scripts had been tested on: Ubuntu 13.04 and MySQL 5.5. The directory and scripts may need to be changed for different Linux distributions.

The Problem

First, we need to answer the questions: what do we want to backup, what do not, and where to save the backup.

What do we want to backup? everything not from the setup, which includes:

  • Posts, Pages and Comments. They are the most important things that we want to backup.
  • Uploaded Media Files. They are the same important with the posts/pages.
  • The Installed Themes and Plugins. I don’t want to search, install, and customize them again, especially the colors.

Ok, that’s sounds reasonable, but is there anything you don’t want to backup? Continue reading “Backup WordPress to Version Control Automatically”

Nginx的DNS解析过程分析

Nginx怎么做域名解析?怎么在你自己开发的模块里面使用Nginx提供的方法解析域名?它内部实现是什么样的?

本文以Nginx 1.5.1为例,从nginx_mail_smtp模块如何进行域名解析出发,分析Nginx进行域名解析的过程。为了简化流程,突出重点,在示例代码中省掉了一些异常部分的处理,比如内存分配失败等。DNS查询分为两种:根据域名查询地址和根据地址查询域名,在代码结构上这两种方式非常相似,这里只介绍根据域名查询地址这一种方式。本文将从以下几个方面进行介绍:

  1. 域名查询的函数接口介绍
  2. 域名解析流程分析
  3. 查询场景分析及实现介绍

一、域名查询的函数接口介绍

在使用同步IO的情况下,调用gethostbyname()或者gethostbyname_r()就可以根据域名查询到对应的IP地址, 但因为可能会通过网络进行远程查询,所以需要的时间比较长。

为了不阻塞当前线程,Nginx采用了异步的方式进行域名查询。整个查询过程主要分为三个步骤, Continue reading “Nginx的DNS解析过程分析”

通过流的方式处理文件压缩,加解密,签名

问题背景介绍

最近的项目需要进行很多的文件处理,因此就有了大量的IO操作。有的地方是先解密,再加密,有的是压缩,加密,再签名,最主要的是所有的非加密文件都需要安全删除,先填充一遍0,再把文件删除。

初始解决方案和问题

开始时我们使用文件来存储处理过程中的临时数据,以文件更换密码为例,需要进行如下处理:

  1. 解密原来的加密文件,写到一个临时文件
  2. 读取解密的临时文件,加密写到最终文件
  3. 将临时文件填充0,并删除

示例代码如下:

FileEncryptor.decrypt(originalEncryptedFile, tempFile);
FileEncryptor.encrypt(tempFile, resultEncryptedFile);
FileEraser.safeErase(tempFile);

这个过程中的IO操作如下图所示: Continue reading “通过流的方式处理文件压缩,加解密,签名”

Linux下打开文件后没有关闭的后果分析

这两天测试文件操作的性能,发现了有的地方打开文件后没有关闭。不关闭文件的后果是比较严重的,尤其是对服务器端程序来说更为严重。那这样会有什么问题呢?
1.不能再打开新文件.
打开大量文件并且不关闭, 很快会达到进程最大允许打开的文件数限制,这样就不能再打开文件。
在Linux上,可以通过ulimit -n 来查看和更改当前session的限制数,比如在我的机器上是:

$ ulimit -n
7168
$ ulimit -n 10000
10000

也可以通过修改/etc/security/limits.conf来永久性的修改限制数

2. 硬盘空间被占满。
如果文件被打开后,再被删除,在文件不被关闭的情况下, Continue reading “Linux下打开文件后没有关闭的后果分析”