How to clean up branches in local Git repository

The git branches in local repository will grow rapidly if you are using branch development, e.g. always create a new branch for any new feature/story/bug-fix.

The branch becomes useless after it merged to master, here’s some commands to clean up branches in local repository.

Remove remote branches which have been deleted on remote repository

Using any of the following commands to remove branches which have been deleted on remote repository, e.g. branch deleted after merge to master

  git fetch --prune
  git pull --prune

If you are on git 1.8.5+ you can set this action to happen during each pull

git config fetch.prune true

Remove local branches which not following remote branch

List local branches which not following remote branch

git branch -vv | grep '\[origin/[^ ]*: gone\]' \
  | tr -s ' ' | cut -d' ' -f2 | grep -v master

Remove the above branches

git branch -vv | grep '\[origin/[^ ]*: gone\]' \
  | tr -s ' ' | cut -d' ' -f2 | grep -v master \
  | xargs git branch -d 

Capture console output when using child_process.execSync in node.js

I’m working on a nodejs project recently, and need to execute a command by using child_process.execSync().

The ideal solution should:

  • Support color output from command, which generally means the stdout/stderr should be a tty
  • Be testable, which means I can get the command output and verify

From the node.js documentation, I can use options.stdio to change stdio fds in child process.

For convenience, options.stdio may be one of the following strings:

‘pipe’ – equivalent to [‘pipe’, ‘pipe’, ‘pipe’] (the default)
‘ignore’ – equivalent to [‘ignore’, ‘ignore’, ‘ignore’]
‘inherit’ – equivalent to [process.stdin, process.stdout, process.stderr] or [0,1,2]

I started from the default behaviour – using pipe, which returns the console output after child process finishes. But it doesn’t support colours by default because piped stdout/stderr is not a tty. Continue reading “Capture console output when using child_process.execSync in node.js”

Backup WordPress to Version Control Automatically

I migrated my WordPress blog to a VPS hosting recently, and after that, the first thing that came to my mind was: Backup.

There were a lot of similar posts on the Internet, but what I found was not good enough for me, so I wrote what I did in this article to help people who want to do the similar things. It would be great if you have better ideas and please feel free to let me know.

The following scripts had been tested on: Ubuntu 13.04 and MySQL 5.5. The directory and scripts may need to be changed for different Linux distributions.

The Problem

First, we need to answer the questions: what do we want to backup, what do not, and where to save the backup.

What do we want to backup? everything not from the setup, which includes:

  • Posts, Pages and Comments. They are the most important things that we want to backup.
  • Uploaded Media Files. They are the same important with the posts/pages.
  • The Installed Themes and Plugins. I don’t want to search, install, and customize them again, especially the colors.

Ok, that’s sounds reasonable, but is there anything you don’t want to backup? Continue reading “Backup WordPress to Version Control Automatically”

Nginx的DNS解析过程分析

Nginx怎么做域名解析?怎么在你自己开发的模块里面使用Nginx提供的方法解析域名?它内部实现是什么样的?

本文以Nginx 1.5.1为例,从nginx_mail_smtp模块如何进行域名解析出发,分析Nginx进行域名解析的过程。为了简化流程,突出重点,在示例代码中省掉了一些异常部分的处理,比如内存分配失败等。DNS查询分为两种:根据域名查询地址和根据地址查询域名,在代码结构上这两种方式非常相似,这里只介绍根据域名查询地址这一种方式。本文将从以下几个方面进行介绍:

  1. 域名查询的函数接口介绍
  2. 域名解析流程分析
  3. 查询场景分析及实现介绍

一、域名查询的函数接口介绍

在使用同步IO的情况下,调用gethostbyname()或者gethostbyname_r()就可以根据域名查询到对应的IP地址, 但因为可能会通过网络进行远程查询,所以需要的时间比较长。

为了不阻塞当前线程,Nginx采用了异步的方式进行域名查询。整个查询过程主要分为三个步骤, Continue reading “Nginx的DNS解析过程分析”

通过流的方式处理文件压缩,加解密,签名

问题背景介绍

最近的项目需要进行很多的文件处理,因此就有了大量的IO操作。有的地方是先解密,再加密,有的是压缩,加密,再签名,最主要的是所有的非加密文件都需要安全删除,先填充一遍0,再把文件删除。

初始解决方案和问题

开始时我们使用文件来存储处理过程中的临时数据,以文件更换密码为例,需要进行如下处理:

  1. 解密原来的加密文件,写到一个临时文件
  2. 读取解密的临时文件,加密写到最终文件
  3. 将临时文件填充0,并删除

示例代码如下:

FileEncryptor.decrypt(originalEncryptedFile, tempFile);
FileEncryptor.encrypt(tempFile, resultEncryptedFile);
FileEraser.safeErase(tempFile);

这个过程中的IO操作如下图所示: Continue reading “通过流的方式处理文件压缩,加解密,签名”

Linux下打开文件后没有关闭的后果分析

这两天测试文件操作的性能,发现了有的地方打开文件后没有关闭。不关闭文件的后果是比较严重的,尤其是对服务器端程序来说更为严重。那这样会有什么问题呢?
1.不能再打开新文件.
打开大量文件并且不关闭, 很快会达到进程最大允许打开的文件数限制,这样就不能再打开文件。
在Linux上,可以通过ulimit -n 来查看和更改当前session的限制数,比如在我的机器上是:

$ ulimit -n
7168
$ ulimit -n 10000
10000

也可以通过修改/etc/security/limits.conf来永久性的修改限制数

2. 硬盘空间被占满。
如果文件被打开后,再被删除,在文件不被关闭的情况下, Continue reading “Linux下打开文件后没有关闭的后果分析”

Development notes for socks5 protocol

The RFC for socks5 protocol: http://www.ietf.org/rfc/rfc1928.txt

Here’s the procedure of CONNECT command for TCP based connection:

   Client   |   Server
----------------------------------------------------
1. Client init connection
2. Client send initial auth method selection message
             3. Server reply the selected auth method

4. [ Authenticate based on selected auth method ] optional, 
   not required when auth method is 0x00: No Auth

5. Client send request for destination address
             6. Server connect to the destination address
             7. Server reply with the bound address and port 
                of the connected target address

8. Begin transfer data between client and destination  
9. Close connection after transfer finished

Notes:

  • The procedure applies to a connection, e.g. each http request opens a new connection, so each request will go through the procedure described above
  • The destination address maybe an IP v4/v6 address or a domain name, when it’s a domain name, it still could be an ip address in dotted format. Here’s a real example:
    www.youku.com is using the CDN service provided by http://www.chinacache.com, so different visitor will visit different server based on the location. I found that the browser (at least chrome) will put ip address like 65.255.34.6 in the domain name field, and mark the address as a domain name
  • When server send response to client, it should send the port number that the server assigned to connect the target address. So we can only send the response after connected to the upstream

Notes for playing with ptrace on 64 bits Ubuntu 12.10

This blog is the notes during I learning the “Playing with ptrace”(http://www.linuxjournal.com/article/6100).

The original examples was using 32 bits machine, which doesn’t work on my 64 bits Ubuntu 12.10.

Let’s start from the first ptrace example:

#include <sys/ptrace.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <unistd.h>
#include <linux/user.h>   /* For constants
                                   ORIG_EAX etc */
int main()
{   pid_t child;
    long orig_eax;
    child = fork();
    if(child == 0) {
        ptrace(PTRACE_TRACEME, 0, NULL, NULL);
        execl("/bin/ls", "ls", NULL);
    }
    else {
        wait(NULL);
        orig_eax = ptrace(PTRACE_PEEKUSER,
                          child, 4 * ORIG_EAX,
                          NULL);
        printf("The child made a "
               "system call %ldn", orig_eax);
        ptrace(PTRACE_CONT, child, NULL, NULL);
    }
    return 0;
}

The compiler shows the following error:

fatal error: 'linux/user.h' file not found
#include <linux/user.h>

Something need to change because of:

  1. The ‘linux/user.h’ no longer exists
  2. The 64 bits register is R*X, so EAX changed to RAX

There are two solutions to fix this: Continue reading “Notes for playing with ptrace on 64 bits Ubuntu 12.10”

Understand the compile time operator: sizeof

During I read the source code of Redis, I found the following code:

dict *d = zmalloc(sizeof(*d));

After searching a different definition of ‘d’, I realized that the d is the same object defined in the same line. In order to understand what happens, Continue reading “Understand the compile time operator: sizeof”

HOW TO: Create ssh tunnel at boot time under Ubuntu

Create ssh tunnel

The simplest command to create a ssh tunnel is:

#The following command will create a sock5 proxy on port 7070, and then you can use it in your browser
ssh -ND 7070 HOSTNAME

Use Autossh instead of ssh

I prefer to use autossh instead of ssh because it will auto reconnect if the connection lost, Continue reading “HOW TO: Create ssh tunnel at boot time under Ubuntu”