log PID of each connection attempt

Finding the PID of an established connection is trivial using netstat or lsof. However, I have a process which is creating a connection ever 60 seconds to our database and locking it up by maxing out the failed connection attempt limit. I can increase the failed connection limit to something extremely high on the database, or I can try to track down what is making the connection, and I have chosen the latter.

Based on tcpdump/wireshark, I can see that what is happening is that a connection is established and then the connecting server immediately closes the connection before the server can even respond. What I don’t know is why.

The first step is to find out what PID is opening the connection. Unfortunately, this seems easier said than done. The problem is that when a connection goes into TIME_WAIT state, it is no longer associated with a PID. Since my connection has a lifetime of less than a tenth of a second, is there any way to record this information?

netstat and lsof appear to be able to poll every second, but this simply isn’t fast enough with the connection attempt I am dealing with. Is there a hook that I can connect to to dump this information to a log? Or is my only option to brute force it with a loop and some coding?

I use CentOS 6.

Asked By: James Shewey


Consider using SystemTap. It is dynamic instrumenting engine that dynamically patches kernel so you can track any in-kernel event such as opening a socket. It is actively developed by RedHat so it is supported in CentOS.


To install SystemTap on CentOS 6:

  1. Enable debuginfo repository:

    sed -i 's/^enabled=0/enabled=1/' /etc/yum.repos.d/CentOS-Debuginfo.repo
  2. Install SystemTap:

    yum install systemtap
  3. Install debuginfo packages for kernel. It can be done manually, but there is a tool that can do it automatically:



SystemTap has not tapset probe for TCP connection, but you may directly bind to kernel functions! You can also do it on socket level.

I.e. create script called conn.stp:

probe kernel.function("tcp_v4_connect") {
    printf("connect [%s:%d] -> %s:%dn", execname(), pid(),
            ip_ntop(@cast($uaddr, "struct sockaddr_in")->sin_addr->s_addr),
            ntohs(@cast($uaddr, "struct sockaddr_in")->sin_port));

This will give you the following output:

# stap conn.stp
connect [nc:2552] ->
connect [nc:2554] ->
connect [nc:2556] ->

However tracking disconnection events seem to be more trickier.

Answered By: myaut
Categories: Answers Tags: , , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.