http-sniffer

A multi-threading tool to sniff TCP flow statistics and embedded HTTP headers from PCAP file. Each TCP flow carrying HTTP is exported to text file in json format.

Project maintained by caesar0301 Hosted on GitHub Pages — Theme by mattgraham

http-sniffer

A multi-threading tool to sniff HTTP header records beyond TCP flow statistics.

MIT licensed.

Features

Support both offline PCAP file and live NIC sniffing
Perform multi-threading process, whereby the tool gets higher performance in face of big tarffic volume
Export statistics of TCP flows
Export HTTP request/response pairs if they are present in the TCP flow
Support JSON format output

Dependencis

The http-sniffer depends on other utilities:

libpcap to extract traffic packet,
json-c to support json parsing, and
scons to build the project.

A known issue about json-c is the unused params errors on linux platform, you can use an alternative to solve it. You may also need to export the library path where libjson-c.* locate, by

export LD_LIBRARY_PATH=/usr/local/lib

How to Use

Run scons in root folder to compile:

$ cd http-sniffer
$ scons

Get your live interface with ifconfig in terminal, e.g. en0, then

$ ./bin/http-sniffer -i en0

Or store output flows as json

$ ./bin/http-sniffer -i en0 -o output.json

Output

In brief CSV format:

[20120921 16:40:09]10.187.179.28:53196-->180.149.134.229:80 1335164797.208360 0.0 0.0 167 5/3 0/0 0 0
[20120921 16:40:09]10.187.179.28:53160-->58.63.234.206:80 1335164789.893109 0.0 0.0 21 4/2 0/0 0 0
[20120921 16:40:09]10.187.179.28:53161-->58.63.234.206:80 1335164789.893219 0.0 0.0 225 4/2 0/0 0 0
[20120921 16:40:09]10.187.179.28:53158-->58.63.234.198:80 1335164789.769004 0.0 0.0 118 4/2 0/0 0 0
[20120921 16:40:09]10.187.179.28:53164-->113.108.216.252:80 1335164790.179680 0.0 0.0 12 4/2 0/0 0 0
[20120921 16:40:09]10.187.179.28:53189-->180.149.134.221:80 1335164797.961918 0.0 0.0 111 3/1 0/0 0 0

In full JSON format: each line records one TCP flow with piggybacked HTTP messages, e.g.

{
   "time_local": "2012-04-23T13:50:09",
   "saddr": "192.168.1.4", 
   "daddr": "192.168.1.5", 
   "sport": 45753, 
   "dport": 80, 
   "time_syn": 1335160209.417475, 
   "time_first_byte": 1335160209.452336, 
   "time_last_byte": 1335160209.488276, 
   "rtt": 248, 
   "src_packets": 4, 
   "dst_packets": 4, 
   "src_bytes": 521, 
   "dst_bytes": 257, 
   "http_pair_count": 1, 
   "force_closed": 0, 
   "http_pairs": [ 
    { 
     "request": { 
      "time_first_byte": 1335160209.452336, 
      "time_last_byte": 1335160209.452336, 
      "bytes_transfered": 521, 
      "http_version": 1, 
      "method": 1, 
      "host": "s1.bdstatic.com", 
      "uri": "\/r\/www\/img\/i-1.0.0.png", 
      "referer": "http:\/\/www.baidu.com\/", 
      "user_agent": "Mozilla\/5.0", 
      "accept": "image\/png,image\/*;q=0.8,*\/*;q=0.5", 
      "accept_encoding": "gzip,deflate", 
      "accept_language": "en-us,en;q=0.5", 
      "accept_charset": "ISO-8859-1,utf-8;q=0.7,*;q=0.7"}, 
     "response": {
      "time_first_byte": 1335160209.488260, 
      "time_last_byte": 1335160209.488260, 
      "bytes_transfered": 257, 
      "http_version": 1, 
      "status": 304, 
      "server": "JSP\/1.0.18", 
      "date": "Mon, 23 Apr 2012 06:02:23 GMT", 
      "expires": "Tue, 29 Mar 2022 09:34:06 GMT", 
      "etag": "\"25f-4a6ebc21c42c0\"", 
      "last_modified": "Thu, 30 Jun 2011 10:56:51 GMT"} 
    }] 
}

About Author

Xiaming Chen chen@xiaming.me

SJTU, Shanghai, China

2012-04-01