Debugging with Netcat

Wanted to spotlight one of my favorite utlities -- NETCAT. It's probably my most favorite of utilities. I've used it for years in debugging network issues, especially web issues. It's been described as the "The TCP/IP Swiss Army Knife." It's very powerful.

I use it frequently to grab and send http requests. It allows you to see the exact bytes sent/received on the wire from your browser.

Using netcat to grab an http request
1) Start netcat in listen mode on a port and save the request.

$ nc -l -p 9999 | tee somerequest.http

2) Perform a sample HTTP request using browser: http://localhost:9999/index.html
3) View the request -- you'll see the entire http request payload (headers & content)

GET /index.html HTTP/1.1
Host: localhost:9999
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008072820 Firefox/3.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive

Using netcat to play back an http request
Easy as capturing a request -- just redirect the saved HTTP request using:

$ nc www.google.com 80 < somerequest.http > someresponse.http

You may notice that the response looks garbled -- this is probably due to the fact that it is GZipped-encoded. Look for a header such as "Content-Encoding: gzip". You could re-submit the request after removing the "Accept-Encoding:" header and it will no longer be in gzip format.

Cool Things to Do to Impress Your Friends
1) Copy a file from 1 server to another
Netcat just reads & dumps byts to and from ports. Very simple. To copy a file from one server to another without using SSH/FTP/RCP/etc, just do this:
On the source server, just redirect a file to a port:

$ nc -l -p 9999 < somefile.txt

On the destination server, just connect to that port and redirect the bytes to a local file:

$ nc source.server.com 9999 > somefile.txt

You may want to do a checksum on the file to ensure contents were not modified or somehow broken.

2) Copy segments of a file (i.e., restarting a transfer)
If you are doing the above transfer and something occurred which caused network to fail, you can simply send just parts of the file and concatenate the new segments to the old file. You just need to know how may bytes the destination file already has, then use "dd" to strip them off. In the following example, the destination already had the first 12,000,150 bytes, so we will skip those.

$ dd bs=1 skip=12,000,150 if=somefile.txt | nc -l -p 9999

Then, just simply append the new contents to what you already have on the destination:

$ nc source.server.com 9999 >> somefile.txt

3) Give shell access
Netcat can be used to pipe STDIN/STDOUT to a process, too. This can be dangerous, but also powerful. :)

This example creates a network pipe to bash, so anyone connecting to the listener port will have the users bash command access:

$ nc -l -p 9999 -e /bin/bash

Probably, a better way to utilize this feature is to perform a quick backup of a directory. On the source server, type (the -q 5 options tells netcat to close the connection 5 seconds after reaching the EOF)

$ tar zcfv - somedir | nc -q 5 -l -p 9999

Then, on the destination server, type:

$ nc myserver.com 9999 > somedir.tar.gz

3b) If you have the "pv" utility installed, you can get progress information displayed to your terminal. Pv just displays information about the bytes traveling through a network pipe.

tar zcf - somedir | pv | nc -l -p 9999
61.3MB 0:00:30 [ 2MB/s] [ <=> ]

4) Port scanning
Netcat can act as a port-scanner, too.

$ nc -v -z localhost 1-100
localhost [127.0.0.1] 80 (www) open

What about HTTPS?
Stunnel is another one of my favorite utilities. It allows you to tunnel TCP/IP connections over SSL. It also can act as an HTTPS proxy so that you can stick with HTTP traffic locally, but switch to HTTPS when you put it on the wire.

This is very handy when you don't have control over the server and it only requests https, but you want to take a look at packets/http messages b/w your client and the server.

I'll do another post soon on how to use stunnel to handle https.

Alternatives
WireShark/Tcpdump -- Packet analyzer. Very nice and powerful (wireshark used to be called Ethereal)
TcpMon -- was bundled with earlier version of Axis 1.x, but not sure where it went now? -- just sat in the middle b/w TCP/IP connections and listened, logged, and fwded in real time.
Firebug Firefox plugin -- Nice for HTTP debugging.