PCAP-GenerationTools and Content-Type Identification Patch
▁▇█▄▂▃▂▃█▅▃▇▄▂▃▄▃▇▁▂▃▄▁▃▄▅▆▆▇▇▂▆▅▁▆▆▆█▆█▆▁▂▃▃█▆▅█▁▃▂▇▇▂▁▂▃▂▃▇▅▆▆ « 📅 published on 18/Jun/2014 »
🔖 tagged code
Introduction
I came across the PCAPGenerationTools project more than a year ago and immediately got hooked to it. A huge shoutout to its author Andrew Griffiths who did an amazing job creating this tool and made it opensource for the community. It is amongst those tools that I didn't create and use on a daily basis.
At its core, PCAPGenerationTools allows you to create pcaps for an input file. That's a description in extreme brevity. To make things clear let's consider an example where you are working on a scenario in which you need to look at a pcap that captures file download. The file being downloaded could be of any type but for our case let's assume it is a PDF file. So, when you host a PDF file on a webserver and request that file from the server, underlying packet exchange is what PCAPGenerationTools will create for you. And it can do this without hosting the file on the server and requesting it while some network sniffer runs in background. Let's give the tool a test run to clarify things a bit:
$ ./pgt.py hello-world.pdf
WARNING: No route found for IPv6 destination :: (no default route?)
[*] Doing FTP
[*] output/hello-world.pdf/FTP_upload_active.pcap...
[*] output/hello-world.pdf/FTP_upload_passive.pcap...
[*] output/hello-world.pdf/FTP_download_active.pcap...
[*] output/hello-world.pdf/FTP_download_passive.pcap...
[*] Doing Email
[*] output/hello-world.pdf/POP3_base64.pcap...
[*] output/hello-world.pdf/POP3_7or8bit.pcap...
[*] output/hello-world.pdf/POP3_quopri.pcap...
[*] output/hello-world.pdf/IMAP_base64.pcap...
[*] output/hello-world.pdf/IMAP_7or8bit.pcap...
[*] output/hello-world.pdf/IMAP_quopri.pcap...
[*] output/hello-world.pdf/SMTP_base64.pcap...
[*] output/hello-world.pdf/SMTP_7or8bit.pcap...
[*] output/hello-world.pdf/SMTP_quopri.pcap...
[*] Doing HTTP
[*] output/hello-world.pdf/HTTP_GET_raw_raw.pcap ...
[*] output/hello-world.pdf/HTTP_GET_raw_chunked.pcap ...
[*] output/hello-world.pdf/HTTP_GET_gzip_raw.pcap ...
[*] output/hello-world.pdf/HTTP_GET_gzip_chunked.pcap ...
[*] output/hello-world.pdf/HTTP_GET_deflate_raw.pcap ...
[*] output/hello-world.pdf/HTTP_GET_deflate_chunked.pcap ...
[*] output/hello-world.pdf/HTTP_POST_raw_raw.pcap ...
[*] output/hello-world.pdf/HTTP_POST_raw_chunked.pcap ...
[*] output/hello-world.pdf/HTTP_POST_gzip_raw.pcap ...
[*] output/hello-world.pdf/HTTP_POST_gzip_chunked.pcap ...
[*] output/hello-world.pdf/HTTP_POST_deflate_raw.pcap ...
[*] output/hello-world.pdf/HTTP_POST_deflate_chunked.pcap ...
[*] Doing TCP
[*] output/hello-world.pdf/TCP_CTS_raw.pcap ...
[*] output/hello-world.pdf/TCP_STC_raw.pcap ...
$
$ ls output/hello-world.pdf/
FTP_download_active.pcap HTTP_GET_deflate_chunked.pcap HTTP_GET_raw_chunked.pcap HTTP_POST_gzip_chunked.pcap IMAP_7or8bit.pcap POP3_base64.pcap SMTP_quopri.pcap
FTP_download_passive.pcap HTTP_GET_deflate_raw.pcap HTTP_GET_raw_raw.pcap HTTP_POST_gzip_raw.pcap IMAP_base64.pcap POP3_quopri.pcap TCP_CTS_raw.pcap
FTP_upload_active.pcap HTTP_GET_gzip_chunked.pcap HTTP_POST_deflate_chunked.pcap HTTP_POST_raw_chunked.pcap IMAP_quopri.pcap SMTP_7or8bit.pcap TCP_STC_raw.pcap
FTP_upload_passive.pcap HTTP_GET_gzip_raw.pcap HTTP_POST_deflate_raw.pcap HTTP_POST_raw_raw.pcap POP3_7or8bit.pcap SMTP_base64.pcap
I'm using sample PDF files from Didier Stevens blog post on Physical and Logical Structure of PDF Files. PCAPGenerationTools created pcaps for FTP, POP3, IMAP, SMTP, HTTP and TCP. Let's have a look at one of the HTTP pcaps in Wireshark:
And here's how the reassembled HTTP stream looks like:
That's pure awesomeness! It created pcaps with different HTTP Content-Type and Transfer-Encoding combinations, active/passive mode in FTP, upload/download over HTTP and FTP and various MIMEtype encodings for POP3/IMAP/SMTP. These pcaps were created without hosting files and using respective clients and without generating any network traffic.
Patch Contributions
I submitted three patches of which two have been merged in the master branch of the project. The first patch auto content type identification and output directory creation which basically added a MIMEtype identification feature for Content-Type headers in HTTP pcaps. In the screenshot above, you will notice that the server response (in blue) shows Content-Type header with the correct MIMEtype of application/pdf. This feature helps create pcaps that simulate real world scenarios.
Another patch that I submitted and which is yet to be merged, adds raw TCP CTS/STC payload pcap generation feature to PCAPGenerationTools. This patch allows creation of pcaps that capture file transfer over TCP itself and where layer 7 protocols are not involved. An example scenario would be transferring files over a netcat session.
Conclusion
PCAPGenerationTools is an amazing project and quite powerful in many ways. Make sure you try and let me know if you share the same excitement as I do about this tool.