27 June 2014

Overview

DSC is a tool for collecting and presenting the dns data from your dns servers. It’s server agnostic, which makes it useful in heterogenous environments like in TLDs.

DSC collector is capturing the network data using the pcap library and produces the XML outuput for presenter.

Why offline ?

As the parsing code seems to be pretty stable we are not very keen to run this software as root on our dns servers. The solution could be network tap or port mirror but this os not a general solution.

The reasons are quite simple

  • if there is a bug in collector it can crash and do whatever with the server
  • the output is suitable only for dsc, no other processing with another software

How I did it

Offline traces

For capturing and archiving offline traces I was inspired by packetQ project. In the collector_sample directory there is pg_tracesplit.pl script. This script is using tracesplit utility and based on the configuration file it stores the traces for given interval in date/time directory structure. Trace files are gziped to save the disk space and network bandwidth.

I wrote small backup/retention script to push the traces to backup server and keep last X days on the DNS server.

DSC collector

When you specify a pcap file as interface on the configuration file the dsc utility will parse the trace file. The disadvantage is that you can specify only one interface line while parsing offline traces and the DSC is not able to parse gzipped offline traces like packetQ.

I wrote small wrapper script, whch based on the configuration

  • will take lastest unprocessed pcap gzip files for gived dns server
  • process all newest files
    • copy this file to tmp location and gunzip this file
    • take dsc collector template file and fill correct interface line
    • run dsc with given configuration
    • remove the temporary file

As this script could process only files for one server at once you could run something like this

echo <LIST OF CONF FILES> | xargs --no-run-if-empty --max-args=1 --max-procs=<NUMBER OF PARALLEL INTANCES> <WRAPPER SCRIPT>

DSC collector patch

I have bug in pcap library on my Debian Wheezy when try to set pcap_setnonblock on the offline file. Here is my workaround for this

Index: collector/dsc/pcap.c
===================================================================
--- collector/dsc/pcap.c	(revision 13802)
+++ collector/dsc/pcap.c	(working copy)
@@ -957,6 +957,7 @@
     char errbuf[PCAP_ERRBUF_SIZE];
     int x;
     struct _interface *i;
+    int live;
 
     if (interfaces == NULL) {
 	interfaces = xcalloc(MAX_N_INTERFACES, sizeof(*interfaces));
@@ -971,7 +972,9 @@
     last_ts.tv_sec = last_ts.tv_usec = 0;
     finish_ts.tv_sec = finish_ts.tv_usec = 0;
 
-    if (0 == stat(device, &sb)) {
+    live = stat(device, &sb);
+
+    if (0 == live) {
 	i->pcap = pcap_open_offline(device, errbuf);
     } else {
 	/*
@@ -988,7 +991,7 @@
 	syslog(LOG_ERR, "pcap_open_*: %s", errbuf);
 	exit(1);
     }
-    if (pcap_setnonblock(i->pcap, 1, errbuf) < 0) {
+    if (live && pcap_setnonblock(i->pcap, 1, errbuf) < 0) {
 	syslog(LOG_ERR, "pcap_setnonblock(%s): %s", device, errbuf);
 	exit(1);
     }

I have tried to subscribe to the dsc@measurement-factory.com to submit the patch, but I did not received the confirmation e-mail.

Next improvements

Processing 5min gziped trace fiels is quite expensive, while tehre is a lot preparation work. The PIDS are rotating very quickly.

For example you could merge the trace files for 1hour period and try process them in one run. The side efect is the delay of getting the statistics.

Maybe it would be good to learn dsc to read gzipped trace files like packetQ does.

Some final notes

This was my first approach to monitor dns traffic. As there are meny other useful utilities I will try to do next work on it.

DSC is great tool, but the development of this project seems not to be continuing. There are other usefull utilities that could be used to process offline traces like packetQ, PassiveDNS or stash53.