Public pages

h1. nfdump suite

{{toc}}

h1. Introduction

This page has information about the nfdump suite.  It is intended to be publicly useful

The name "nfdump suite" is potentially misleading because nfdump is one of six Netflow utilities in the suite: nfcapd, nfdump, nfexpire, nfprofile, nfreplay and nftrack.  nfdump translates the binary netlow record files created by nfcapd (Netflow capture daemon) into human readable form

The nfdump suite was closely related to the now defunct nfsen.  Nfsen was a web GUI which used nfdump to generate data for interactive display

h2. Related documentation

* https://github.com/phaag/nfdump

* man pages

h1. nfdump setup

This is actually about setting up an nfcapd service but the default names include nfdump

h2. semop() error in bookkeeper.c line ...: Invalid argument"

These error messages were fixed by adding ... 

<pre>

RemoveIPC=no

</pre>... to /etc/systemd/logind.conf and effecting it by

<pre>

systemctl daemon-reload

systemctl restart systemd-logind.service

</pre>

h2. Setup itself

Example for "edge1" using directory sub hierarchy scheme number 1.  Directory sub hierarchy schemes are listed in the nfcapd man page

The network device had been set up to send netflow packets to the nfdump suite server.  That was specific to the type of device so is not shown here

Setup took a long time because no complete guide was found.  Hence this page

The following procedure was assembled from notes so may not be exactly right but it should be a good starting point

* Created user nfcapd

* Created directory /var/cache/nfdump/edge1 nfcapd:nfcapd rwxr-xr-x

* Set up netflow record expiry.  As user nfcapd

<pre>

nfexpire -t $((365*3))d -u /var/cache/nfdump/edge1

</pre>

* Created /etc/nfdump/edge1.conf by copying /etc/nfdump and editing

<pre>

root@storage3.iciti.av:~# diff /etc/nfdump/default.conf /etc/nfdump/edge1.conf

...

< #cache_directory=/var/cache/nfdump

< #user=root

< #group=root

---

> cache_directory=/var/cache/nfdump/edge1

> user=nfcapd

> group=nfcapd

28c56

< options='-l /var/cache/nfdump -p 2055'

---

> options='-e -g nfcapd -l /var/cache/nfdump/edge1 -p 2055 -S1 -u nfcapd'

</pre>

* Fixed a /run permissions difficulty by creating /etc/systemd/system/nfdump@.service

<pre>

diff /lib/systemd/system/nfdump@.service /etc/systemd/system/nfdump@.service

10,11c10,11

< ExecStart=/usr/bin/nfcapd -D -P /run/nfcapd.%I.pid $options

< PIDFile=/run/nfcapd.%I.pid

---

> ExecStart=/usr/bin/nfcapd -D -P /tmp/nfcapd.%I.pid $options

> PIDFile=/tmp/nfcapd.%I.pid

</pre>

* Created a templated service  

<pre>

ln -s /etc/systemd/system/nfdump@.service /etc/systemd/system/nfdump@edge1.service

systemctl daemon-reload

systemctl start nfdump@edge1.service

systemctl enable nfdump@edge1.service

</pre>

h2. Logcheck

<pre>

cat /etc/logcheck/ignore.d.server/local-nfdump

^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ nfcapd\[[[:digit:]]+\]: Add extension: 

^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ nfcapd\[[[:digit:]]+\]: Bound to IPv

^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ nfcapd\[[[:digit:]]+\]: Current size: .*, Current lifetime: .*, Number of files: [[:digit:]]+$

^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ nfcapd\[[[:digit:]]+\]: Ident: .* Flows: [[:digit:]]+, Packets: [[:digit:]]+, Bytes: [[:digit:]]+, Sequence Errors: 0, Bad Packets: 0$

^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ nfcapd\[[[:digit:]]+\]: Init IPFIX: Max number of IPFIX tags: [[:digit:]]+$

^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ nfcapd\[[[:digit:]]+\]: Limits: Filesize .*, Lifetime .*, Watermark: [[:digit:]]+%$

^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ nfcapd\[[[:digit:]]+\]: Process_v9: New exporter: SysID: [[:digit:]]+, Domain: [[:digit:]]+, IP: 

^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ nfcapd\[[[:digit:]]+\]: Process_v9: \[[[:digit:]]+\] Add template [[:digit:]]+$

^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ nfcapd\[[[:digit:]]+\]: Run expire on 

^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ nfcapd\[[[:digit:]]+\]: Signal launcher$

^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ nfcapd\[[[:digit:]]+\]: Startup\.$

^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ nfcapd\[[[:digit:]]+\]: Terminating nfcapd\.$

^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ nfcapd\[[[:digit:]]+\]: Total ignored packets: 0$

^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ nfcapd\[[[:digit:]]+\]: expire completed - nothing to expire\.$

</pre>