Project

General

Profile

nfdump suite

Introduction

This page has information about the nfdump suite. It is intended to be publicly useful

The name "nfdump suite" is potentially misleading because nfdump is one of six Netflow utilities in the suite: nfcapd, nfdump, nfexpire, nfprofile, nfreplay and nftrack. nfdump translates the binary netlow record files created by nfcapd (Netflow capture daemon) into human readable form

The nfdump suite was closely related to the now defunct nfsen. Nfsen was a web GUI which used nfdump to generate data for interactive display

Related documentation

nfdump setup

This is actually about setting up an nfcapd service but the default names include nfdump

semop() error in bookkeeper.c line ...: Invalid argument"

These error messages were fixed by adding ...

RemoveIPC=no
... to /etc/systemd/logind.conf and effecting it by
systemctl daemon-reload
systemctl restart systemd-logind.service

Setup itself

Example for "edge1" using directory sub hierarchy scheme number 1. Directory sub hierarchy schemes are listed in the nfcapd man page

The network device had been set up to send netflow packets to the nfdump suite server. That was specific to the type of device so is not shown here

Setup took a long time because no complete guide was found. Hence this page

The following procedure was assembled from notes so may not be exactly right but it should be a good starting point
  • Created user nfcapd
  • Created directory /var/cache/nfdump/edge1 nfcapd:nfcapd rwxr-xr-x
  • Set up netflow record expiry. As user nfcapd
    nfexpire -t $((365*3))d -u /var/cache/nfdump/edge1
    
  • Created /etc/nfdump/edge1.conf by copying /etc/nfdump and editing
    root@storage3.iciti.av:~# diff /etc/nfdump/default.conf /etc/nfdump/edge1.conf
    ...
    < #cache_directory=/var/cache/nfdump
    < #user=root
    < #group=root
    ---
    > cache_directory=/var/cache/nfdump/edge1
    > user=nfcapd
    > group=nfcapd
    28c56
    < options='-l /var/cache/nfdump -p 2055'
    ---
    > options='-e -g nfcapd -l /var/cache/nfdump/edge1 -p 2055 -S1 -u nfcapd'
    
  • Fixed a /run permissions difficulty by creating /etc/systemd/system/nfdump@.service
    diff /lib/systemd/system/nfdump@.service /etc/systemd/system/nfdump@.service
    10,11c10,11
    < ExecStart=/usr/bin/nfcapd -D -P /run/nfcapd.%I.pid $options
    < PIDFile=/run/nfcapd.%I.pid
    ---
    > ExecStart=/usr/bin/nfcapd -D -P /tmp/nfcapd.%I.pid $options
    > PIDFile=/tmp/nfcapd.%I.pid
    
  • Created a templated service
    ln -s /etc/systemd/system/nfdump@.service /etc/systemd/system/nfdump@edge1.service
    systemctl daemon-reload
    systemctl start nfdump@edge1.service
    systemctl enable nfdump@edge1.service
    

Logcheck

cat /etc/logcheck/ignore.d.server/local-nfdump
^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ nfcapd\[[[:digit:]]+\]: Add extension: 
^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ nfcapd\[[[:digit:]]+\]: Bound to IPv
^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ nfcapd\[[[:digit:]]+\]: Current size: .*, Current lifetime: .*, Number of files: [[:digit:]]+$
^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ nfcapd\[[[:digit:]]+\]: Ident: .* Flows: [[:digit:]]+, Packets: [[:digit:]]+, Bytes: [[:digit:]]+, Sequence Errors: 0, Bad Packets: 0$
^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ nfcapd\[[[:digit:]]+\]: Init IPFIX: Max number of IPFIX tags: [[:digit:]]+$
^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ nfcapd\[[[:digit:]]+\]: Limits: Filesize .*, Lifetime .*, Watermark: [[:digit:]]+%$
^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ nfcapd\[[[:digit:]]+\]: Process_v9: New exporter: SysID: [[:digit:]]+, Domain: [[:digit:]]+, IP: 
^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ nfcapd\[[[:digit:]]+\]: Process_v9: \[[[:digit:]]+\] Add template [[:digit:]]+$
^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ nfcapd\[[[:digit:]]+\]: Run expire on 
^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ nfcapd\[[[:digit:]]+\]: Signal launcher$
^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ nfcapd\[[[:digit:]]+\]: Startup\.$
^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ nfcapd\[[[:digit:]]+\]: Terminating nfcapd\.$
^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ nfcapd\[[[:digit:]]+\]: Total ignored packets: 0$
^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ nfcapd\[[[:digit:]]+\]: expire completed - nothing to expire\.$