Step by Step Network Monitoring using NFR

Version .4 980527

Why is there a Network Flight Recorder at all?

The problem the designers of NFR are trying to solve is the lack of tools to allow an organization to see what is really happening on their networks. There are tools that can capture data and tools that can search captured data, but these fall short of the mark. With the programability of NFR, organizations have the ability to implement a wide range of internal monitoring, archiving and alerting practices as required by the individual corporate needs. These flexible tools will become common place because of the position that networks are playing in today’s business.

Network monitoring is not just intrusion and misuse detection. Best business practices will become the norm as auditors try to put some structure to the electronic economy. Organization's are transitioning from paper to digital media, yet they still have the same business requirements they had before. For example, Companies will need to:


These challenges require network infrastructure tools. Organizations need the metrics that are often just beyond the scope of traditional network management tools: change detection, threshold exceptions, proactive capacity planning, activity logging, and usage statistics.

NFR provides a base foundation to accomplish these tasks using programmable packages "backends" to adapt the tool to specific tasks.


Traffic Analysis based intrusion detection compared to Content Analysis.

The network data collection and analysis system, is based on libpcap. This software package developed by the Network Research Group at Lawrence Berkeley Laboratory, provides an interface for application programs such as NFR to read data collected by the network interface on Unix systems. NFR has modified libpcap so that they can all of the headers and content of the packet. NFR application programs can concentrate their analysis on the headers of the collected packet instead of its content. This approach to intrusion detection is called traffic analysis. NFR application programs can also parse the content. This approach to intrusion detection is called content analysis.

The strength of a traffic analysis approach is that it is possible to examine and record every packet that passes by on the network. The weakness of this approach is that some attacks can only be detected by content analysis. Another approach to intrusion detection is to examine the contents of packets for certain strings that indicate an attack, this is a content, or string based approach. The strength of the content based approach is that it is able to collect all of the data of a connection once an attack string is detected. The weakness of this approach is that it is virtually impossible to analyze and record all of the traffic on a network, therefore many services, or ports are filtered out making the system blind to attacks using these ports or services. The cider systems are primarily based on traffic analysis, though some content is used.


Overview of Network Flight Recorder

NFR combines data collection and analysis and storage onto a single platform. These systems would (usually) be located outside your firewall and between your firewall and your internet connection, an area often called the DMZ. NFRs can also be placed internally in areas of high value to monitor and analyze potential insider attacks. We recommend that you install a small hub, or switch, to support the sensor. This way if there is a serious incident you will have a place to connect a dedicated sensor for evidence collection if needed.

Analysis is accomplished by scripts based on N-code, NFR's flexible language for traffic analysis. Information is displayed to a web based interface. NFR has a real time alerting capability and a storage subsystem that allows data to be stored, rotated, and archived to other external devices. Please see Appendix A How to write an NFR backend for further information.

NFR runs on several different Unix platforms. See for the most recent list. FreeBSD is considered the fastest, or most efficient, free platform for both NFR and the Step System due to FreeBSD's implementation of BPF (Berkeley Packet Filter). You should have at least 64MB of RAM, 192MB swap space and 9 GB of disk space. The machine should be dedicated to running the intrusion detection software.

"The fallacy of intrusion detection is that it's impossible for somebody who doesn't know your network to understand what really should and shouldn't happen on that network" Marcus Ranum


Step 1: Acquiring the software for Network Flight Recorder

The NFR software is available at no cost from Network Flight Recorder. It compiles on many Unix systems including BSDI 3 and above, Solaris 2.5 and above, HPUX 10.20 and above and FreeBSD. One thing worth mentioning about NFR is that it will compile with the manufacturer's "C" compiler if you do not have gcc.

Action 1.1 Download and install NFR from

Action 1.2 Download the ID example package from:

Action 1.3 Check the system for uneeded processes. Check inetd and crontabs and remove any process that is not absolutely needed. Updatedb can cause performance degradation and should be removed from cron files. Locate is a wonderful utility, but everything that you will be looking for should be under $NFRHOME.

Action 1.4 Install the NFR ID example package. Log in to the NFR system as the NFR user. Change directory to where you installed NFR, this should be $NFRHOME (cd $NFRHOME). Untar the example ID package ($ tar xvf nfr_id.tar).

Action 1.5 Configure the NFR ID example package. Change directory to the id directory ($ cd current/packages/id). The files with the extension ".nfr" are N-code backends that flight recorder uses to decide what actions to take on the network traffic it sees. The following .nfr files need to be configured: badhosts.nfr, badports.nfr, hostscan.nfr and portscan.nfr.

What the example ID Packages do:

badhosts.nfr: This file holds the IP addresses of systems that have attacked your site in the past and if NFR sees them it will trigger an alert. This is an excellent place to add your broadcast address(es) so that broadcast attacks and probes will be detected.

badports.nfr: Badports is a list of ports that are interesting and often signal an attack or probe. If your site routinely supports one of the ports listed, you should comment that out with a "#".

hostscan.nfr: Hostscan will detect an outside host connecting to multiple inside hosts. You should configure your internal networks with the variable watch_nets.

icmpfrag.nfr: Detects fragmented ICMP packets. Since ICMP packets should be small, this is an indicator something is amiss.

land.nfr: Detects the exploit in which the SRC and DST IP are the same.

portscan.nfr: Detects a system connecting to multiple ports on another system. The variables at the top of the file: checkFreq, portPurge, portMax, purgeFreq affect memory use. The larger the numbers here, the more memory your system requires.

failedlogins.nfr: Detects multiple failed telnet attempts.

manyresets.nfr: Detects port scans by implication. If a service such as FTP or Telnet or whatever does not exist on the destination host and a connection is attempted a RESET is returned. This can also detect attempts by sequence number predictors, the ones that fail anyway.



Action 1.6 Shutdown and restart NFR. su to root, cd to $NFRHOME, kill NFR (# sh etc/stop_nfr), wait for all NFR processes to complete, restart NFR (# sh etc/start_nfr). You may see errors about missing alert functions. Ignore them for the moment. We will fix that in the next step.

Action 1.7 Configure the alerts. Open your web browser, go to the NFR host and port (this defaults to port 2001), and log in (the default is username "nfr" and password "demo") We recommend that you change this at your earliest opportunity. Click on the "Start" button. Click on the configuration icon (the one in the middle with a wrench) and then the "Alert Configuration" button.

Action 1.7.1: Configure the alert sources: Click on the "User Sources"tab, then the "new" button. You will need to add the following sources:










To add the BAD_HOST source, type "BAD_HOST" in "Id String". Case is important. Type "Bad Host" in "Alert Source Name", then "Traffic to or from hosts where there should be none". Click the "Add" button.

Action 1.7.2: Configure the alert messages: Click on the "User Messages" tab, then the "new" button. Add the following message:Id String: SUSPICIOUS_ACTIVITY Message format String: This looks suspicious: $(1) Click on the "Warning" radio button under message severity, and the "NFR log facility" under message destinations. At this point, you probably don't want to have the alerts go into the acknowledgement system. Finally, click the "Apply" button, and (once the "Apply button" stops being greyed out), the "Quit" button. At this point, you can look at the messages either by clicking on the alert icon, or by looking at the file data/alerts/<machine name>/<date>/log.txt. Watch the alerts carefully, the values you entered in the .nfr files may require tweaking.


If you run into problems there is an NFR mailing list. To join the list send e-mail to, in the body of the message type "subscribe nfr-users". There is also a searchable archive of the mailing list. It is available at




Step 6. Display the information for maximum analytical value


Appendix A How to write a backend for NFR

Creation of a Backend, From Soup to Nuts----------------------------------------By M. Dodge Mumford, May, 1998. Rev 0.3Executive summary------------------This document goes through the steps necessary to

create a backend in Network Flight Recorder, a network traffic analysis

tool. What the backend will do is recognize when an attacker is attempting

to use the reader's network as an intermediary for a smurf attack (see

Introduction for details).






As is true with most of life, the really hard part of finding a solution

to a problem is breaking down the problem into small, manageable chunks.

The problem of acting as an accomplice for a smurf attack

( can be broken down

by asking the question "What does someone performing a smurf attack need

from my network?" The answer is "To be able to use any broadcast address."

The best way to tell if you are being used as a relay point for a smurf

attack is to monitor your broadcast addresses for use by any IP address that

is not on that local network.


Of course, it is rarely that simple. You may, for some reason, want to permit

a specific host to ping an entire subnet (perhaps as a quick way

to find out if all the hosts are alive). There needs to be a way to set up

such exceptions.


The traditional smurf attack uses ICMP echo requests. However, there is no

reason why a different IP protocol can't be used, so our backend will look

at all IP traffic.



What is a backend and how do we create one?



Backends exist within packages. A package is a group of backends grouped

together for convenience. Backends within a package can share code, or

they can operate totally independently of each other. In this intrusion

detection package, each backend works independently of the others.


For a package to exist, all that is required is a directory name and a .cfg

file. The contents of the .cfg file minimally needs to have the text







If enabled is set to "false", then none of that package's backends will be

loaded into the engine. Optionally, a package can also have a .nfr file,

containing package-wide global variables and N-Code. A package can also

optionally have a .desc file, which provides a text description of the package

and its functions.


For a backend to exist, it must be located within the package's directory. It

must have a .cfg and a .nfr file. A .desc file can also be used. The .cfg

file should minimally look like this:







(Hint-- if you don't want the backend to show up on the GUI, comment out the

title line).


A very minimalist (and mildly obtuse) .nfr file is:



echo ( "Hello, World!\n" ) ;



The next time you start and stop the NFR engine, "Hello, World" will be

appended to nfrd.log.



Let's create a backend.



We know that we're only going to want to look at specific IP addresses, so

the first thing we should do is define those IP addresses. Then we can

print those out to verify they get entered correctly. Create this

packages/id/new_badhosts.nfr file:



badhosts = [,,, ] ;

echo ( "This is badhosts: ", badhosts, "\n" ) ;



Rather than restarting the engine, you can use bin/test-nfrd to verify the

syntax and execute any non-packet related functions. If you type this, you

should get:



$ bin/test-nfrd packages/id/new_badhosts.nfr

This is badhosts: [,,,]




Hint: notice that the variable badhosts is of the type "list". Within a

single list, it is possible to have multiple types of variables--integers,

IP address, IP networks, strings, and even other lists. An example of a list

containing all of the above would be:



fred = [ 16842752,,, "Private Network",

[, ] ] ;




How do we know when a packet arrives?



For this next step, we want our N-Code to execute whenever the NFR engine sees

an IP packet. First, we will need to create a .cfg file for this backend.

Since this backend bears an uncanny resemblence to the badhosts backend,

we'll call this one new_badhosts. Put the following into





title=New Bad Hosts



Next, replace packages/id/new_badhosts.nfr with the following:



badhosts = [,,, ] ;


filter new_badhosts ip ( ) {

echo ( "Here 00: ", ip.src, "->", ip.dst, "\n" ) ;




To see this take effect, you will need to stop and restart the NFR engine (as

root, execute "sh etc/stop_nfr ; sh etc/start_nfr). If you 'tail nfrd.log',

you should see many entries like:



Here 00:>

Here 00:>

Here 00:>



The filter statement says to look at all IP packets, and execute the code

between the curly braces. The parentheses are there because the filter

statement is capable of doing basic, preliminary analysis. However, our

requirements exceeds that functionality in this case.


In the echo statement, you see a couple of variables we haven't defined. They

are supplied during run time by the engine. Their meaning should be fairly




But we only want to know about specific hosts



The next thing we want to do is ignore the packet if it is not one of the ones

we're interested in. Make the appropriate changes in

packages/id/new_badhosts.nfr so that it reads:



badhosts = [,,, ] ;


filter new_badhosts ip ( ) {

if ( ! ( ip.src inside badhosts || ip.dst inside badhosts ) )

return ;

echo ( "Here 00: ", ip.src, "->", ip.dst, "\n" ) ;




And restart the engine. Try pinging, and in nfrd.log you should


Here 00:>

Here 00:>

Here 00:>


The neat thing here is the way we tell if an IP address is inside a previously

defined network--using the keyword "inside".


The important thing here is to note the use of parenthesis in the if statement.

N-Code does not prioritize logical operators, so if you were to have said



if ( ! ip.src inside badhosts || ip.dst inside badhosts )



the end result would almost assuredly not be what you wanted.



Now it's triggering every time my internal hosts hit the broadcast addresses.



We need a bunch of exceptions, or people who are allowed to communicate with

to the broadcast addresses. Modify packages/id/new_badhosts.nfr to look like




badhosts = [,,, ] ;


exceptions [ ] = [ ] ;

exceptions [ ] = [ ] ;

exceptions [ ] = [, ] ;

exceptions [ ] = [,,, ] ;


filter new_badhosts ip ( ) {

if ( ! ( ip.src inside badhosts || ip.dst inside badhosts ) )

return ;

if ( ip.dst inside exceptions[ip.src] ||

ip.src inside exceptions[ip.dst] )

return ;


echo ( "Here 00: ", ip.src, "->", ip.dst, "\n" ) ;




Notice the introduction of arrays. Arrays can be of the types integer, string,

IP address, and IP network. Unfortunately, arrays are currently

uni-dimensional. But if you like, you can use a list as the key to an array.

For example,



foo [ [, ] ] = "Hi" ;



is valid.


What is happening in this instance is that the current packet's source and

destination IP address pair are compared against the table to see if they are

"allowed" to talk with each other. If so, N-Code exits.


Stop and restart the engine to see it work.



Great, so they're communicating. What kind of stuff are they saying?



Let's say we want to know what IP protocol is being communicated. If it is a

protocol that uses ports, we want to record that information, too. And, just

for fun, if it's a TCP packet, let's get the TCP flags. A quick

glance at /etc/protocols tells us what protocols are valid for IP networks.

Locating and muddling through RFC 793 tells us where within the TCP headers

the TCP flags are, and what order they should appear in.


After the exceptions definitions, but before the filter statements, add:



proto_translate[0] = "IP" ;

proto_translate[1] = "ICMP" ;

proto_translate[2] = "IGMP" ;

proto_translate[3] = "GGP" ;

proto_translate[6] = "TCP" ;

proto_translate[12] = "PUP" ;

proto_translate[17] = "UDP" ;

proto_translate[22] = "IDP" ;

proto_translate[255] = "RAW" ;



and replace the echo statement with (yes, really, all of it):



if ( ip.protocol == 6 || ip.protocol == 17 ) {

$sport = short ( ip.blob, 0 ) ;

$dport = short ( ip.blob, 2 ) ;

} else {

$sport = 0 ;

$dport = 0 ;


$flagString = "" ;

if ( ip.protocol == 6 ) {

$TCPflags = byte ( ip.blob, 13 ) ;

if ( $TCPflags & 1 )

$flagString = cat ( $flagString, "fin " ) ;

if ( $TCPflags & 2 )

$flagString = cat ( $flagString, "syn " ) ;

if ( $TCPflags & 4 )

$flagString = cat ( $flagString, "rst " ) ;

if ( $TCPflags & 8 )

$flagString = cat ( $flagString, "psh " ) ;

if ( $TCPflags & 16 )

$flagString = cat ( $flagString, "ack " ) ;

if ( $TCPflags & 32 )

$flagString = cat ( $flagString, "urg " ) ;



$protoString = "Unknown" ;

$protoString = proto_translate[ip.protocol] ;

echo ( "Here 00: ", ip.src, "->", ip.dst, ": ",

$protoString, "\n" ) ;



Restart the engine, ping the broadcast address, try telnetting to it, and in

nfrd.log you should see something like:



Here 00:> ICMP

Here 00:> ICMP

Here 00:> TCP syn

Here 00:> TCP syn



Interesting things we've added here that are noteworthy:


ip.protocol is a pointer to the appropriate byte of the IP packet, as described

in RFC 791.


ip.blob contains the entire contents of the IP packet payload. According to

RFCs 793 and 768, if the IP packet is of the type TCP or UDP, the first 16

bits (or 2 bytes, or the first short) contain the source port number. The

second 16 bits (or 2 bytes, etc.) contain the destination port number. The

term "blob" is synonymous with the word "string" within N-Code.


You may have also noticed the "short" and "byte" functions. They return

the short value (16 bits) or byte value (8 bits) of a blob, starting at

a specific position.


With the ports, we also introduce a new kind of variable--one that begins

with a dollar sign ($). All other variables we have dealt with thus far have

either been defined automaticaly by the system (e.g. ip.src), or have been

defined globally, outside of a function or filter statement (e.g.

exceptions[]). Variables that start with a $ are local to that function, and

cannot be accessed from outside that function.


We have also introduced bitwise operators. '&' is a bitwise 'and' . For

example, ( 1 & 1 ) returns true, ( 2 & 1 ) returns false, ( 3 & 1 ) returns

true, and so forth. '|' is 'or', and '^' is xor,



Cool! How do I get it to record this stuff to disk?



Now things start getting a little more complicated, but not horribly so.

The first thing we're going to do is decide exactly what to record, field

by field. They will be recorded thusly:

- source IP address

- source IP port

- destination IP address

- destination IP port

- IP protocol

- TCP flags


First, change packages/id/new_badhosts.cfg (yes, .cfg, not .nfr). The .cfg

file describes to the GUI the format in which data has been recorded. It

should end up looking like this:






# implicit zero-eth column is time








column_1_label=Source Addr

column_2_label=Source Port

column_3_label=Dest Addr

column_4_label=Dest Port


column_6_label=TCP Flags










In the default distribution, there are two different types of recorders:

list and histogram. The list recorder is the simpler of the two; it

is more or less like a sequential database. When you query it, you get

a list of all the events matching your criteria. Histogram, on the other hand,

is designed to count events. When queried for only a specific IP address, for

example, it will show the IP address and the number of times that IP address

has been recorded. We will write this backend to use the list recorder.


num_columns defines the number of fields that are being recorded. Note that

each record automatically gets recorded with the system time, and that its

presence is assumed.


The column_n_type variable refers describes the type of data held in that

field. column_n_label gives a title to the field for the GUI.


If we tried to keep the data forever, the disk would fill up quickly.

Spaceman, the space management utility, will look at rollover_size and

rollover_time to decide when to rotate and archive the data.


The enginne will look at archive_path to find out where the data should be

saved. Unless the string starts with a slash, the $NFRHOME directory is

assumed to be the root directory. The various % macros should be used

consistently, so we know where the data is.


The current revision of the format of the data should be kept in cfversion.


Now edit packages/id/new_badhosts.nfr (yes, .nfr). Add the following after the

"proto_translate" but before the "filter" statements.



new_badhosts_schema = library_schema:new ( 1, [ "time", "ip", "int",

"ip", "int", "string", "string" ], scope() ) ;

new_badhosts_recorder = recorder (

"bin/list packages/id/new_badhosts.cfg",

"new_badhosts_schema" ) ;



Also, after the echo statement, add:



record system.time, host(ip.src), $sport, host(ip.dst), $dport,

$protoString, $flagString to new_badhosts_recorder ;



badhosts_schema is a variable that tells the recorder what format to record

the data in. Note that the first argument is always 1--this is for future

expansion. The next argument is time--remember how in the .cfg file it was

stated that the zero-eth implicit column is system time? The last argument is

the scope of the current function, easily referenced through the scope()



badhosts_recorder is a variable that contains the information about the

recorder, like where it is, what configuration file to use, and the name of the

variable that contains the schema.


The record statement actually records the information to disk.


To verify, restart the engine, start a Java enabled Web browser, start the

GUI, select the backend, and press the "Query" button. When the next window

appears, click on either the 'Display as Text' or 'Display as HTML' button.

You should see data like:



Thu Apr 30 16:45:59 1998 0 0 ICMP

Thu Apr 30 16:46:00 1998 0 0 ICMP

Thu Apr 30 16:46:01 1998 0 0 ICMP

Thu Apr 30 16:46:10 1998 8256 23 TCP syn

Thu Apr 30 16:46:13 1998 8256 23 TCP syn

Thu Apr 30 16:46:19 1998 8256 23 TCP syn

Thu Apr 30 16:46:27 1998 8275 23 TCP syn




Amazing! How do I trigger an alert on this?


After the record statement, add this:



$message = cat ( host(ip.src), "(", $sport, ") -> ",

host(ip.dst), "(", $dport, "): ", $protoString, " ",

$flagString ) ;


$message ) ;



The alert statement sends the contents of $message to the alerting system,

with a source of NEW_BAD_HOST and a message of SUSPICIOUS_ACTIVITY. Those are

defined through the GUI. Restart the engine, and start your browser, and go

to the NFR main window.


Click on the configuration icon (the wrench), then the "Alert Configuration"

button. Click on the "User Sources" tab. Click on the "New" button. Enter

"NEW_BAD_HOST" in the "Id String" field, and relevant text in the other two

fields. Then click Add.


Next, click on the User Messages tab. If you don't see a message called

SUSPICIOUS_ACTIVITY, click on the "new" button. put "SUSPICIOUS_ACTIVITY" in

the "Id String" field, and "This looks suspicious: $(1)" in the "message

format string" field. Then click on the NFR Log Facility check button. Finally,

click on the "Apply" button.


Now, try pinging one of the bad hosts. If you go into the alert section (from

the main NFR window), you should see your alert.



Wonderful! How do I set it up to e-mail me?


In the Alert Configuration window, click on the destinations tab and the "new"

button. Create a unique Id String (such as EMAIL_ME). Next to Display Name,

type "E-mail Me". The Facility Type should be "mail". Click the "add" button,

then the "apply" button. Click on the "Destinations" tab again, then highlight

"EMAIL_ME". Click the "Properties" button. Put your full e-mail address in the

recipient field; you can leave the others as they are. Click "OK", then



Try a ping and a few telnets. In a few minutes, you will receive an e-mail

message regarding your activity.






This backend barely touches the tip of what NFR can do. We have not looked at

the payload of the packet at all. We have filtered only on IP packets--it is

possible to look at the ethernet frame, as well as follow a TCP stream. But

this should describe in suitable detail how to create a new backend that looks

at those other network layers, you will just have to look up the specific

variable names. You will also want to look at the various backends included

with the distribution as examples.


Good Luck!





(C) 1998 Network Flight Recorder, Inc. All rights reserved. This document

may be distributed freely as long as this copyright notice remains intact.