[EN] FreeBSD QOS with FreeBSD DUMMYNET

The latest released FreeBSD version 6.2  comes with 3 firewall systems:

- pf (ported from OpenBSD)
- ipfw
- ipfilter

All these three are stable as of this release. I've recently needed to run a QOS system, and chosen
ipfw "DUMMYNET" for this purpose.

My network is about couple hundreds computers and I needed limit bandwidth for each user.

I chose DUMMYNET because of it's simplicity, and "dynamic pipe" creation. I know "pf" (especially HFSC alghoritm) is a good solution too, but natively it supports up to 64 queues
per interface, and you need to patch the source in order to enable more. It's safe to use hundreds of queues, but I simply don't like patching.. (at least by now).

DUMMYNET provides FreeBSD users with dynamic pipe creation.

Suppose you want to create a new 256Kbit pipe for incoming, and 128Kbit pipe for outgoing traffic.

# ipfw pipe 1 config bw 256Kbit/s mask dst-addr 0xffffffff
# ipfw pipe 1 config bw 128Kbit/s mask src-addr 0xffffffff

Now, you need to put the traffic into these pipes.

# ipfw add pipe 1 ip from any to 192.168.0.0/24, 192.168.1.0/24
# ipfw add pipe 2 ip from 192.168.0.0/24, 192.168.1.0/24 to any

By these 4 lines of code you will get hunderds of pipes for each ip in the network for
both incoming and outgoing traffic.

You could write it this way as well:

# ipfw pipe 1 config bw 256Kbit/s
# ipfw pipe 2 config bw 256Kbit/s
...
# ipfw pipe 345 config bw 256Kbit/s

and appropriate rules:
# ipfw add pipe 1 ip from any to 192.168.0.2
# ipfw add pipe 2 ip from any to 192.168.0.3
....
# ipfw add pipe 345 ip from any t0 192.168.1.23

I think the first solution is better!

Now that we stand with another problem - some users have more bandwidth enabled, some less.

You could write pipes with mask-src (dynamic), and put hosts into these.. which would cause lots of rules written.

ipfw (ipfw2 for FreeBSD 4.X) comes with lookup tables, which are extremely fast!

There are 2 values stored in each record of the table:
ip address or network, integer

To create a new table and put the record you would:

# ipfw table 1 add 192.168.0.2 23
# ipfw table 1 add 192.168.0.3,345

These 2nd values lets filter by these in the rules.

# ipfw add allow ip from table\(1\,23) to any # ( = ipfw add allow ip from 192.168.0.2 to any)

Now, in order to create more complex ruleset for pipe assigning look at the example below:

# incoming traffic
# ipfw pipe 1 config bw 1024Kbit/s mask dst-addr 0xffffffff
# ipfw pipe 2 config bw 512Kbit/s mask dst-addr 0xffffffff
# ipfw pipe 3 config bw 256Kbit/s mask dst-addr 0xffffffff

# outgoing traffic
# ipfw pipe 11 config bw 1024Kbit/s mask src-addr 0xffffffff
# ipfw pipe 12 config bw 512Kbit/s mask src-addr 0xffffffff
# ipfw pipe 13 config bw 256Kbit/s mask src-addr 0xffffffff

# Create 2 lookup tables, one with values (IP,INCOMING_BANDWIDTH), second with (IP, OUTGOING_BANDWIDTH)
# TABLE 1 (incoming)
# ipfw table 1 add 192.168.0.2 1024
# ipfw table 1 add 192.168.0.3 1024
# ipfw table 1 add 192.168.1.0/24 512 # each host in network 192.168.1.0/24 gets 512 Kbit/s)
....
# ipfw table 1 add 192.168.2.23 256

# TABLE 2 (outgoing)
# ipfw table 2 add 192.168.0.2 512
# ipfw table 2 add 192.168.0.3 512
# ipfw table 2 add 192.168.1.0/24 256 # yeah, network 192.168.1.0 got too much traffic last month, now they will get less

# RULES
# rule for incoming pipe 1024kbit (pipe #1)
# ipfw add pipe 1 ip from any to table\(1,1024\)
# rule for incoming pipe 512 (pipe #2)
# ipfw add pipe 2 ip from any to table\(1,512\)
# rule for incoming pipe 256 (pipe #3)
# ipfw add pipe 3 ip from any to table\(1,256\)

# outgoing traffic
# fule for outgoing pipe 1024Kbit (pipe #11)
# ipw add pipe 11 ip from table\(2,1024\) to any
# rule for outgoing pipe 512Kbit (pipe #12)
# ipfw add pipe 12 ip from table\(2,512\) to any
# rule for outgoing pipe 256Kbit (pipe #13)
# ipfw add pipe 13 ip from table\(2,256\) to any

That's all! No more hundreds of static pipes, hundreds of rules.

Keep in mind, that this example is a strcitly pipe related.
Depending on the NAT you are using, make sure these rules are evaluated
before being nated (if using "natd") for outgoing traffic, and after being nated (if using "natd")
for incoming traffic.

You can get this by using rules like:

# ipfw add 100 divert natd ip from any to any in recv $OUTGOING_INTERFACE

Put all the pipe rules here

# ipfw add 4000 divert natd ip from any to any out xmit $OUTGOING_INTERFACE

NOTE:
1. Remember to set sysctl variable "net.inet.ip.fw.one_pass" to 0, which prevents packets from
not being reinjected into the firewall.

2. You may also want to put a "skipto" rule after ip to pipe assigning (if you are using lots of such rules).

3. Personally I'm using ipnat+dummynet, which works fine for bigger networks. natd is not
fast, as it works in userland. ipnat is entirely kernel attached, so it's faster.

4. If using FreeBSD 4.x make sure you apply the ipfw/ipfilter order patch (lookup for "ipfw ipnat order patch" in google)

5. Assign appropriate "queue" values for each pipe depending on the pipe bw parameter.

6. In FreeBSD 5.5? 6.X you can even simplify the ruleset, read about "tablearg" parameter for ipfw!

7. Tune all "net.inet.ip.dummynet.*" sysctls if there are lots of hosts in your network, and there's a danger of creating lots of dynamic pipes.

8. Some guys use pf+dummynet for FreeBSD 6.X nat. I think it's tricky too, as pf is a great firewall too!

Some useful links:
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/firewalls-ipfw.html
http://info.iet.unipi.it/~luigi/ip_dummynet/