netbenches

FreeBSD network forwarding performance benchmark results

This repository includes bench scripts (need netmap pkt-gen and nanobsd like), bench configuration files, raw results and gnuplot graphs on few setups. Bench script documentation is available too.

Hardware benched are:

• SuperMicro X10SRW-F (16 cores Intel Xeon E5-2697A v4 @ 2.60GHz) and: 40G Chelsio T580-LP-CR, 100G Mellanox ConnectX-4 MCX416A-CCAT
• Dell PowerEdgre R630 (2x12 cores Intel Xeon E5-2650 v4 @ 2.2GHz) and: Intel 82599, Chelsio T520-CR, Mellanox ConnectX-3 and ConnectX-4 (MCX4121A-XCA)
• HP ProLiant DL360p Gen8 (8 cores Intel Xeon E5-2650 @ 2.60GHz) and 10-Gigabit Chelsio T540-CR
• SuperMicro X9SRL-F (10 cores Intel Xeon E5-2650L v2 @ 1.70GHz) and 10-Gigabit Chelsio T540-CR
• SuperMicro 5018A-FTN4 (8 cores Atom C2758 @ 2.40GHz) with:
  • 10-Gigabit Chelsio T540-CR
  • 10-Gigabit Intel 82599
  • 10-Gigabit Chelsio T520-SO
• Aoostar WRT MAX (AMD_Ryzen7-Pro-8845HS_8Cores with 10G Intel X710)
• Netgate RCC-VE 4860 (4 cores Intel Atom C2558 @ 2.40GHz) and quad port Intel i350 (the dual port Intel i211 are not benched)
• PC Engines APU2C4 (4 cores AMD GX-412TC @ 1GHz) and 4 Intel i210AT Gigabit NIC

Specific benches:

• Impact of firewalls (ipfw/pf/ipf) on forwarding performance on:

  • 16 cores Xeon E5-2697A
    • 100G Mellanox ConnectX-4 MCX416A-CCAT
    • 40G Chelsio T580
  • 2x12 cores Xeon E5-2650
    • 10G Mellanox ConnectX-4 Lx
    • 10G Chelsio T520-CR
    • 10G Intel 82599ES
  • 10 cores Xeon E5-2650L
    • 10G Chelsio T540
  • 8 cores AMD Ryzen7 Pro 8845HS
    • 10G Intel X710
  • 8 cores Xeon E5-2650
    • 10G Chelsio T540
  • 8 cores Atom C2758
    • 10G Chelsio T540
    • 10G Chelsio T520-SO (FreeBSD 16-CURRENT n313366 / BSDRP 2.3)
    • 10G Chelsio T520-SO (FreeBSD 15-n302145)
    • 10G Intel 82599 (FreeBSD 16-CURRENT n313366 / BSDRP 2.3)
    • 10G Intel 82599 (FreeBSD 15-n302145)
    • 10G Intel 82599 (FreeBSD 14-c276570)
  • 4 cores Atom C2558
    • Intel i350
  • 4 cores AMD GX-412TC
    • Intel i210 (FreeBSD 16-CURRENT n313366 / BSDRP 2.3)
    • Intel i210 (FreeBSD 16-CURRENT n311215, no option RSS - direct dispatch)
    • Intel i210 (FreeBSD 16-CURRENT n311066, option RSS + net.isr tuning)
    • Intel i210 (FreeBSD 13-c255632)
  • (old)4 cores Xeon L5630
  • (old)2 cores AMD G-T40E

• One year evolution:

  • One year (2017) of forwarding performance with 8 cores Xeon E5-2650 & Chelsio T540
  • One year (2017) of forwarding performance with 4 cores Atom 2558 & Intel i350
  • One year (2016) of forwarding performance with 8 cores Atom 2758 & Chelsio T540
  • One year (2016) of forwarding performance with 4 cores Atom 2558 & Intel i350
  • One year (2016) of forwarding performance with 4 cores AMD GX-412TC & Intel i210
  • One year (2015) of fastforwarding performance evolution with with 8 cores Xeon E5-2650
  • One year (2015) of fastforwarding performance evolution with 4 cores Atom C2558

• Hyper Threading impact:

  • 16 cores Xeon E5-2697A & 100G Mellanox ConnectX-4
  • 10 cores Xeon E5-2650L & Chelsio T540
  • 8 cores Xeon E5-2650 & Chelsio T540
  • 8 cores AMD Ryzen7 Pro 8845HS & Intel X710

• IPFW-at-NIC-level (DDoS protection)

  • 16 cores Xeon E5-2697A
    • 100G Mellanox ConnectX-4
    • 100G Chelsio T580
  • 10 cores Xeon E5-2650L & Chelsio T540

• comparing 12-head and 11-stable:

  • fbsd12-head vs 11-stable r318516 with 4 cores AMD GX-412TC

• comparing 11.0 and 11.1-BETA1:

  • FreeBSD 11.0 vs 11.1-BETA1 on a 8 cores Xeon E5-2650 with Chelsio T540
  • FreeBSD 11.0 vs 11.1-BETA1 on a 4 cores AMD GX-412TC with Intel i210

• Impact of entropy sources on forwarding performance:

  • FreeBSD 15
    • 8 cores AMD Ryzen 7 with Intel X710
  • FreeBSD 11.1
    • 8 cores Xeon E5-2650
    • 8 cores Atom C2758
    • 4 cores Atom C2558
    • 4 cores AMD GX-412TC

• Impact of adding VIMAGE support (without using it) on FreeBSD 11.1 forwarding&firewalling performance:

  • 8 cores Xeon E5-2650
  • 8 cores Atom C2758
  • 4 cores Atom C2558
  • 4 cores AMD GX-412TC
  • 4 cores Xeon L5630

• Impact of Intel's TurboBoost on FreeBSD 11-STABLE r327668:

  • 8 cores Atom C2758
    • Chelsio T540
    • 82599

• Impact of if_bridge on FreeBSD 11.1:

  • 8 cores cores Xeon E5-2650
  • 8 cores Atom C2758 & Chelsio T540

• Impact of enabling ALTQ on FreeBSD 11-STABLE r332393:

  • 4 cores Atom C2558
  • 4 cores AMD GX-412TC

• Impact of number of static routes on FreeBSD 10.2:

  • 4 cores Xeon L5630

• Mellanox ConnectX-4 tuning:

  • Impact of queue number on forwarding performance with 2x12 cores Xeon E5-2650

• Chelsio NIC T540-CR tuning:

  • Impact of queue number on forwarding performance with 8 cores Xeon E5-2650
  • Impact of queue number on forwarding performance with 8 cores Atom C2758
  • Impact of disabling cxgbe.toecaps_allowed on forwarding performance with 8 cores Xeon E5-2650
  • Impact of pinning IRQ queue to CPU with 8 cores Xeon E5-2650
  • Impact of pinning IRQ queue to CPU with 8 cores Atom C2758

• Intel 82599EB NIC tuning:

  • Impact of Intel 82599EB queue number (1 queue per core) on forwarding performance with 4 cores Xeon L5630
  • Impact of random.harvest.mask and Intel 82599EB queue number (1 queue per core) on forwarding performance with 4 cores Xeon L5630
  • Impact of Intel 82599EB AIM on forwarding performance with 4 cores Xeon L5630
  • Impact of Intel 82599EB Rx|Tx process limit on forwarding performance with 4 cores Xeon L5630
  • Impact of Intel 82599EB descriptors per queue on forwarding performance with 4 cores Xeon L5630
  • Impact of AIM on Intel 82599EB

• Intel 82599 tuning:

  • Impact of iflib tx_abdicate on forwarding performance with 8 cores Atom C2758 (FreeBSD 16-CURRENT n313366 / BSDRP 2.3: +21.3%)
  • Impact of iflib simple_tx on forwarding performance with 8 cores Atom C2758 (FreeBSD 16-CURRENT n313366 / BSDRP 2.3: +13.2%, still 7.1% below tx_abdicate)

• System tuning:

  • Impact of harvest_mask (entropy sources) on forwarding performance with 8 cores Atom C2758 / Chelsio T520-SO (FreeBSD 16-CURRENT n313366 / BSDRP 2.3: no difference proven)

• Intel i350 tuning:

  • Impact of Intel i350 number of queues(1 queue per core) on forwarding performance with 4 cores Atom C2558
  • Impact of AIM on Intel i350

• Intel i210 tuning:

  • Impact of iflib tx_abdicate on forwarding performance with 4 cores AMD GX-412TC (FreeBSD 16-CURRENT n311215)
  • Impact of iflib simple_tx on forwarding performance with 4 cores AMD GX-412TC (FreeBSD 16-CURRENT n311215)
  • Impact of D58513 (e1000 UDP RSS hash type) on UDP forwarding, with simple_tx, 4 cores AMD GX-412TC (FreeBSD 16-CURRENT n311215)
  • Impact of D58513 (e1000 UDP RSS hash type) on UDP forwarding, with tx_abdicate, 4 cores AMD GX-412TC (FreeBSD 16-CURRENT n311215)
  • Impact of iflib simple_tx on forwarding performance with 4 cores AMD GX-412TC (FreeBSD 16-CURRENT n312956: +78% once D58513 landed)
  • Impact of D58901 (iflib bounded buf_ring for simple_tx) on UDP forwarding, 4 cores AMD GX-412TC (FreeBSD 16-CURRENT n312956: -11.7% regression)

• Packet filter (pf) tuning:

  • Impact of states_hashsize table on pf performance on FreeBSD 12-head r332390 on Atom 8 core
  • Impact of states_hashsize table on pf performance on FreeBSD 12-head r332390 on AMD GX
  • Impact of number of flows on pf performance on FreeBSD 11.1-yandex

• ipfw tuning:

  • Impact of number of flows with maximum ipfw hash table on FreeBSD 11.1-yandex

• IPsec performance:

  • 4 cores AMD GX-412TC (PC Engines APU2), FreeBSD 16-CURRENT n313366, VTI (route-based), one ESP tunnel per address family, IPv4 and IPv6: VTI beats policy-based by 4 to 13% on IPv4 and 9 to 31% on IPv6
  • 4 cores AMD GX-412TC (PC Engines APU2), FreeBSD 16-CURRENT n313366, policy-based (spdadd), IPv4 and IPv6: AES-CBC 3.6x faster than 13-head
  • SuperMicro 5018A-FTN4 (8 cores Atom C2758) and 10G Chelsio T540-CR, FreeBSD 16-CURRENT n313366, VTI (route-based), one ESP tunnel per address family: the first VTI set here that really measures IPv6 IPsec, and IPv6 still forwards 6 to 14% more packets per second than IPv4 whenever crypto is on
  • SuperMicro 5018A-FTN4 (8 cores Atom C2758) and 10G Chelsio T540-CR, FreeBSD 16-CURRENT n313366, VTI (route-based), IPv4 at 2000 flows (its IPv6 column is withdrawn: both arms shared one IPv4 tunnel)
  • SuperMicro 5018A-FTN4 (8 cores Atom C2758) and 10G Chelsio T540-CR, FreeBSD 16-CURRENT n313366, VTI (route-based), IPv4 and IPv6: enabling Intel QuickAssist (QAT) costs 63 to 79% of the throughput, the single acceleration engine is slower than AES-NI and caps both families at the same packet rate
  • 4 cores AMD GX-412TC (PC Engines APU2C4)
  • 4 cores Atom C2558 (Netgate RCC-VE 4860)
  • 8 cores Atom C2758 & Chelsio T540
  • 8 cores Xeon E5-2650 & Chelsio T540
  • 2x12 cores Xeon E5-2650v4 & Chelsio T520
  • (old) 4 cores Intel Xeon L5630 (IBM x3550-M3)
  • (old) 2 cores AMD G-T40E (PC Engines APU1D)

• OpenVPN performance:

  • 4 cores AMD GX-412TC (PC Engines APU2), FreeBSD 16-CURRENT n313366, OpenVPN 2.7.7 userland vs DCO: DCO 9x to 19x faster
  • 4 cores AMD GX-412TC (PC Engines APU2C4)
  • 4 cores Atom C2558 (Netgate RCC-VE 4860)
  • 8 cores Atom C2758 & Chelsio T540
  • 4 cores Intel Xeon L5630 (IBM x3550-M3)
  • (old)2 cores AMD G-T40E (PC Engines APU1D)
  • Impact of OpenVPN fast-io (PC Engines APU2C4)

• Wireguard:

  • PC Engines APU2 (4 cores AMD GX-412TC @ 1GHz) and Intel i210AT, in-kernel if_wg, IPv4 and IPv6, FreeBSD 16-CURRENT n313366
  • (kernel vs userland) PC Engines APU2C4 (4 cores AMD GX-412TC @ 1GHz) and 4 Intel i210AT Gigabit NIC
  • SuperMicro 5018A-FTN4 (8 cores Atom C2758) and 10G Chelsio T540-CR, in-kernel if_wg, IPv4 and IPv6, FreeBSD 16-CURRENT n313366
  • Netgate RCC-VE 4860 (4 cores Intel Atom C2558 @ 2.40GHz) and quad port Intel i350
  • 8 cores Atom C2758 & Chelsio T540

• Tinc performance:

  • 4 cores AMD GX-412TC (PC Engines APU2C4)

Synthesis:

• 
• 
• 
• 
• 
•