IPQ806x Hardware acceleration
Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada
NSS acceleration model • Features • Designed for Home Gateways (CPE) • Flow detection based “All-or-nothing” offload • Acceleration supports: •
IPv4, IPv6, NAT, PPPoE, L2TP, VLAN, Qdisc
• Performance gain • Linux: 640k pps (bridged) – 220kpps (routed) • NSS: 7200k pps (bridged) – 7200kpps (routed) • 11x (bridged) – 32x (routed)
• Functional behavior • 0% cpu load seen in Linux • Keep Linux counters up to date • Does not require functional changes at an upper level (user space)
Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada
ECM Front End Inspect Packets and Events • Inspect all outgoing packets at POSTROUTING chain by registering post routing hooks. • Inspect conntrack and device events to destroy and regenerate connection. • Inspect NSS status and stats to update the connection state and statistics info in Linux and ECM DB.
IPCT_DESTROY / IPCT_MARK
INTERFACE NOTIFIER
BOND NOTIFIER
CONNTRACK NOTIFIER
LINUX CONNTRACK
MTU / UP/ DOWN /
Bond Link/ Release/ Enslave
L3 PKTs L2 PKTs
NETFILTER IPvX POST ROUTING
L3 PKTs
FRONT END
L2 PKTs
NETFILTER PF_BRIDGE POST ROUTING
NSS Status and Stats Sync NSS Driver
Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada
NSS Firmware
Example of IPv4 TCP rule creation Linux Linux net stack
ECM
Connection Established
Routing or Bridging Netfiter Postrouting
Netfiter Prerouting
Check TCP Connection can be accel?
NSS Driver Send the Packet to Linux
No TCP: ACK Or Syn+ACK TCP Data
Match Any Accelerated Rule?
Create IPv4 Rule
Transmit the packet
Add the rule & reply the establish cmd
Packet The Accelerated TCP flow
NSS Firmware
Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada
Example of IPv4 TCP rule destroy Linux Kernel
ECM
Linux net stack
Remove the connection
Conntrack destroy event
Conntrack
Connection Destroy ?
Routing / Bridging
Netfiter Postrouting
Netfiter Prerouting
NSS Driver Send the Packet to Linux
Destroy IPv4 Rule
IPv4 Rule Sync
Transmit The packet
No TCP: FIN
TCP Data
Is it a connected TCP Data?
Add the rule & reply the establish cmd
The Accelerated TCP flow
NSS Firmware
Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada
Packet
Example of IPv4 rule API • Common messaging interfaces • • •
Protocol type: IPv4, IPv6, PPP, LAG… Rule type: CREATE, DESTROY, CONN_STAT_SYNC , NODE_STAT_SYNC… Callback/args pointers: will be passed back in the FW ACK/NACK reply
• IPv4 rule create message structure example
struct nss_ipv4_rule_create_msg { /* Request */ uint16_t valid_flags;
struct nss_ipv4_protocol_tcp_rule tcp_rule; struct nss_ipv4_pppoe_rule pppoe_rule; struct nss_ipv4_qos_rule qos_rule; struct nss_ipv4_dscp_rule dscp_rule; struct nss_ipv4_vlan_rule vlan_primary_rule; struct nss_ipv4_vlan_rule vlan_secondary_rule;
/* Indicate which of the parameters below is filled-in Indirectly says which operation to be done on the flow */ /* Bit flags associated with the rule */ /* src_ip, dst_ip, src_port, dst_port, proto */ /* src_mac, dst_mac, src_iface, dst_iface, src_mtu, dst_mtu, nat_src_ip, nat_dst_ip, nat_src_port, nat_dst_port */ /* TCP related accleration parameters */ /* flow_session_id, flow_remote_mac, ret_session_id, ret_remote_mac */ /* flow_qos_tag, ret_qos_tag, */ /* flow_dscp, ret_dstp */ /* ingress_vlan_tag, egress_vlan_tag */ /* ingress_vlan_tag, egress_vlan_tag – for QinQ */
/* Response */ uint32_t index;
/*Slot ID for cache stats to host OS */
uint16_t rule_flags; struct nss_ipv4_5tuple tuple; struct nss_ipv4_connection_rule conn_rule;
};
Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada
Interfaces & Connections statistics update • Stats updates sent periodically from the Firmware • Per-interfaces stats (update net_devices) • Per-connections stats (update conntracks)
• Minor modifications to ppp/l2tp/ipsec… layers for iface look-up and stats update
conntracks
stats++
Linux ECM Look-up conntrack
ethN
ppp0 br0 tun0
Look-up interface
stats++ NSS Driver
NODE_STATS_SYNC message NSS Firmware Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada
CONN_STATS_SYNC message
Qdisc acceleration tbf
nsstbf
1:
10:
10:
prio 10:1 100:
tbf
f i p f o
r e d
1000:
nssprio
10:3
10:1
10:2
tbf
1:
100:
200: 300:
2000:
r e d
10:2
nsstbl
nsstbl
n f s i s f p o
n r s e s d
1000:
10:3
200: 300:
2000:
n r s e s d
# tc qdisc add dev eth0 root handle 1: tbf rate 1000Mbit burst 100k limit 100 # tc qdisc add dev eth0 root handle 1: nsstbl rate 1000Mbit burst 100k # tc qdisc add dev eth0 parent 1: handle 10: prio bands 3 # tc qdisc add dev eth0 parent 1: handle 10: nssprio bands 3 # tc qdisc add dev eth0 parent 10:1 handle 100: tbf rate 2Mbit burst 10k limit 100 # tc qdisc add dev eth0 parent 10:1 handle 100: nsstbl rate 2Mbit burst 10k # tc qdisc add dev eth0 parent 100: handle 1000: pfifo limit 100 # tc qdisc add dev eth0 parent 100: handle 1000: nsspfifo limit 100 # tc qdisc add dev eth0 parent 10:2 handle 200: tbf rate 40Mbit burst 30k limit 100 # tc qdisc add dev eth0 parent 10:2 handle 200: nsstbl rate 40Mbit burst 30k # tc qdisc add dev eth0 parent 200: handle 2000: red limit 100k min 30k max 80k \ # tc qdisc add dev eth0 parent 200: handle 2000: nssred limit 100k min 30k max 80k \ avpkt 1k burst 55 probability 0.20 avpkt 1k burst 55 probability 0.50 # tc qdisc add dev eth0 parent 10:3 handle 300: red limit 100k min 30k maxof80k # tcOttawa, qdiscOn,add dev eth0 parent 10:3 handle 300: nssred limit 100k min 30k max 80k \ Proceedings netdev\0.1, Feb 14-17, 2015, Canada avpkt 1k burst 55 probability 0.30 avpkt 1k burst 55 probability 0.50 set_default