ECMP voodo magic

Hi! Today I would like to talk about ECMP in Cisco IOS. As you probably know there are two different methods – per-packet and per-destination load balancing. http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a0080094820.shtml

Default method for IOS is to use per-destination load balancing with CEF. It is possible also to configure per-packet load balancing with CEF, or you can disable CEF and fast-switching to do per-packet.

Let’s look at the following simple topology:

Figure 1. ECMP topology

There is no really nothing unclear with except of two facts. On R1 ECMP is working.  R5 prefer R3 as the default route.   Routing protocol used here has no difference to achieve this configuration.  You can use static routing for simplicity.

Let’s look at the Router2 cef table for prefix 196.254.1.1:

Router2#sh ip cef 196.254.1.1/32 detail

196.254.1.1/32, epoch 0

  NetFlow: Origin AS 0, Peer AS 0, Mask Bits 32

  nexthop 9.8.7.217 GigabitEthernet0/1.996

Then let’s look at the traceroute (1):

Router2#traceroute 196.254.1.1 numeric source l0

Type escape sequence to abort.

Tracing the route to 196.254.1.1

  1 9.8.7.217 0 msec 4 msec 0 msec

  2 172.29.94.83 4 msec 0 msec 4 msec

  3 196.254.1.1 4 msec 0 msec 0 msec

As we expected it, it is per-destination and all three packets gone through 172.29.94.83 gateway, although on Router2 equal cost multipath load sharing is working.  Let’s look at the Router1 CEF table for prefix 196.254.1.1/32 and confirm this:

Router1#sh ip cef 196.254.1.1/32 detail

196.254.1.1/32, epoch 0, per-destination sharing

  NetFlow: Origin AS 0, Peer AS 0, Mask Bits 32

  nexthop 172.29.94.83 GigabitEthernet0/1.324

  nexthop 172.29.94.84 GigabitEthernet0/1.324

Let’s look at the traceroute results now (2):

Router1#traceroute 196.254.1.1 numeric source l0

Type escape sequence to abort.

Tracing the route to 196.254.1.1

  1 172.29.94.84 0 msec

    172.29.94.83 4 msec

    172.29.94.84 4 msec

  2 196.254.1.1 0 msec 0 msec 4 msec

Router1#

Do you see this strange behavior? Nothing strange really except it is per-packet, not per-destination as we was expecting.

Let’s recheck it again with extended ping command on Router2 (3):

Router2#ping

Protocol [ip]:

Target IP address: 196.254.1.1

Repeat count [5]:

Datagram size [100]:

Timeout in seconds [2]:

Extended commands [n]: y

Source address or interface: loopback0

Type of service [0]:

Set DF bit in IP header? [no]:

Validate reply data? [no]: yes

Data pattern [0xABCD]:

Loose, Strict, Record, Timestamp, Verbose[none]: Record

Number of hops [ 9 ]:

Loose, Strict, Record, Timestamp, Verbose[RV]:

Sweep range of sizes [n]:

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 196.254.1.1, timeout is 2 seconds:

Packet sent with a source address of 172.22.255.100

Reply data will be validated

Packet has IP options:  Total option bytes= 39, padded length=40

 Record route: <*>

   (0.0.0.0)

   (0.0.0.0)

   (0.0.0.0)

   (0.0.0.0)

   (0.0.0.0)

   (0.0.0.0)

   (0.0.0.0)

   (0.0.0.0)

   (0.0.0.0)

Reply to request 0 (4 ms).  Received packet has options

 Total option bytes= 40, padded length=40

 Record route:

   (9.8.7.218)

   (172.29.94.81)

   (172.27.175.1)

   (196.254.1.1)

   (196.254.1.1)

   (172.29.94.83)

   (9.8.7.217)

   (172.22.255.100) <*>

   (0.0.0.0)

 End of list

Reply to request 1 (4 ms).  Received packet has options

 Total option bytes= 40, padded length=40

 Record route:

   (9.8.7.218)

   (172.29.94.81)

   (172.27.175.2)

   (196.254.1.1)

   (196.254.1.1)

   (172.29.94.83)

   (9.8.7.217)

   (172.22.255.100) <*>

   (0.0.0.0)

 End of list

Reply to request 2 (4 ms).  Received packet has options

 Total option bytes= 40, padded length=40

 Record route:

   (9.8.7.218)

   (172.29.94.81)

   (172.27.175.1)

   (196.254.1.1)

   (196.254.1.1)

   (172.29.94.83)

   (9.8.7.217)

   (172.22.255.100) <*>

   (0.0.0.0)

 End of list

Reply to request 3 (4 ms).  Received packet has options

 Total option bytes= 40, padded length=40

 Record route:

   (9.8.7.218)

   (172.29.94.81)

   (172.27.175.2)

   (196.254.1.1)

   (196.254.1.1)

   (172.29.94.83)

   (9.8.7.217)

   (172.22.255.100) <*>

   (0.0.0.0)

 End of list

Reply to request 4 (4 ms).  Received packet has options

 Total option bytes= 40, padded length=40

 Record route:

   (9.8.7.218)

   (172.29.94.81)

   (172.27.175.1)

   (196.254.1.1)

   (196.254.1.1)

   (172.29.94.83)

   (9.8.7.217)

   (172.22.255.100) <*>

   (0.0.0.0)

 End of list

Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/4 ms

Now it is again very confusing! It is per-packet load sharing again. Packets go through 172.27.175.1 and 172.27.175.2 in round-robin fashion! What’s wrong with it?

As I already mentioned, nothing strange there. Answer is simple – packet destined to device itself is processed by control plane of the router and control plane packet is process switched on CPU on IOS. If you know this fact you can understand this behavior easily. In (1) packets processed through router are simply managed by data plane, TTL exceeded is sent.  In (2) and (3) packets processed by R1 control plane. That’s all. There is no cisco voodoo magic.