Differences

This shows you the differences between two versions of the page.

--- networking:mtu [2022/06/29 09:19] – aperez
+++ networking:mtu [2025/10/04 15:48] (current) – aperez
@@ Line 1: / Line 1: @@
 **Maximum transmission unit (MTU)**
-A maximum transmission unit (MTU) is the largest packet or frame size, specified in octets (eight-bit bytes) that can be sent in a packet- or frame-based network such as the internet. The internet’s transmission control protocol (TCP) uses the MTU to determine the maximum size of each packet in any transmission. MTU is usually associated with the Ethernet protocol, where a 1500-byte packet is the largest allowed in it (and hence over most of the internet).
+{{ :networking:mtu-image-0.png?300 |}}
+{{ :networking:mtu-image-4.png?500 |}}
+{{ :networking:mtu-image-5.png?400 |}}
+{{ :networking:mtu-image-6.png?400 |}}
+{{ :networking:mtu-image-7.png?400 |}}
+{{ :networking:mtu-image-8.png?400 |}}
+{{ :networking:mtu-image-9.png?300 |}}
+**A maximum transmission unit (MTU)** is the largest packet or frame size, specified in octets (eight-bit bytes) that can be sent in a packet- or frame-based network such as the internet. The internet’s transmission control protocol (TCP) uses the MTU to determine the maximum size of each packet in any transmission. MTU is usually associated with the Ethernet protocol, where a 1500-byte packet is the largest allowed in it (and hence over most of the internet).
 One of the most common problems related to MTU is that sometimes higher-level protocols may create packets larger than a particular link supports, and you’ll need to make adjustments to make it work.
@@ Line 12: / Line 24: @@
+----
+**Xubuntu - Linux**
+<code bash>
+aperez@St-Francis:~$ ping -c 4 -M do -s 1472 8.8.8.8
+PING 8.8.8.8 (8.8.8.8) 1472(1500) bytes of data.
+bytes from 8.8.8.8: icmp_seq=1 ttl=116 time=8.30 ms
+bytes from 8.8.8.8: icmp_seq=2 ttl=116 time=8.06 ms
+bytes from 8.8.8.8: icmp_seq=3 ttl=116 time=8.53 ms
+bytes from 8.8.8.8: icmp_seq=4 ttl=116 time=8.32 ms
+--- 8.8.8.8 ping statistics ---
+packets transmitted, 4 received, 0% packet loss, time 3004ms
+rtt min/avg/max/mdev = 8.064/8.304/8.530/0.165 ms
+</code>
+----
+<note tip>
+**MTU & PMTU Validation (side note)**
+Quick reference on how to confirm the effective MTU and Path MTU (PMTU) end-to-end. Use these tests when enabling jumbo frames or troubleshooting connectivity.
+----
+**1) ICMP Check with ping**
+Command:
+<code bash>
+ping -M do -s 1472 8.8.8.8
+</code>
+Expected output (OK, MTU 1500):
+<code>
+PING 8.8.8.8 (8.8.8.8) 1472(1500) bytes of data.
+bytes from 8.8.8.8: icmp_seq=1 ttl=114 time=8.0 ms
+bytes from 8.8.8.8: icmp_seq=2 ttl=114 time=7.9 ms
+--- 8.8.8.8 ping statistics ---
+packets transmitted, 2 received, 0% packet loss
+</code>
+👉 If **1472 passes**, effective MTU ≈ **1500**.
+👉 If it fails (e.g., *Frag needed*), the link is forcing MTU <1500.
+----
+**2) Path MTU Discovery with tracepath**
+Command:
+<code bash>
+tracepath -n 8.8.8.8
+</code>
+**Case A — Standard network (PMTU 1500):**
+<code>
+: 192.168.1.1          0.45ms
+: 192.168.1.1          0.39ms pmtu 1500
+: 10.10.10.1           3.21ms
+: 8.8.8.8              7.84ms reached
+     Resume: pmtu 1500 hops 3 back 3
+</code>
+**Case B — Jumbo end-to-end (PMTU ~9000):**
+<code>
+: 10.0.0.1             0.22ms
+: 10.0.0.1             0.21ms pmtu 9000
+: 172.20.0.1           0.95ms
+: 8.8.8.8              6.74ms reached
+     Resume: pmtu 9000 hops 3 back 3
+</code>
+👉 If you see **`pmtu 1500`**, the path is limited to standard frames.
+👉 If you see **`pmtu 9000`** (or similar), jumbo frames are preserved across the path.
+👉 If it drops (e.g., 9000 → 1500 mid-path), a hop does not support jumbo.
+----
+**Validation summary**
+^ Test                  ^ Expected outcome                           ^
+| `ping -M do -s 1472`  | Successful reply ⇒ effective MTU ≈ 1500    |
+| `tracepath -n`        | Reports PMTU 1500 (standard) or 9000 (jumbo)|
+</note>
+----
+**Windows**
+<code dos>
+C:\Users\Administrator>ping 8.8.8.8 -f -l 1440
+Pinging 8.8.8.8 with 1440 bytes of data:
+Reply from 8.8.8.8: bytes=1440 time=8ms TTL=116
+Reply from 8.8.8.8: bytes=1440 time=8ms TTL=116
+Reply from 8.8.8.8: bytes=1440 time=8ms TTL=116
+Reply from 8.8.8.8: bytes=1440 time=8ms TTL=116
+</code>
+----
+<code dos>
+C:\Users\Administrator>ping 8.8.8.8 -f -l 1440
+Pinging 8.8.8.8 with 1440 bytes of data:
+Reply from 8.8.8.8: bytes=1440 time=8ms TTL=116
+Reply from 8.8.8.8: bytes=1440 time=8ms TTL=116
+Reply from 8.8.8.8: bytes=1440 time=8ms TTL=116
+Reply from 8.8.8.8: bytes=1440 time=8ms TTL=116
+</code>
+**Technical Explanation:**
+The parameter `-l` in Windows `ping` specifies only the ICMP payload (data).
+But the real MTU also includes protocol headers:
+  * **20 bytes** → IP header (source/destination IP, TTL, checksum, fragmentation flags).
+  * **8 bytes**  → ICMP header (type, code, checksum, sequence).
+Therefore:
+  Real MTU = Payload (-l) + 20 + 8
+  Example: `1440 + 28 = 1468` → Real MTU = **1468 bytes**
+👉 This is why you must always add **28 bytes** to the `-l` value to obtain the true MTU on the link.
+----
+> **Note: PMTU and PMTUD (Path MTU / Path MTU Discovery)**
+The *Path Maximum Transmission Unit (PMTU)* is the **largest IP packet size that can travel end-to-end without fragmentation**.
+It is determined by the **smallest MTU along the entire path**.
+Example:
+  * Link 1: MTU = 9000
+  * Link 2: MTU = 1500
+  * Link 3: MTU = 1400
+  → **PMTU = 1400 bytes**
+---
+**Path MTU Discovery (PMTUD):**
+A dynamic mechanism to discover the PMTU using the **DF (Don’t Fragment)** bit.
+Steps:
+  - Source sends a large packet with DF=1.
+  - If a router cannot forward due to its MTU, it discards the packet and replies with ICMP *Fragmentation Needed* (Type 3, Code 4).
+  - The source reduces the size until it learns the **real PMTU**.
+---
+**Verification on Linux**::
+  ping -M do -s 1472 8.8.8.8
+    → If it replies: path supports 1500 (1472+28 headers).
+    → If it fails: PMTU is smaller.
+  tracepath -n 8.8.8.8
+    → Displays estimated PMTU along the route.
+---
+**Common pitfalls**
+  * Blocking ICMP → breaks PMTUD (TCP sessions may hang).
+  * Tunnel/VPN overhead → reduces MTU (e.g., IPSec 1500 → ~1400).
+  * Misconfigured Jumbo Frames → one 1500 hop breaks 9000 end-to-end.
+---
+**In summary:**
+  * **PMTU** = max packet size without fragmentation.
+  * **PMTUD** = process to dynamically discover it using ICMP + DF.
+----
+{{ :networking:set-default-mtu.png?600 |}}
+{{ :networking:set-default-mtu -2.png?600 |}}
+{{ :networking:set-default-mtu -3.png?600 |}}
+{{ :networking:set-default-mtu -4.png?600 |}}
+{{ :networking:set-default-mtu -5.png?600 |}}
+**Command to test jumbo frame packets (9000 Bytes) in an Esxi 7.0**
+{{ :networking:set-default-mtu-vmware-7.png?600 |}}
+[[info.support.huawei.com/info-finder/encyclopedia/en/MTU.html|What Is Maximum Transmission Unit (MTU)?]]
 ----
@@ Line 28: / Line 235: @@
 So in our case, setting the jumbo frame to the size that will allow the packet to go through is enough, don't worry about the IP MTU.
-{{ :networking:mtu-image-1.png?600 |}}
+{{ :networking:mtu-image-1.png?500 |}}
 **As mentioned, the common value of MTU in the internet is 1500 bytes**.
@@ Line 36: / Line 243: @@
 Now, imagine that we are using the GRE protocol in our network, encapsulating the original packet and adding 24 bytes for the GRE header.
-{{ :networking:mtu-image-2.png?600 |}}
+{{ :networking:mtu-image-2.png?400 |}}
 The total size of this kind of packet will be 1524 bytes, exceeding the 1500 bytes MTU value. The “data” size in this packet is 1460, but we can and should decrease it in order to make sure the total size will be 1500 bytes or less. And this is where TCP MSS comes into the picture.
-TCP MSS, the maximum segment size, is a parameter of the options field of the TCP header that specifies the largest amount of data, specified in bytes, that a computer or communications device can receive in a single TCP segment. It does not include the TCP header or the IP header. This value will dictate the maximum size of the “data” part of the packet. In the following case for the GRE tunnel, we will set the tcp mss value to be 1436 or lower, while the default size is 1460.
+**TCP MSS, the maximum segment size**, is a parameter of the options field of the TCP header that specifies the largest amount of data, specified in bytes, that a computer or communications device can receive in a single TCP segment. It does not include the TCP header or the IP header. This value will dictate the maximum size of the “data” part of the packet. In the following case for the GRE tunnel, we will set the tcp mss value to be 1436 or lower, while the default size is 1460.
 The MSS announcement (often mistakenly called a negotiation) is sent during the three-way handshake by both sides, saying: “I can accept TCP segments up to size x”. The size (x) may be larger or smaller than the default. The MSS can be used completely independently in each direction of data flow.
@@ Line 75: / Line 282: @@
 [[https://www.cisco.com/c/en/us/support/docs/lan-switching/8021q/17056-741-4.html|Inter-Switch Link and IEEE 802.1Q Frame Format]]
 ----
+**Configuring TCP MSS for Incapsula Infrastructure Protection**
+In Incapsula Infrastructure Protection, we usually use GRE tunnels. Therefore, we need to address the possibility of fragmentation if we are not adjusting the TCP MSS. Our solution, which is asymmetric by definition (meaning in-bound traffic via Incapsula, outbound traffic via the original ISP) requires us to rewrite the TCP MSS value of SYN packets that are being sent by our protected customer to their end clients. We can achieve this in a Cisco Router, for example by configuring “ip tcp mss-adjust 1436” on the WAN interface or the interface that faces the provider.
+Let’s look at the following diagram.
+{{ :networking:mtu-image-3.png?600 |}}
+As you can see, the customer server sends the packet with the MSS value of 1460, but in the router’s interface we will adjust it to 1420. It’s important to understand that this value will affect the size of the traffic coming from the end client toward the customer’s server. This is the traffic that will go via the tunnel and therefore be encapsulated with 24 bytes of GRE. The outbound traffic, from the customer’s server toward the end client, goes directly via the original ISP, therefore we don’t need to adjust the SYN packets the end user sends as well.
+The diagram above shows how the SYN packets in the three-way handshake travel. After the three-way handshake is completed and the connection established, the end user will send packets whose “data” won’t exceed the 1420 bytes size. In addition the customer’s server will send packets whose “data” won’t exceed the default 1460 bytes.
+A common mistake is to configure the “ip tcp mss-adjust 1420” command on the tunnel interface of the customer. The reason it doesn’t work is because the SYN packets that are sent from the server toward the end user are not going via the GRE tunnel interface, but via the original ISP’s interface.
+----
+----
+{{ :cisco:switch:9500:mtu_utm_switch_6400_9500.pdf |}}
+{{pdfjs 46em >:cisco:switch:9500:mtu_utm_switch_6400_9500.pdf }}
+----
+----
+{{ :aruba_networks:switch:6400:mtu_pmtu.pdf |}}
+{{pdfjs 46em >:aruba_networks:switch:6400:mtu_pmtu.pdf}}
+----
+----