MTR is an excellent tool for identifying packet-loss or latency suspects when troubleshooting networking path issues. However, I’ve seen many who don’t really understand what MTR is doing (or even engineers) act on false positives produced from this tool. I’ve learned these lessons the hard way about MTR and iperf both. Ideally, MTR is used when packet-loss or latency is first detected using simple tools like ping or hping. It’s so effective however that some people turn to it immediately, and I want to explain why this is a problem and highlight a common false positive.
The take-away here though, MTR is a tool. If we don't understand the tools we use, we can't accurately understand their output. Ideally, any Engineer/Admin/Dev should ensure they understand how MTR output is produced if they're going to rely on it for actionables and investment of time while troubleshooting.
How MTR works
This post assumes you understand how to interpret MTR results, and that ‘packet-loss’ or latency on a single hop in the path is not a problem. But just to ensure it’s clear, let’s review what MTR does.
- MTR sends ICMP Echo request OR TCP SYN packets to a target IP.
- It sends the first packet with an IP TTL of 1, the second with 2, third with 3, etc.
- This continues until the target IP responds, at which the expected TTL is established. There are some logic checks here and additional packets with yet incremented TTLs can be sent.
- Meanwhile, most of the packets with a TTL not large enough to reach the target IP solicit an ICMP TTL expiry message from the hop they’ve ‘died’ on.
So from this, we should understand that just because a ‘hop’ (router) does not take the time to reply to a packet originator with an ICMP message doesn’t mean a network issue. Same for latency, an ICMP message is not as important as passing correctly crafted packets. The only time MTR means anything is when ‘negative’ metrics aggregates to the next-hop, and the next, all the way to the target.
With that, let’s discuss about a common occurrence…
Our app is running slow due to network loss!
Scenario:
A team comes to you saying their application is running slow and they’ve detected packet-loss. They have the data to prove it! The application is pulling data from 44.240.95.XX, and TCP/443 MTR shows packet-loss.
You’re the network guy on a multi-stakeholder conference call. It’s your problem now!
ubuntu@ip-172-31-17-88:~$ mtr -nc 5 44.240.95.XX -TP 443 --report
Start: 2022-11-25T06:31:23+0000
HOST: ip-172-31-17-88 Loss% Snt Last Avg Best Wrst StDev
1.|-- ??? 100.0 5 0.0 0.0 0.0 0.0 0.0
2.|-- 240.3.16.99 0.0% 5 0.4 0.3 0.3 0.4 0.0
3.|-- 240.0.52.84 0.0% 5 0.4 0.3 0.3 0.4 0.0
4.|-- 240.3.16.24 0.0% 5 0.3 0.3 0.3 0.3 0.0
5.|-- 240.0.32.15 0.0% 5 0.5 0.4 0.3 0.5 0.1
6.|-- 240.0.48.28 0.0% 5 0.3 0.4 0.3 0.5 0.1
7.|-- 240.0.48.22 0.0% 5 0.3 0.3 0.3 0.4 0.0
8.|-- 242.0.155.177 0.0% 5 0.6 0.5 0.4 0.6 0.1
9.|-- 52.93.28.83 0.0% 5 10.3 2.6 0.4 10.3 4.3
10.|-- 240.0.32.25 0.0% 5 0.4 1.7 0.4 4.3 1.5
11.|-- 100.92.31.99 0.0% 5 60.8 37.9 0.4 64.3 34.0
12.|-- 100.100.18.21 0.0% 5 62.8 50.1 0.6 63.2 27.7
13.|-- 100.100.8.20 0.0% 5 7.0 40.4 1.3 64.8 33.1
14.|-- 100.100.82.77 0.0% 5 64.2 62.9 60.6 64.3 1.8
15.|-- 100.100.2.97 0.0% 5 60.6 62.7 60.6 64.8 1.5
16.|-- 100.100.64.66 0.0% 5 59.5 64.5 59.5 77.1 7.2
17.|-- 100.100.93.13 0.0% 5 62.4 63.2 62.0 66.0 1.7
18.|-- 108.166.232.45 0.0% 5 61.5 61.8 61.1 63.3 0.9
19.|-- 100.100.6.18 0.0% 5 66.2 63.6 61.0 66.2 1.9
20.|-- 108.166.240.31 0.0% 5 60.7 63.4 60.7 66.6 2.2
21.|-- 108.166.232.44 0.0% 5 64.0 63.7 62.5 64.3 0.7
22.|-- 108.166.240.17 60.0% 5 61.6 63.1 61.6 64.6 2.1
23.|-- 108.166.240.17 80.0% 5 62.3 62.3 62.3 62.3 0.0
24.|-- 44.240.95.XX 20.0% 5 61.5 63.1 61.5 64.8 1.7Clearly there’s packet-loss incrementing on the link between hops 21 and 22, even 23, and finally, the target IP! Right?
NO
MTR is reporting packet-loss. However, the team did not verify packet-loss using other tools. So let’s try some of those tools to validate these results.
If all MTR is doing is sending TCP SYN packets on the specific port, we can do the same with Hping3. Lets’ try…
ubuntu@ip-172-31-17-88:~$ sudo hping3 -T -S -p 443 44.240.95.XX
HPING 44.240.95.XX (ens3 44.240.95.XX): S set, 40 headers + 0 data bytes
len=46 ip=44.240.95.XX ttl=222 DF id=0 sport=443 flags=SA seq=0 win=26883 rtt=67.9 ms
len=46 ip=44.240.95.XX ttl=224 DF id=0 sport=443 flags=SA seq=1 win=26883 rtt=67.8 ms
len=46 ip=44.240.95.XX ttl=221 DF id=0 sport=443 flags=SA seq=2 win=26883 rtt=67.8 ms
len=46 ip=44.240.95.XX ttl=222 DF id=0 sport=443 flags=SA seq=3 win=26883 rtt=63.8 ms
len=46 ip=44.240.95.XX ttl=222 DF id=0 sport=443 flags=SA seq=4 win=26883 rtt=63.7 ms
len=46 ip=44.240.95.XX ttl=224 DF id=0 sport=443 flags=SA seq=5 win=26883 rtt=63.7 ms
len=46 ip=44.240.95.XX ttl=224 DF id=0 sport=443 flags=SA seq=6 win=26883 rtt=63.6 ms
len=46 ip=44.240.95.XX ttl=222 DF id=0 sport=443 flags=SA seq=7 win=26883 rtt=63.6 ms
len=46 ip=44.240.95.XX ttl=222 DF id=0 sport=443 flags=SA seq=8 win=26883 rtt=63.5 ms
len=46 ip=44.240.95.XX ttl=222 DF id=0 sport=443 flags=SA seq=9 win=26883 rtt=71.5 ms
len=46 ip=44.240.95.XX ttl=221 DF id=0 sport=443 flags=SA seq=10 win=26883 rtt=63.4 ms
len=46 ip=44.240.95.XX ttl=221 DF id=0 sport=443 flags=SA seq=11 win=26883 rtt=63.4 ms
len=46 ip=44.240.95.XX ttl=224 DF id=0 sport=443 flags=SA seq=12 win=26883 rtt=71.3 ms
^C
--- 44.240.95.XX hping statistic ---
13 packets transmitted, 13 packets received, 0% packet loss
round-trip min/avg/max = 63.4/65.8/71.5 msWhat’s different? Why is MTR showing Packet-loss, but Hping3 isn’t? Bridges of trust are burning as our long relied on tools are providing conflicting results!
What is MTR really doing…
Any good network engineer using MTR should at some point perform a pcap of MTR to better understand what it’s doing. With that, they might notice what I mentioned above, it increments TTL until the target is reached. In the Hping output above, though, notice the TTL of the response packets. 224-221, 4 different path lengths. It appears MTRs logic check for variable length path has failed and it may not being sending a large enough TTL to test the path.
MTR does NOT account for Variable Length Paths in a way that’s always reliable.
ECMP allows equal cost paths over variable length hop paths. What results, is in some scenarios wildly different hop counts per-packet. While ECMP hashing algorithms are not being covered here, they can be based on src/dst IP; protocol; src/dst port; dependent on router config. This means that typically, ICMP MTR is not subject to this behavior since there’s no port numbers. However, TCP MTR is because each SYN request has a different ephemeral port.
If we were to take a look at a pcap from MTR in this case, we could find the largest TTL MTR is sending for this traffic scenario…
Alright, then if MTR experiences periodic packet-loss with a TTL of 25, hping should too, right?
ubuntu@ip-172-31-17-88:~$ sudo hping3 -T -S -p 443 -t 25 44.240.95.XX
HPING 44.240.95.XX (ens3 44.240.95.XX): S set, 40 headers + 0 data bytes
len=46 ip=44.240.95.XX ttl=222 DF id=0 sport=443 flags=SA seq=0 win=26883 rtt=63.9 ms
len=46 ip=44.240.95.XX ttl=221 DF id=0 sport=443 flags=SA seq=1 win=26883 rtt=63.8 ms
len=46 ip=44.240.95.XX ttl=222 DF id=0 sport=443 flags=SA seq=2 win=26883 rtt=63.8 ms
len=46 ip=44.240.95.XX ttl=222 DF id=0 sport=443 flags=SA seq=3 win=26883 rtt=63.7 ms
len=46 ip=44.240.95.XX ttl=222 DF id=0 sport=443 flags=SA seq=4 win=26883 rtt=63.7 ms
len=46 ip=44.240.95.XX ttl=221 DF id=0 sport=443 flags=SA seq=5 win=26883 rtt=63.7 ms
len=46 ip=44.240.95.XX ttl=224 DF id=0 sport=443 flags=SA seq=6 win=26883 rtt=63.6 ms
len=46 ip=44.240.95.XX ttl=224 DF id=0 sport=443 flags=SA seq=8 win=26883 rtt=63.5 ms
len=46 ip=44.240.95.XX ttl=222 DF id=0 sport=443 flags=SA seq=9 win=26883 rtt=63.5 ms
len=46 ip=44.240.95.XX ttl=222 DF id=0 sport=443 flags=SA seq=12 win=26883 rtt=63.4 ms
^C
--- 44.240.95.XX hping statistic ---
14 packets transmitted, 10 packets received, 29% packet loss
round-trip min/avg/max = 63.4/63.7/63.9 msGreat! Now we should be getting that confidence in our tools back because we’ve found cause, and were able to reproduce! Let’s try a TTL of 27, that’s plenty.
ubuntu@ip-172-31-17-88:~$ sudo hping3 -T -S -p 443 -t 27 44.240.95.XX
HPING 44.240.95.XX (ens3 44.240.95.XX): S set, 40 headers + 0 data bytes
len=46 ip=44.240.95.XX ttl=222 DF id=0 sport=443 flags=SA seq=0 win=26883 rtt=63.9 ms
len=46 ip=44.240.95.XX ttl=222 DF id=0 sport=443 flags=SA seq=1 win=26883 rtt=71.9 ms
len=46 ip=44.240.95.XX ttl=222 DF id=0 sport=443 flags=SA seq=5 win=26883 rtt=63.7 ms
len=46 ip=44.240.95.XX ttl=222 DF id=0 sport=443 flags=SA seq=6 win=26883 rtt=63.6 ms
len=46 ip=44.240.95.XX ttl=222 DF id=0 sport=443 flags=SA seq=7 win=26883 rtt=63.6 ms
^C
--- 44.240.95.XX hping statistic ---
8 packets transmitted, 5 packets received, 38% packet loss
round-trip min/avg/max = 63.6/65.3/71.9 msNo? is something wrong here? What about 28?
ubuntu@ip-172-31-17-88:~$ sudo hping3 -T -S -p 443 -t 28 44.240.95.XX
HPING 44.240.95.XX (ens3 44.240.95.XX): S set, 40 headers + 0 data bytes
...
.
...
--- 44.240.95.XX hping statistic ---
19 packets transmitted, 19 packets received, 0% packet loss
round-trip min/avg/max = 63.2/67.5/71.1 msSo MTR was 3 TTL hops away from providing accurate results. Is that MTR’s fault? No, not really. accounting for such a variable length path would require quite a bit of logic in the code-base, and delay meaningfully output. This is a known issue and can crop up in some situations. See:
https://github.com/traviscross/mtr/issues/361
Summary
Always validate findings. MTR is great, I’m not saying it’s not. But it’s optimized for finding issues in the path, which is added complexity. If you suspect end-to-end packet-loss, try the simplest tools first, as there’s less variables.