We are testing getting a BR running for MAP-T, but are seeing unexpected traffic and lack of the MAP domain processing traffic (inbound and outbound). These two symptoms seem related (perhaps the VPP graphs aren't fully loaded?).
I'm looking for help with determining how to better troubleshoot MAP on TNSR and see where traffic is dropping. It seems to be reaching the proper interface from both the v4 and v6 sides.
Also, what needs to be done to get MAP to properly pick up and translate traffic? It seems broken ...
Troubleshooting High Level Checks done Routing Checked to/from the v4 and v6 sides, respectively for MAP v4 and v6 Prefixes - it's working this is verified with tcpdump v4 inbound flows, and v6 reachability (i.e. ping) to our DMR (Default Mapping Rule aka BR IPv6 interface address), and show route table default ADDR checks for routes on upstream / downstream routers) toggling MAP enable / disable on the interface in case MAP didn't apply properly verified proper bits for the MAP-T config (e.g. EA bits, PSID offset, PSID length, Prefix, etc) Related: also verified some of the addresses seen on the wire with https://github.com/ejordangottlieb/pyswmap code and manual calculationsWith that said ... is there better debugging logs or counters for errors or "non-security checked" traffic hitting the MAP flows in VPP? Troubleshooting seems extremely limited from what the docs say, especially if v6 flows aren't being shown in tcpdump (which seems to be the case, see below troubleshooting)
NDPs showing up for IPv6 Destination addresses from CPEWhat's interesting is we see NDPs going OUT from the BR DMR.
This is unexpected.
These addresses are inbound from the CPE and should be ingested by MAP, not sent anywhere else. We verified the SRC/DST v6 addresses on the WAN side via wireshark as an additional sanity check to ensure they are properly formatted for the given IPv6 Prefix, Destination DMR, Destination v4 host, PSID, etc.
An example tcpdump snippet on the BR interface showing unexpected NDPs and no other ICMPv6, ICMP, HTTP traffic (test traffic from the CPEs bound for the v4 Internet was ICMP and/or HTTP):
02:23:51.094333 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::20c:29ff:feca:40bb > ff02::1:ff00:0: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8:64:0:1:101:100:0 source link-address option (1), length 8 (1): 00:0c:29:ca:40:bb 0x0000: 000c 29ca 40bb 02:23:52.098581 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::20c:29ff:feca:40bb > ff02::1:ff00:0: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8:64:0:1:101:100:0 source link-address option (1), length 8 (1): 00:0c:29:ca:40:bb 0x0000: 000c 29ca 40bb 02:23:52.975038 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::20c:29ff:feca:40bb > ff02::1:ff00:0: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8:64:0:68:1223:f00:0 source link-address option (1), length 8 (1): 00:0c:29:ca:40:bb 0x0000: 000c 29ca 40bb 02:23:53.104128 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::20c:29ff:feca:40bb > ff02::1:ff00:0: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8:64:0:1:101:100:0 source link-address option (1), length 8 (1): 00:0c:29:ca:40:bb 0x0000: 000c 29ca 40bb 02:23:53.977166 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::20c:29ff:feca:40bb > ff02::1:ff00:0: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8:64:0:68:1223:f00:0 source link-address option (1), length 8 (1): 00:0c:29:ca:40:bbSpecifically, what's interesting:
tcpdumps don't show the incoming packets destined to the DMR interface from CEs this could be ok depending on how vpp input graphs are laid out, but we would need confirmation from devs To the above point, DMR-destined traffic from CEs is getting to the BR somehow and being processed. Those NDP v6 addresses seen above in the tcpdump (2001:db8:64:0:1:101:100:0, 2001:db8:64:0:68:1223:f00:0) are the proper translated destination packets for cloudflare DNS (1.1.1.1) and a cloudflare web site (104.18.35.15)It almost seems that the v6 inbound packets are getting processed for routing on the local subnet (thus the NDP), but we would expect MAP to pick this up, translate it to v4, and dump it out toward the next-hop with the proper v4 public address (A.B.C.128 in this case since PSIDs are 0 and 1 with a sharing ratio of 8 on this MAP domain)
Additional TroubleshootingTo further investigate, we dumped the VPP map stats (in vppctrl) and they are:
Very one-sided (Encapsulated packets > 0, Decapsulated = 0) But also imbalanced. Organic Internet traffic toward v4 addresses in the MAP domain + other scapy crafted packets destined for our CPE number in the hundreds and thousands of packets per second, but watching the show map stats output, we'll see 10-20 pps on the Encapsulated packets counter. It's inconclusive if these are the traffic flows from the Internet, but we strongly suspect no due to the counts being off. tcpdump shows inbound flows (v4 internet to v6 CPE) which should get translated along with our outbound flows (v6 translated CPE traffic to v4 Internet), so we'd expect processing counts for bothThis implies MAP is not being applied / activated properly. We even toggled MAP on the interface (map disable; exit followed by map enable; exit) to "repush" config, but that doesn't change any of the above observed behavior; traffic still seems broken through MAP.
Are there other counters / logs / taps that can be created to specifically troubleshoot MAP flows?
Reference Output:
vpp# show map stats MAP domains structure: 64 MAP domains: 1 (64 bytes) MAP rules: 0 (0 bytes) Total: 64 bytes) MAP pre-resolve: IP6 next-hop: un-set, IP4 next-hop: un-set MAP traffic-class: copy MAP IPv6 inbound security check: enabled, fragmented packet security check: disabled ICMP-relay IPv4 source address: A.B.C.D ICMP6 unreachables sent for unmatched packets: enabled Inner fragmentation: disabled Fragment packets regardless of DF flag: disabled Encapsulated packets: 4035 bytes: 180880 Decapsulated packets: 0 bytes: 0 ICMP relayed packets: 0 Relevant TNSR MAP config nat nat64 map parameters icmp source-address A.B.C.D icmp6 unreachable-msgs enable security-check enable exit nat nat64 map br ipv4 prefix A.B.C.128/25 ipv6 prefix 2001:db8:ce1::/48 ipv6 source 2001:db8:64::/64 embedded-address bit-length 10 port-set offset 3 port-set length 3 mtu 1500 exit interface br enable ip address 172.31.252.29/31 ipv6 address 2001:db8:64::1/64 ipv6 router-advertisements send-advertisements false max-rtr-adv-interval 600 managed-flag false other-config-flag false exit map enable map translate exit