Day 43

Connectivity Troubleshooting Toolkit

connectivity-troubleshooting-toolkit

At a glance

Troubleshoot L2/L3 connectivity by reading show output (interfaces, VLANs, trunks, MAC table, routes, logs), then proving one flow with ping, extended ping, traceroute, and a packet capture. Extended ping sets the source, size, repeat count, and DF bit. Traceroute stars and capture patterns (ARP, DHCP, DNS, TCP, ICMP unreachables) tell you where to look next. Maps to CCNA v2.0 objective 2.4.

Deep chapter

Long-form companion reading (original prose).

Objective 2.4 is a reading exam

CCNA v2.0 objective 2.4 asks you to troubleshoot basic Layer 2 and Layer 3 connectivity and device operations using show commands, including logs, plus ping, extended ping, traceroute, and packet capture output. The skill is interpretation. A stem pastes half a screen and asks why a host cannot reach another host, which command you would issue next, or which line in the capture explains the symptom. You are not decorating a topology. You are defending a conclusion from evidence.

Exam trap: “the route exists” is not “the user can connect.” Say which source address you tested, which interface the route uses, and which return path that source would need.

A method you can repeat under a timer

Start with the sentence. Who cannot reach whom, since when, and what changed. Then walk the path from the failing source. Confirm the access port: `show interfaces status`, error counters, and the last log lines for that interface. Confirm the VLAN and the MAC: `show vlan brief`, `show interfaces switchport`, `show mac address-table`. Confirm the gateway answers ARP and ICMP from that subnet. Confirm the route on each L3 hop with `show ip route` to the destination, then the return route toward the source. Only then open ACLs, NAT, or the far server.

Logs are part of the method, not a last resort. `show logging` with timestamps lines the flap, the err-disable, the duplex mismatch CDP complaint, and the ACL deny up with the minute the user called. If your session is remote, `terminal monitor` shows new lines as they happen. `logging buffered` and `service timestamps log datetime msec` are what make that buffer usable. A log without a clock is a story without an order.

Interface, VLAN, trunk, and MAC

`show ip interface brief` is the router’s front door. Address, method, status, and protocol. Up/up with the address you expect is the starting assumption. Up/down is Layer 2 down: encapsulation, clocking on a WAN, or a switch port that is not forwarding. Administratively down is `shutdown`, which no amount of routing will fix. `show interfaces` adds the negotiated speed and duplex, the input and output rates, and the counters that name a bad cable: input errors, CRC, runts, giants, collisions, late collisions. Late collisions on a port that claims full duplex are a duplex mismatch until proven otherwise.

On a switch, `show interfaces status` is the faster table: connected or not, VLAN, duplex, speed, and media. `show vlan brief` lists access ports in each VLAN and will not list a trunk as a member of every carried VLAN. Trunks need `show interfaces trunk` for mode, native VLAN, allowed list, and the VLANs in forwarding. A VLAN missing from the allowed list, or pruned, or not created on this switch, drops that VLAN even when the trunk is up/up. Native VLAN mismatch does not always black-hole traffic; it mis-delivers untagged frames and can produce the kind of “some protocols work” complaint that wastes an hour if you never compare both ends.

`show mac address-table dynamic address …` answers “which port learned this host?” If the MAC is missing, the host is not sending, you are on the wrong switch, or the VLAN filter in the command does not match. If the MAC is on the uplink instead of the access port, you are looking at a different switch’s table or the host is attached somewhere else. Port-channels show as Po interfaces; a MAC on Po1 is learned via the bundle, and `show etherchannel summary` tells you whether LACP is bundled or stuck in individual. Spanning tree belongs in the same glance when a port is connected and forwarding no frames: `show spanning-tree vlan` will show blocking, or BPDU guard err-disable will already be in the log.

Exam trap: `show vlan brief` is the wrong command to prove a trunk carries VLAN 40. Use `show interfaces trunk` and read the active list, not only the allowed list. Allowed and not forwarding is still an outage. A connected access port in VLAN 1, when the gateway SVI is VLAN 20, produces APIPA or a dead static address with a perfectly green link light.

Routes, neighbors, ARP, and the ACL on the interface

`show ip route 203.0.113.10` is better than dumping the whole table and skimming. The output names the prefix, the mask, the next hop or the connected interface, the administrative distance, and the metric. A connected route beats a static, which beats OSPF, when the lengths are equal; the longest match wins before any of that. If the matched route is a default and you expected a specific prefix, you have a missing route, not a “slow network.” `show ipv6 route` is the same habit for an IPv6 stem.

Connected does not mean reachable at Layer 2. `show ip arp` on the router should hold the host or the next hop. An incomplete ARP entry means the router is asking and nobody is answering in that VLAN: wrong VLAN, host down, or the SVI is down because no port is forwarding in the VLAN. `show ip ospf neighbor` belongs here when the expected route is OSPF and the prefix is simply absent. Neighbors down, neighbors stuck in ExStart, or neighbors up with an empty LSDB for that area are three different fixes. `show ip protocols` confirms the networks or interfaces you meant to advertise.

ACLs hide in `show ip interface` under “inbound access list” and “outbound access list,” and in `show access-lists`, where match counters increment. A deny on the inbound LAN ACL drops the user before the routing decision you spent ten minutes admiring. NAT hides in `show ip nat translations`. No translation for an inside host that should be overloaded is a NAT or ACL-match problem, and it looks like a routing problem from the host because the ping simply times out. Check the translation table while the ping is running, not a minute after you gave up.

Exam trap: the router can ping a destination using a source address the ACL permits, while the user’s source is denied. That is why extended ping exists. Read the next section before you close an ACL ticket with “but the router can ping it.”

Ping, and the codes on the right

A plain `ping 192.0.2.1` from IOS sends five ICMP echoes. The source address is the address of the interface the routing table uses to leave toward that target. Success prints exclamation marks. A dot is a timeout. `U` is an unreachable message came back. `M` means the packet could not be forwarded without fragmentation. Those codes are the exhibit. Five dots tell you nothing about where the packet died. A `U` from the second hop is a different incident from a timeout at the far firewall, which often drops ICMP and looks like dots.

Exam trap: ping from the wrong device is a common false pass. If you are logged into the core and you ping the server, you have tested the core’s source, not the branch VLAN. Either ping from the branch router with an extended source, or ping from the host. Write the source down next to the result.

Extended ping: source, size, DF, repeat

Extended ping is an ordinary ping whose fields you set. On IOS you type `ping`, press return, and answer the prompts, or you use the one-line form when you already know the flags. The fields that solve tickets are the target, the repeat count, the datagram size, the source address or interface, the DF bit, and the sweep of sizes.

Source is the one you will use most. `Source address or interface: Vlan20` or the SVI address sends the echo from the user’s gateway address instead of from the WAN interface the default route would have chosen. If the ping with source Vlan20 fails and a ping sourced from the WAN succeeds, the return route or the ACL depends on which address you used. That single comparison localizes asymmetric routing and “we only permitted the router’s outside address.” Repeat count of 100 is how you see loss that five packets missed. A timeout longer than two seconds belongs on a high-delay path so you do not call a slow link dead.

Size and DF find MTU problems. A 100-byte ping with DF clear may succeed across a tunnel, a PPPoE circuit, or an IPsec overlay that cannot carry a 1500-byte packet. The same destination with size 1500 and DF set returns `M` when some router on the path would have to fragment and is not allowed to. Sweeping sizes shows the largest datagram that still fits. The fix is then the MTU on the tunnel, a TCP MSS clamp, or a misconfigured interface MTU, which you confirm with show commands. Guessing “the ISP is down” because a large copy failed is how MTU bugs survive for months.

Traceroute and the hop that misleads

Traceroute sends probes with increasing TTL. Each router that decrements the TTL to zero should answer with ICMP Time Exceeded, and the last probe should get a response from the destination. IOS traceroute uses UDP to high ports by default. Windows `tracert` uses ICMP echo. Linux `traceroute` is usually UDP. A firewall that permits ICMP and drops UDP will look different from a PC than from a Cisco router. If the stem’s PC traceroute is full of stars and the router’s ping works, check what the probe was before you declare the path down.

Read the last hop that answered, then log into that device. `show ip route` for the original destination tells you where it wanted to send the packet. `show ip interface` on that egress interface tells you whether an ACL or a down protocol stopped it. Three stars at hop 4 mean hop 4 did not answer the probe. Hop 4 might still be forwarding user traffic and filtering TTL-exceeded or UDP probes. A later hop that answers proves the stars were a filtered control-plane reply, not a black hole. Stars through the end, with a failing user ping, are consistent with a real drop.

Exam trap: the last responding hop is the last device that talked, which may be the failure or may be the last device willing to answer probes. Correlate with ping from the right source and with the routing table at that hop before you replace a circuit.

Packet captures you can actually read

You are not expected to be a full-time analyst. You are expected to recognize a handful of patterns and tie them to a cause. An ARP request for the gateway, repeated, with no ARP reply: the host is on the wire and the gateway is not answering in this VLAN. No ARP request at all, and the destination is remote: the host never tried to reach a gateway, which points at a missing default route or a mask that makes the destination look on-link. An ARP reply from a MAC that is not the router’s MAC matches the ARP spoofing story from the security day.

DHCP Discover with no Offer means the server or the relay never answered. Discover, Offer, Request, NAK means a server answered and then refused the request: conflict, wrong scope, or a second server. DNS query with no response means the resolver did not answer (reachability or UDP 53). A DNS response with NXDOMAIN means the resolver answered and the name is not there. A DNS response with an A record you do not recognize is a record problem or a spoofed response, and you compare it to an authoritative query. Those three DNS shapes are different tickets.

TCP SYN, retransmitted, no SYN-ACK: the handshake never completed. The server, the port, or a filter in the path is dropping or the return path is gone. TCP RST means something was reachable and refused the port or reset the session. ICMP unreachable type 3 code 13, communication administratively prohibited, is an ACL or a firewall telling you the truth. ICMP type 3 code 4, fragmentation needed, with the DF bit in the original packet, is the capture version of the extended-ping `M`. A capture full of the same broadcast at a ridiculous rate is a storm, and you leave the capture to look at storm control and at spanning tree.

Exam trap: a capture that shows only the request is incomplete evidence of “the server is down.” Check the capture point. If you spanned the client port, you have not proved the server failed to answer; you have proved the client did not see an answer. Look one hop closer to the server before you reboot it.

Self-check

The self-check is short. Which command proves a trunk is forwarding VLAN 40, as opposed to merely allowing it in the configuration? What does extended ping add that plain ping from the same router hides? What does an `M` in an extended ping with DF set tell you to inspect? What does a capture of repeated DHCP Discover and zero Offers tell you, and what does a Discover/Offer/NAK sequence tell you instead? Why might traceroute show stars at hop 3 while a sourced ping through that hop still succeeds? If you can answer those five without looking down, you can sit objective 2.4.

Key ideas

  • Write source, destination, and protocol. Show commands say what the device believes; ping and traceroute test one source; a capture shows what was on the wire.
  • show interfaces status, show vlan brief, and show interfaces trunk answer different questions. Trunk forwarding is the active VLAN list, not the access-port table.
  • Plain ping uses the egress interface as the source. Extended ping sets source, repeat, size, and DF. An M result with DF set points at path MTU.
  • IOS traceroute is UDP by default and Windows tracert is ICMP. Stars can mean a filtered probe rather than a black hole. A single slow hop is often ICMP rate limiting.
  • Repeated ARP with no reply, DHCP Discover with no Offer, NXDOMAIN, TCP SYN with no SYN-ACK, and ICMP type 3 code 13 are different faults.
  • If the router can ping it and the user cannot, test again with an extended source in the user VLAN before you close the ticket. Logs and counters put the failure on a clock.

Self-check

  • Which show command proves VLAN 40 is forwarding on a trunk, and why is show vlan brief the wrong one?
  • What does extended ping add, and what does an M in the output tell you when DF is set?
  • Why can traceroute print stars at hop 3 while a sourced ping through that hop still succeeds?
  • A capture shows DHCP Discover with no Offer. How does that differ from Discover, Offer, Request, NAK?
  • The RIB has a route and a capture on the egress interface shows nothing. What do you check before you redesign OSPF?