A network failure is a broken requirement somewhere inside a dependency chain
Topic 4.1 becomes much easier when “the network is down” is replaced by a precise statement: which users, locations, networks, destinations and services fail; what still works; when the failure occurs; and which shared dependency could explain that exact population. The objective is not to name a device quickly. It is to locate the earliest unresolved dependency and gather evidence that can separate competing explanations.
Use Scope → Known-Good Checkpoints → Shared Dependency → Failure Domain → Discriminating Evidence → Targeted Change → Requirement-Matched Verification. For intermittent incidents, insert a timeline before changing state.
Count affected endpoints, segments, destinations, protocols, locations, and time windows
| Observed scope | Dependencies that become more plausible | High-value control |
|---|---|---|
| One endpoint; peers work | Endpoint interface/configuration, local access path, security, application | Same workload from peer on same intended network |
| One VLAN/subnet | VLAN path, DHCP/relay, SVI/gateway, VLAN-specific policy | Another VLAN sharing physical/core infrastructure |
| One access switch/floor | Switch power, uplink, trunk, shared configuration | Neighbor switch using same upstream services |
| All site VLANs lose remote access; local works | WAN edge, default route, NAT/firewall, provider circuit | Local inter-VLAN and edge-interface state |
| One destination/service only | Destination DNS, specific route/filter, remote service, exact port | Other destinations and another service on same host |
| Only at peak periods | Capacity, queues, airtime, resource exhaustion, scheduled load | Same path/workload off-peak |
Swipe horizontally to see the full table.
Scope is not merely descriptive. It identifies what the failed population shares. If every device behind one access switch fails simultaneously while neighboring switches remain healthy, a public DNS outage is a poor first hypothesis because those neighbors use the same DNS service.
Known-good checkpoints are active evidence
A successful gateway test preserves the tested endpoint-to-gateway path. A successful remote numeric test preserves some routed IP connectivity without DNS. A successful DNS answer preserves name resolution for that query but does not prove TCP, TLS, or the application. One VLAN successfully crossing a trunk proves that the physical trunk can carry at least some traffic, but not that every VLAN is allowed and tagged correctly.
This rule prevents two opposite mistakes: restarting at dependencies already proven for the reproduced failure, and assuming one success proves the entire network.
A failure domain is defined by dependency, not by an arbitrary map boundary
Imagine twenty clients on one floor. They may share an access switch and uplink, but also share sitewide DHCP, DNS, firewall, and WAN services with other floors. If only that floor fails, the unique floor dependencies rise in priority. If every floor fails only for Internet traffic while internal services remain available, the shared external path becomes more important.
Unique dependency
A component/path used only by the failed population can explain selective failure.
Shared healthy dependency
If failed and healthy controls both use it successfully, it becomes less useful as the first fault.
Cascading dependency
One early failure can make many downstream services appear broken without each service failing independently.
Order symptoms by the earliest missing prerequisite
A wired client may require physical link, correct VLAN, IP configuration, gateway reachability, upstream routing, DNS, exact transport, and application service. A wireless client adds visibility, association, authentication/authorization and RF/airtime before the IP chain. If physical attachment is absent, missing DHCP and DNS are downstream consequences rather than separate root causes.
Conversely, if the client reaches a remote numeric IP but not a hostname, link, addressing, gateway and some routing already succeeded. The first unresolved dependency has moved forward to DNS or later application behavior.
Intermittent failures require state capture, not just repeated repair attempts
A persistent outage can be tested repeatedly. An intermittent outage may disappear after a port renegotiates, route converges, lease renews, AP roam completes, queue drains, or service recovers. Capture the state before rebooting, renewing, reseating, or moving the endpoint whenever practical.
Illustrative incident timeline 14:07:12 Users report 30-second outage on Floor 2 14:07:14 Access-SW2 uplink transitions DOWN 14:07:16 Neighbor floors remain reachable 14:07:41 Access-SW2 uplink transitions UP 14:07:43 Gateway reachability returns 14:07:46 User transactions recover Strong boundary: Floor 2 shared uplink path Still unresolved: optic, fiber, port, power, negotiation, or switch-side cause
Timestamps localize the event but do not automatically prove root cause. Interface counters, optic/media evidence, power/environment data, and logs must distinguish the remaining mechanisms.
Unreachable and slow require different measurements
A hard failure may appear as link down, absent route, failed gateway, dead service, or complete policy denial. Degradation can leave connectivity intact while latency, jitter, loss, retransmissions, retries, queue drops, or transaction time become unacceptable. A single successful ping is therefore weak evidence against a performance complaint.
| Complaint | Measure | Mechanism evidence | Comparison |
|---|---|---|---|
| High latency | Delay over time | Queue/utilization/path timing | Idle vs load |
| Low throughput | Transfer rate | Loss/retransmission, link rate, bottleneck utilization | Peer/path control |
| Jitter | Delay variation | Queue variability, wireless contention, path changes | Same flow/time window |
| Packet loss | Loss rate/location | Errors, drops, congestion, RF retries | Hop/interface correlation |
Layer 1 can remove every higher dependency at once
A failed access-switch power supply, disconnected fiber, damaged copper pair, failed optic, unstable PoE feed, or administratively disabled interface can eliminate downstream DHCP, gateway, DNS, and applications even though those services themselves are healthy. Match physical evidence to scope: a dead AP may affect one coverage area; a failed switch uplink may affect every VLAN behind that switch.
Use both ends of the link when possible. Interface state, negotiation, error-counter deltas, optical levels where available, and controlled cable/port substitutions provide more information than a single LED.
Different IP mistakes produce different boundaries
A 169.254 IPv4 address indicates failed normal DHCP acquisition, but not why it failed. An address from an unexpected subnet can reveal wrong VLAN or scope. A wrong subnet mask changes the host's local-versus-remote decision. A wrong default gateway can preserve same-subnet traffic while breaking remote access. Duplicate addressing can create alternating ARP ownership and intermittent connectivity.
Inspect the entire Layer 3 state—address, prefix/mask, gateway, DNS, lease source—and compare it with the intended design and a healthy peer. “The device has an IP” is not enough.
One infrastructure service can create selective downstream symptoms
DHCP
New clients can fail while clients with valid leases continue operating. Compare lease age, scope, relay, VLAN, and exchange evidence.
DNS
Remote numeric connectivity can work while hostnames fail. Compare configured resolver, response type, and returned record.
Authentication
Physical/IP paths can exist while access is denied because identity, certificate, role, or policy evaluation fails.
The key is to test the dependency directly. A browser error is not a DNS test, and a Wi-Fi association icon is not an authentication/authorization test.
Read the route selected for the failed destination—not merely whether a route exists
A missing default route can affect many unknown remote networks. A wrong more-specific route can affect one prefix while the default route works. A wrong next hop can blackhole selected destinations. A routing loop can consume TTL and repeat forwarding between devices. Return-path failure can break a transaction even when the forward route looks correct.
Use the exact destination to determine longest-prefix match, next hop, adjacency/forwarding progress, destination arrival, return route, and policy. Traceroute silence alone is not proof that the silent hop is the failure; diagnostic responses can be filtered while transit traffic is forwarded.
A blocked flow is a failure only when it violates the intended requirement
A firewall denying guest-to-management traffic can be correct enforcement. The same deny is a defect if the documented requirement explicitly permits that source, destination, protocol, service, zone, and state. Troubleshooting therefore requires both technical evidence and policy intent.
Inspect the exact flow and matching rule/log. Avoid broad “allow any” changes. After correction, test the approved flow and a prohibited control to prove that access was restored without weakening segmentation.
For exact-flow reasoning, use Firewall Rules Explained and Port Blocked Troubleshooting.
Recent change is a hypothesis generator, not proof
Ask what changed before the incident: VLANs, routes, firewall policy, DHCP scopes, DNS records, firmware, cabling, optics, authentication policy, provider circuits, server configuration, power work, or monitoring thresholds. Then ask whether the change can mechanistically explain the observed scope and timing.
A change on the site firewall is less convincing for a single access-port failure if every other user through that firewall remains healthy. A trunk allowed-list edit immediately before one VLAN disappears across that trunk fits both timing and scope much better. Compare current state with a known-good baseline and roll back only when the evidence justifies it.
A good control shares almost everything with the failing path except one meaningful variable
| Failure | Strong control | What the contrast can isolate |
|---|---|---|
| One endpoint fails | Healthy peer on same VLAN/access switch | Endpoint vs shared network |
| One VLAN fails Internet | Another VLAN through same firewall/WAN | VLAN-specific vs shared edge/provider |
| One hostname fails | Numeric destination + another hostname | Routing vs DNS/destination specificity |
| One TCP port fails | Known working service on same host | Host/path-wide vs exact service/policy |
| Wireless client slow | Wired/local-server or alternate-band test | Wireless-specific vs shared upstream |
Testing an unrelated application through a different network is a weak control because too many variables change simultaneously.
“It worked after reboot” is restoration evidence, not a root-cause statement
A switch reboot may restore an unstable uplink but erase volatile evidence. A DNS workaround may restore public names while bypassing an intended internal resolver and breaking private zones. A firewall broad allow may make the application work while violating security policy. Record what action restored service separately from what evidence establishes the cause.
If root cause remains unresolved, document that explicitly. “Service restored after interface reset; cause not yet established; pre-reset counters unavailable” is more useful than inventing a hardware failure.
One floor drops for 30 seconds while every other floor stays online
Users on Floor 2 report simultaneous outages. The site's DNS, firewall, WAN, and core continue serving other floors. Monitoring shows the Floor 2 access-switch uplink transitions down and up at matching timestamps. That combination localizes the failure domain to the floor's shared uplink path.
The next evidence should distinguish optic/media, port, power, negotiation, or switch behavior. Replacing DNS or changing site routing would ignore the healthy controls.
Every VLAN reaches internal services but none reaches the Internet
Multiple VLANs retain gateway and internal routed connectivity. Remote numeric destinations fail from all of them. The shared edge is up, but route evidence shows the expected default route missing after a maintenance change. Restoring the approved default route returns remote connectivity.
Internal success preserves access switching, local addressing, gateways, and internal routing. The cross-VLAN external scope points to a dependency shared only by remote traffic.
New clients fail while existing clients keep working
New endpoints in one subnet receive 169.254 addresses, while endpoints with existing DHCP leases continue reaching the gateway and Internet. The DHCP scope has no available addresses. After the approved scope correction, a new endpoint receives the intended address, gateway, and DNS settings and completes the required service.
The contrast between new and existing clients is the high-information clue. APIPA alone would not distinguish scope exhaustion from wrong VLAN, relay failure, filtering, or server outage.
One SaaS destination fails while the rest of the Internet works
Users resolve and reach many remote services, but one required SaaS hostname resolves to an address whose specific prefix follows a stale more-specific route. The default route is healthy for other destinations. Removing/correcting the stale approved route restores only the affected path.
“Internet outage” would be the wrong scope. Destination specificity plus longest-prefix evidence explains why most remote services remain healthy.
A denied firewall flow is actually correct behavior
A guest client cannot reach a management interface. Firewall logs show the exact source, destination, protocol, and port matching the documented guest-to-management deny. Other permitted guest Internet traffic works. No repair is justified because the observed behavior matches the requirement.
This is an important Topic 4.1 distinction: a user complaint can describe intentional policy enforcement rather than a network failure.
Escalate with the last verified checkpoint and the first dependency you cannot verify
Before escalating to an ISP, collect WAN interface state, local edge routing, affected scope, timestamps, provider-facing errors, and useful path evidence. Before escalating to a server team, show destination resolution, route/transport evidence, policy decision, and exact application/port behavior. Before escalating to wireless engineering, identify BSSID/AP, association/authentication state, RF evidence, VLAN/DHCP state, and a shared-upstream control.
A strong escalation says, for example: “All site VLANs reach internal resources and the firewall inside interface; the provider-facing interface is up, but the upstream next hop is unreachable from the edge beginning at 14:07. Neighboring internal paths remain healthy.” That is actionable evidence, not “Internet broken.”
Use before-and-after evidence that matches the original requirement
| Changed area | Positive verification | Guardrail/control |
|---|---|---|
| Physical/uplink | Link remains stable under reproduced traffic | Error counters do not reproduce abnormal growth |
| DHCP | Fresh endpoint receives complete intended lease | Other scopes/clients remain healthy |
| Route | Affected prefix follows intended path and succeeds | Unrelated prefixes retain intended routes |
| Firewall | Required exact flow succeeds | Prohibited comparison flow remains denied |
| Performance | Original metric meets requirement under comparable load | Use the same workload/time conditions |
| DNS | Original name resolves correctly and application works | Private/other required names remain correct |
Separate observations, hypotheses, actions, and conclusions
- Record affected scope, exact requirement, start/end time, and recurrence pattern.
- List successful and failed checkpoints before configuration changes.
- Preserve relevant logs, counters, routes, leases, policy decisions, and topology context.
- Document recent changes as hypotheses, not automatic causes.
- State the root cause only to the strength supported by evidence.
- Record corrective action separately from root-cause evidence.
- Include positive verification and controls/guardrails.
- If unresolved, identify the first dependency still unverified and the next evidence needed.
Turn a broad failure into a bounded technical explanation
Observation: “Every endpoint behind Access-SW2 loses connectivity for about 30 seconds, while endpoints behind other access switches remain healthy.” Bounded inference: “The failed population shares Access-SW2 and its uplink; sitewide DNS/WAN services remain usable from controls.” Discriminating evidence: “Access-SW2's uplink down/up events align with the outage timestamps.” Mechanism boundary: “The shared uplink path explains the scope, but the specific optic/media/port/power cause is not yet established.” Next action: “Inspect both-end interface, optical/media, error, power, and event evidence.” Verification: “After the justified correction, reproduce normal load and confirm sustained link plus healthy counters while neighboring paths remain unchanged.”
24 Topic 4.1 network-failure flashcards
Practice failure scope, dependency chains, control tests, intermittent evidence, routing, services, security policy, root cause, escalation, and verification.