🤖 Network SRE & Observability Learning Path¶

🚀 Production Reliability & Automated Operations: Master production incident mitigation, sub-second BFD failover, gNMI real-time streaming telemetry, Prometheus anomaly detection, and automated PyATS health gate checks on real Arista cEOS fabrics.

📊 Learning Path Overview¶

Metric	Target Specification
Estimated Completion Time	30 – 35 Hours (Hands-on lab driven, scenario-based)
Milestone Stages	5 Progressive Stages (Convergence → Automated Assertions → Telemetry → Control Plane Hardening → Incident Drills)
Lab Framework	Containerlab + Arista cEOS (Runs 100% locally on macOS OrbStack or Linux Docker)
Target Roles	Network SRE, Production Infrastructure Engineer, Network Observability Lead, Cloud Network SRE
Target Employers	Google, Meta, Apple, AWS, Microsoft, ByteDance, Netflix, Stripe, and High-Scale SaaS Platforms

🧠 Core Network SRE Engineering Pillars¶

SRE Pillar	Key Production Technologies	Engineering Objective
Sub-Second Failover	BFD (Bidirectional Forwarding Detection)	Detect optical/physical link drop in under 300ms without relying on slow IGP keepalives
Push-Based Observability	gNMI / OpenConfig / Prometheus / Grafana	Stream high-cardinality time-series metrics over gRPC HTTP/2 instead of legacy 5-min SNMP
Automated Testing Gates	Cisco PyATS / Genie / Robot Framework	Eliminate human error by running automated pre-change and post-change state assertions
Control Plane Resilience	CoPP (Control Plane Policing) & Rate Limits	Protect switch Supervisor engine / CPU from route table exhaustion and broadcast storms
Error Budgets & SLOs	SLI/SLO Management & Blast Radius	Define quantifiable availability targets (99.99%) and isolate failure domains

🗺️ 5-Stage Progressive Milestone Roadmap¶

01

Stage 1 · Fast Convergence & Link Resilience

Sub-Second Failover

Achieve sub-second link fault detection with hardware-offloaded BFD and suppress routing churn with BGP flap dampening.

BFD Hardware Offload BGP Flap Dampening Graceful Restart RFC 4724

02

Stage 2 · Automated Testing & Verification Gates

Codified Assertions

Replace human CLI error with programmatic pre- and post-maintenance test suites using Cisco PyATS and Genie structured parsers.

PyATS Testbeds Genie JSON Parsers Operational State Diffs

03

Stage 3 · Real-Time Telemetry & Observability

Streaming Metrics

Eliminate monitoring blind spots by deploying gNMI streaming telemetry, OpenConfig YANG models, and Prometheus scraping.

gNMI gRPC Protobuf OpenConfig YANG Prometheus Grafana

04

Stage 4 · Control Plane Defense & Blast Radius Isolation

Switch Hardening

Shield switch Supervisor engines from denial of service attacks using Control Plane Policing (CoPP) and VRF microsegmentation.

CoPP CPU Protection MQC Rate Limiting VRF Route Leaking iACLs

05

Stage 5 · Chaos Engineering & Incident Drills

Production Drills

Simulate live link drop storms, diagnose routing oscillations under pressure, and practice blameless post-mortem triage.

Link Flap Injection BGP Churn Triage MTTR Optimization

🚀 Interactive Lesson Directory (Click Any Lesson to Start)¶

Milestone Stage	SRE Pillar Focus	Clickable Lessons & Hands-on Labs	Runnable Lab	Action
Stage 1 `Fast Convergence`	BFD Sub-Second Failover, BGP Flap Dampening, Graceful Restart	• Phase 2 · Lab 03: Sub-Second BFD Peering • Phase 6 · Lab 03: WAN Edge BFD Failover	`labs/wan-edge-lab`	Start Stage 1 →
Stage 2 `Automated Gates`	PyATS Testbeds, Genie Parsers, Pre/Post-Maintenance State Diffing	• Phase 5 · Lab 02: PyATS State Verification • Phase 5 · Lab 05: CI/CD Pipeline Automation	`labs/netdevops-lab`	Start Stage 2 →
Stage 3 `Observability`	gNMI Streaming Protobuf, OpenConfig YANG, Prometheus, Grafana	• Phase 7 · Lab 01: gNMI & OpenConfig YANG • Phase 7 · Lab 02: pygnmi Python Streams • Phase 7 · Lab 03: Prometheus Metric Collectors • Phase 7 · Lab 04: Real-Time Grafana Dashboards	`labs/telemetry-lab`	Start Stage 3 →
Stage 4 `Control Plane Defense`	CoPP CPU Protection, MQC Rate-Limiting, VRF Microsegmentation	• Phase 8 · Lab 01: CoPP CPU Protection • Phase 8 · Lab 02: VRF Microsegmentation • Phase 8 · Lab 03: Infrastructure ACLs (iACL)	`labs/security-lab`	Start Stage 4 →
Stage 5 `Incident Drills`	Link Flaps, BGP Churn, MTTR Reduction & Blameless Post-Mortems	• Phase 1 · Lab 04: Multihomed Edge Failure Triage • System Design Incident Response Drills	`labs/bgp-lab`	Start Stage 5 →

🧪 Detailed Milestone Curricula¶

📍 Stage 1: Fast Convergence & Link Resilience¶

Core Focus: Eliminating downtime caused by silent link degradation or slow routing protocol keepalives.
Key Concepts: BFD microsecond timers, hardware offload to ASICs, BGP route flap dampening, and graceful restart (RFC 4724).
Interactive Labs:
- Phase 2: BGP Dual-Homed Internet Access
- Phase 6: Enterprise WAN Edge & BFD
Local Runner:
```
cd labs/wan-edge-lab
./run.sh --guided
```

📍 Stage 2: Automated Testing & Verification Gates¶

Core Focus: Replacing manual CLI show commands with codified assertion suites that gate automated deployments.
Key Concepts: PyATS testbeds, Genie parsers, comparing pre-change vs. post-change state diffs, and routing table integrity checks.
Interactive Labs:
- Phase 5: Network Automation & CI/CD
- Lab 02: PyATS State Verification

Local Runner:

cd labs/netdevops-lab
./run.sh --guided

📍 Stage 3: Real-Time Telemetry & Observability¶

Core Focus: Transitioning from passive, high-overhead SNMP polling to sub-second gRPC push streams.
Key Concepts: gNMI Subscribe RPC (STREAM, SAMPLE, ON_CHANGE), OpenConfig interface and BGP schemas, Prometheus scraping, and Grafana dashboard visualization.
Interactive Labs:

Local Runner:

cd labs/telemetry-lab
./run.sh --guided

📍 Stage 4: Control Plane Defense & Blast Radius Isolation¶

Core Focus: Ensuring that data plane traffic anomalies or malicious traffic cannot crash the device control plane.
Key Concepts: CoPP MQC policies (classifying BGP, OSPF, SSH, ICMP), policing bandwidth limits, and hardware TCAM allocation.
Interactive Labs:
Local Runner:
```
cd labs/security-lab
./run.sh --guided
```

📍 Stage 5: Chaos Engineering & Incident Drills¶

Core Focus: Simulating real production outages to test monitoring alerts, automated failovers, and incident response playbooks.
Topics: Injecting link degradation, simulating leaf switch reboots, analyzing BGP convergence delays, and writing blameless post-mortems.

🛠️ Executable Local Lab Environment¶

NetForge Labs uses automated step runners to let you immediately spin up and verify topologies locally.

# 1. Navigate to the Telemetry & Observability lab
cd labs/telemetry-lab

# 2. Launch the guided interactive runner
./run.sh --guided

# Or deploy the complete verified topology in one command
./run.sh --all

🎓 Career Defense: Portfolio Projects¶

Sub-Second BFD Failover Fabric:
Defend your design of BFD timers vs. CPU utilization trade-offs across 1,000+ peers.
End-to-End gNMI Observability Pipeline:
Demonstrate how you captured microsecond queue buffer congestion via gNMI that went completely undetected by SNMP.
Automated CI/CD Maintenance Gate:
Showcase a PyATS testbed that automatically aborts a router OS upgrade if packet drop occurs during traffic shift.