kappy.prasad
← Articles

Network automation on a global backbone

Python tooling that logged into core routers, ran the standard diagnostics and returned a clean summary, taking the repetitive work out of 24×7 operations on a global MPLS/IP backbone.

Use casesUse case 6 of 72 min read

PythonJuniperCiscoBGPMPLSTACACS+Linux

Monitoring a global backbone around the clock teaches you something quickly: the first ten minutes of every incident look the same. An alert fires. An engineer logs into the affected router, then its neighbors, and runs the same handful of commands: interface state, optical levels, BGP session status, recent log entries. They copy the output into a ticket, then start actually thinking.

Those first ten minutes were the most time-critical part of the incident, and they were spent on manual, repetitive typing.

The environment

The network was a large MPLS/IP core built on Juniper routers, with a Juniper and Cisco edge, connected by long-haul transport circuits running from OC-48 up to OC-192. Day-to-day work meant:

  • Watching BGP peering with carriers and partners, and investigating traffic shifts.
  • Troubleshooting transport circuits with carriers and field technicians.
  • Scheduling and running maintenance, both vendor-requested and break/fix, with minimal service impact.
  • Supporting data center deployments as new capacity came online.

All of it ran 24×7 with shift handovers, so consistency between engineers really mattered.

Scripting the first ten minutes

The fix was a Python tool that did the repetitive part. Given a device, or a device and its neighbors, it would:

  1. Authenticate through TACACS+ with the engineer’s own credentials, so there was no shared service account and every login stayed attributable.
  2. Run a standard set of read-only diagnostic commands.
  3. Parse the output and pull out the things that matter.
  4. Return a short summary an engineer could read in seconds and paste into a ticket.

Design choices that mattered

  • Read-only by design. The tool only ever ran show commands. Diagnosis was automated; changes stayed with a human.
  • Individual credentials. Every session was tied to the engineer running it, so the network’s existing access controls and audit trail still applied.
  • Output for people, not just for logs. The summary was the product. Raw output was kept for anyone who wanted to dig deeper, but the first thing you saw was what was wrong.
  • Consistent across shifts. Every engineer ran the same checks the same way, so handovers carried comparable information.

Beyond the tool

Automation was one part of the job. The other was people. Training new NOC engineers on the network, the tooling and the troubleshooting approach meant the knowledge didn’t live in a few senior heads. Tools like this one helped, because the checks themselves documented what an experienced engineer would look at first.

What I’d tell another team

  • Automate the first ten minutes. That’s where the repetitive, time-critical work is.
  • Start read-only. Let automation earn trust by gathering information before it’s allowed to change anything.
  • Keep identity attached, with no shared automation accounts on critical infrastructure.
  • Design the output for a person under pressure.
Outcome

Faster and more consistent diagnosis during incidents, and a team that could share and scale its knowledge.