Tag Archives: pyez

Juniper Config Backup on GITLAB

Automation, JNCIP, JNCIS, Juniper, Junos, Networking, SDN, , , , ,

Hi All

In this blog, we will see how we can take the backup of router configs and push it to gitlab for version control.

We will take Juniper routers here for example and build the script in Python and run our task as a cronjob on CentOS 7 to automatically backup the configs onto GITLAB.
We will use power of Git here to store the differentials in configs and we can see the differences in Gitlab UI.

So let’s see how we can achieve it.
Before starting with config backup, let’s build the device store on which we will be running the script or in other words backup will be running. For this we are storing the values in YAML file http://yaml.org/ as dictionary. You can either store the values as list or Dictionary in YAML. You can use json also to store the same data instead of YAML.

Yaml file (.yml)
---
# Add devices here in form Hostname: 'ip address'
  Manchester_MX10003: '10.198.206.3'
  Glasgow_MX10003: '10.198.206.6'
  London_MX104: '10.198.206.9'
  Leeds_MX104: '10.198.206.12'
  Bristol_MX104: '10.198.206.15'

Above are the file contents, where we are storing the data as Key, Value pair where Key is hostname of device and Values are its IP Addresses.

Now our data store is done, we can start with Juniper PYez (https://www.juniper.net/documentation/en_US/junos-pyez/topics/concept/junos-pyez-overview.html) to get the configs first.

We will be importing the Pyez modules, set the .yml input file location and run ‘for’ loop on our dictionary. We will store the backup config files as .txt file with hostnames as Title for easy accessibility. Don’t forget to import yaml module as we will be working on yaml file.

from jnpr.junos import Device
from jnpr.junos.exception import ConnectError
import yaml

input_file = '/home/sun/gitlab/device_list.yml'

for key, value in yaml.load(open(input_file)).items():

    dev_username = test
    dev_password = test
    dev = Device(host=value, user=dev_username, passwd=dev_password, port='830')

    try:
        dev.open()
    except ConnectError as err:
        print ("Cannot connect to device: {0}".format(err))
        sys.exit(1)
    except Exception as err:
        print (err)
        sys.exit(1)

    config = str(dev.cli("show configuration | no-more", warning=False))

    #####Create/Write to Config to File

    file_name = key + '.txt'


   dev.close()

That was pretty simple. Let me know if you have any issues. Now let’s work out how to push it to GIT.

We will be using the subprocess module to do this work for us where we will be running series of commands in serial to put the files across the gitlab. You are starting by going into particular directory which is your starting git project directory. You are looking at each branch within your git project using ‘git branch’ command and then checking out a particular branch on which you to work or store the data. In our case it’s Core_Network. Once you have moved to that, you can move into other specific folders and ‘mv’ your backup .txt files into this folder.

Once that is done, you will use git commands, like ‘git add –all’ to add all files as part of staging. ‘git commit –m “some description”’ to commit the files and then use ‘git push’ to push them to remote gitlab server.

Code will be:

from subprocess import PIPE
from subprocess import Popen
import sys
import os
import yaml
cmds =['cd /home/sun/gitlab/athena-test', 'git branch', 'ls -l', 'git checkout " Core_Network"', 'cd "Core Network"', 'cd "Device backups"', 'mv /home/sun/gitlab/*.txt /home/sun/gitlab/athena-test/"Core Network"/"Device backups"/', 'git add --all', 'git commit -m "Updated config files"', 'git push' ]
encoding = 'utf8'
p = Popen('/bin/sh', stdin=PIPE, stdout=PIPE, stderr=PIPE)
for cmd in cmds:
p.stdin.write(cmd + "\n")
p.stdin.close()
#print p.stdout.read()
print "Push to GitLab Completed"

One main thing you have to keep in mind is that you won’t be able to push the files until unless you have some remote servers configured under your server.

For example, mine is below where I have removed certain values and replaced them with xxx, yyy and ip address.. xxx is username, yyy is password and hostname  is gitlab server address.

sun@ $ git remote -v
origin https://xxx:yyy@<hostname>/athena-test.git (fetch)
origin https://xxx:yyy@<hostname>/athena-test.git (push)

Let’s see this in action by changing the config on one of the router.

write@re0.Manchester.MX10003.uk> edit
Entering configuration mode

write@re0.Manchester.MX10003.uk# run show configuration system services ssh
root-login allow;
protocol-version v2;
max-sessions-per-connection 32;
connection-limit 20;
rate-limit 10;

[edit]
write@re0.Manchester.MX10003.uk# set system services ssh connection-limit 10

[edit]
write@re0.Manchester.MX10003.uk# show | compare
[edit system services ssh]
- connection-limit 20;
+ connection-limit 10;

[edit]
write@re0.Manchester.MX10003.uk# commit
commit complete

Run the script now from CentOS Server

sun@ pwd

/home/sun/gitlab

sun@ $ python git_config_backup.py

Successfully Collected Configuration from Device: Bristol_MX104

Successfully Collected Configuration from Device: Manchester_MX10003

Successfully Collected Configuration from Device: Glasgow_MX10003

Successfully Collected Configuration from Device: London_MX104

Successfully Collected Configuration from Device: Leeds_MX104

Push to GitLab Completed

 

You can see config was pushed and if we see the gitlab, it’s clearly showing the difference from the previous backup it had. Red one is one that is removed and Green one is what has been added.

So you can use this utility as version control of your configs and once you are happy with the configs you can push them to master for production use 🙂

 

Gitlab
GitLab side by side comparison

 

One last thing, if you have to run this as a cronjob on Centos, just edit the crontab file using ‘crontab –e’ and add the path to the script and at what particular time you want to run and that’s all.

I hope you liked this blog. Let me know if you have any queries.

 

Regards

Mohit

Advertisement

JunOS Automation using PyEZ and Northstar REST APIs

Automation, Juniper, Junos, MPLS, Networking, NFV, RSVP, SDN, , , , , , , , , , ,

Hi All, in this session lets discuss some Automation.

During past few days, I was looking at some REST APIs for Juniper Northstar Controller. Now Northstar is good for LSP creation/deletion/modification but it cant configure the service E2E. Offcourse that tool is not meant to do all this but Juniper has recently released one beta version of it which can bind your LSP to some service which is excellent step forward. We will see that in a moment. Juniper is leveraging Jinja templates in NS to achieve this binding.

However as I said still service creation is not E2E and for that I thought of adding one more layer of automation and for this I have used Juniper own PyEZ framework which is basically Juniper Python library for automating tasks. Brilliant lets see how this work.

Juniper PyEZ is a framework which is easily grasped by Network engineers and you don’t need to be programmer to fully understand it.

https://www.juniper.net/documentation/en_US/junos-pyez/topics/concept/junos-pyez-overview.html

REST (REpresentational State Transfer) is a set of useful conventions and principals about transfer of information over the World Wide Web.

Many Web services are now using the principals of REST in their design.

When you type a URL into your browser, like http://example.net, your browser software creates an HTTP header that identifies:

  • a desired action: GET (“get me this resource”).
  • a target machine (www.domain-name.com).

The NorthStar RESTful APIs are designed to enable access over HTTP to most of the same data and analytics that are available to you from both the NorthStar GUI and the NorthStar CLI.

https://www.juniper.net/documentation/en_US/northstar3.1.0/information-products/api-ref/api-ref.html

Below is the pictorial representation of what we will be doing. I have used a Windows server on which we will write a script which will talk to Northstar using REST APIs and other components of Juniper Pes using PyEZ.

L2VPN CCC
Automation Model

 

Our Script will be written in Python and you can write the variables value in excel and pass it to the script.

Our excel format:

L2VPN_CCC_Data

import httplib
import json
import time
import re
import sys
import pandas as pd
from jnpr.junos import Device
from jnpr.junos.utils.config import Config
from pprint import pprint

df = pd.read_excel("L2VPN_CCC_Data.xlsx","Sheet1")

PE1 = str((df['PE1'].values.tolist())[0])
PE2 = str((df['PE2'].values.tolist())[0])
Interface_PE1 = str((df['Interface_PE1'].values.tolist())[0])
Unit_PE1 = str((df['Unit_PE1'].values.tolist())[0])
Vlan_PE1 = str((df['Vlan_PE1'].values.tolist())[0])
Interface_PE2 = str((df['Interface_PE2'].values.tolist())[0])
Unit_PE2 = str((df['Unit_PE2'].values.tolist())[0])
Vlan_PE2 = str((df['Vlan_PE2'].values.tolist())[0])
LSP_Name_PE1 = str((df['LSP_Name_PE1'].values.tolist())[0])
LSP_Name_PE2 = str((df['LSP_Name_PE2'].values.tolist())[0])
VPN_CCC_PE1 = str((df['VPN_CCC_PE1'].values.tolist())[0])
VPN_CCC_PE2 = str((df['VPN_CCC_PE2'].values.tolist())[0])

dev1 = Device(host=''+PE1+'', user='demo', password='password', port='22')
dev1.open()
dev1.timeout = 300

with Config(dev1, mode='private') as cu: 
cu.load('set interfaces '+Interface_PE1+' unit '+Unit_PE1+' description L2VPN-CCC encapsulation vlan-ccc vlan-id '+Vlan_PE1+' family ccc', format='set')
cu.pdiff() #Printing the difference in the configuration after the load
cu.commit()

dev1.close()
dev2 = Device(host=''+PE2+'', user='demo', password='password', port='22')
dev2.open()
dev2.timeout = 300

with Config(dev2, mode='private') as cu: 
cu.load('set interfaces '+Interface_PE2+' unit '+Unit_PE2+' description L2VPN-CCC encapsulation vlan-ccc vlan-id '+Vlan_PE2+' family ccc', format='set')
cu.pdiff() #Printing the difference in the configuration after the load#
cu.commit() #commit#

dev2.close()
conn = httplib.HTTPConnection('10.198.123.180:8091')
Bandwidth = raw_input('Please enter LSP Bandwidth on '+PE1+' (e.g 100k): ')
Setup_Pri = raw_input('Please enter Set up Priority: ')
Hold_Pri = raw_input('Please enter Hold Priority: ')
payload = str('{\r\n\"name\": \"'+LSP_Name_PE1+'\",\r\n\"creationConfigurationMethod\": \"NETCONF\",\r\n\"provisioningType\": \"RSVP\",\r\n  \"pathType\": \"primary\",\r\n  \"from\": {\r\n\"topoObjectType\": \"ipv4\",\r\n\"address\": \"'+PE1+'\"\r\n },\r\n  \"to\": {\r\n\"topoObjectType\": \"ipv4\",\r\n\"address\": \"'+PE2+'\"\r\n},\r\n\"plannedProperties\": {\r\n\"bandwidth\": \"'+Bandwidth+'\",\r\n\"setupPriority\": '+Setup_Pri+',\r\n\"holdingPriority\": '+Hold_Pri+',\r\n\"userProperties\": {\r\n \"ccc-vpn-name\": \"'+VPN_CCC_PE1+'\",\r\n \"ccc-interface\": \"'+Interface_PE1+'.'+Unit_PE1+'\",\r\n\"transmit-lsp\": \"'+LSP_Name_PE1+'\",\r\n\"receive-lsp\": \"'+LSP_Name_PE2+'\"\r\n    }\r\n  }\r\n}\r\n')
headers = {
 'content-type': "application/json",
'cache-control': "no-cache",
 }

conn.request ("POST", "/NorthStar/API/v2/tenant/1/topology/1/te-lsps", payload, headers
res = conn.getresponse()
data = res.read()
print 'Please wait while we get the status of LSP you created :)'
for i in xrange(25,0,-1):
 time.sleep(1)
 sys.stdout.write(str(i)+' ') 
 sys.stdout.flush()
 conn.request("GET", str('/NorthStar/API/v2/tenant/1/topology/1/te-lsps/search?name=' + LSP_Name_PE1), headers=headers
 res = conn.getresponse()
 data = res.read()

LSP_Status = re.search('operationalStatus":(.*?),', data).group(1)
if LSP_Status == '"Active"':
  print ('\nSuccess: LSP "'+LSP_Name_PE1+'" is Created and Active')
elif LSP_Status == "Down":
   print ('\nFailed: LSP "'+LSP_Name_PE1+'" is created however Down')
else:
  print ('\nFailed: LSP "'+LSP_Name_PE1+'" is not created and is in Unknown State on Northstar')

time.sleep(10)

conn = httplib.HTTPConnection('10.198.123.180:8091')
Bandwidth = raw_input('Please enter LSP Bandwidth on '+PE2+' (e.g 100k): ')
Setup_Pri = raw_input('Please enter Set up Priority: ')
Hold_Pri = raw_input('Please enter Hold Priority: ')

payload = str('{\r\n\"name\": \"'+LSP_Name_PE2+'\",\r\n\"creationConfigurationMethod\": \"NETCONF\",\r\n\"provisioningType\": \"RSVP\",\r\n  \"pathType\": \"primary\",\r\n  \"from\": {\r\n\"topoObjectType\": \"ipv4\",\r\n\"address\": \"'+PE2+'\"\r\n },\r\n  \"to\": {\r\n\"topoObjectType\": \"ipv4\",\r\n\"address\": \"'+PE1+'\"\r\n},\r\n\"plannedProperties\": {\r\n\"bandwidth\": \"'+Bandwidth+'\",\r\n\"setupPriority\": '+Setup_Pri+',\r\n\"holdingPriority\": '+Hold_Pri+',\r\n\"userProperties\": {\r\n \"ccc-vpn-name\": \"'+VPN_CCC_PE2+'\",\r\n \"ccc-interface\":\"'+Interface_PE2+'.'+Unit_PE2+'\",\r\n\"transmit-lsp\": \"'+LSP_Name_PE2+'\",\r\n\"receive-lsp\": \"'+LSP_Name_PE1+'\"\r\n    }\r\n  }\r\n}\r\n')
headers = {
 'content-type': "application/json",
 'cache-control': "no-cache",
   }

conn.request ("POST", "/NorthStar/API/v2/tenant/1/topology/1/te-lsps", payload, headers)
res = conn.getresponse()
data = res.read()
print 'Please wait while we get the status of LSP you created :)'
for i in xrange(25,0,-1):
   time.sleep(1)
   sys.stdout.write(str(i)+' ')
   sys.stdout.flush()

conn.request("GET", str('/NorthStar/API/v2/tenant/1/topology/1/te-lsps/search?name=' + LSP_Name_PE2), headers=headers)
res = conn.getresponse()
data = res.read()
LSP_Status = re.search('operationalStatus":(.*?),', data).group(1)
if LSP_Status == '"Active"':
    print ('\nSuccess: LSP "'+LSP_Name_PE2+'" is Created and Active')
elif LSP_Status == "Down":
    print ('\nFailed: LSP "'+LSP_Name_PE2+'" is created however Down')
else:
    print ('\nFailed: LSP "'+LSP_Name_PE2+'" is not created and is in Unknown State on Northstar')

time.sleep(5)

dev1.open()
dev2.open()

print (dev1.cli('show connections remote-interface-switch '+VPN_CCC_PE1+'', warning=False))

print (dev2.cli('show connections remote-interface-switch '+VPN_CCC_PE2+'', warning=False))

dev1.close()
dev2.close()

In this script we are making reading the values from the excel and using it as variables in or script.

After that using PyEZ, making a SSH connection to PE1 and PE2 and configuring the layer 2 sub-interfaces with vpn-ccc encapsulations. Once that is done, connection to Northstar server 10.198.123.180 using httplib libraris/modules is made and waiting for Northstar to configure the LSP. At this stage Northstar is also binding that LSPs in connections using Jinja template. Once Northstar has created the LSPs we are using regular expression to get the LSP Index from Northstar and checking whether LSP creating in Success or failed.

At last we are printing the show command output to find out if everything is up and running 🙂

Lets see by running the script

C:\Program Files (x86)\Python\Northstar_Scripts\Working\Juniper\L2VPN_CCC>python
 E2E_L2VPN_CCC_Script.py
[edit interfaces xe-2/0/0]
+ unit 601 {
+ description L2VPN-CCC;
+ encapsulation vlan-ccc;
+ vlan-id 601;
+ family ccc;
+ }
[edit interfaces xe-2/0/0]
+ unit 601 {
+ description L2VPN-CCC;
+ encapsulation vlan-ccc;
+ vlan-id 601;
+ family ccc;
+ }
Please enter LSP Bandwidth on 10.198.123.100 (e.g 100k): 70m
Please enter Set up Priority: 5
Please enter Hold Priority: 0
Please wait while we get the status of LSP you created :)
25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1
Success: LSP "l2vpn-ccc-1" is created and is Active
Please enter LSP Bandwidth on 10.198.123.205 (e.g 100k): 70m
Please enter Set up Priority: 5
Please enter Hold Priority: 0
Please wait while we get the status of LSP you created :)
25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1
Success: LSP "l2vpn-ccc-2" is created and is Active
CCC and TCC connections [Link Monitoring On]
Legend for status (St): Legend for connection types:
 UN -- uninitialized if-sw: interface switching
 NP -- not present rmt-if: remote interface switching
 WE -- wrong encapsulation lsp-sw: LSP switching
 DS -- disabled tx-p2mp-sw: transmit P2MP switching
 Dn -- down rx-p2mp-sw: receive P2MP switching
 -> -- only outbound conn is up Legend for circuit types:
 <- -- only inbound conn is up intf -- interface
 Up -- operational oif -- outgoing interface
 RmtDn -- remote CCC down tlsp -- transmit LSP
 Restart -- restarting rlsp -- receive LSP
Connection/Circuit Type St Time last up # Up tran
s
l2vpn-ccc rmt-if Up Nov 25 12:52:10
1
 xe-2/0/0.601 intf Up
 l2vpn-ccc-1 tlsp Up
 l2vpn-ccc-2 rlsp Up

CCC and TCC connections [Link Monitoring On]
Legend for status (St): Legend for connection types:
 UN -- uninitialized if-sw: interface switching
 NP -- not present rmt-if: remote interface switching
 WE -- wrong encapsulation lsp-sw: LSP switching
 DS -- disabled tx-p2mp-sw: transmit P2MP switching
 Dn -- down rx-p2mp-sw: receive P2MP switching
 -> -- only outbound conn is up Legend for circuit types:
 <- -- only inbound conn is up intf -- interface
 Up -- operational oif -- outgoing interface
 RmtDn -- remote CCC down tlsp -- transmit LSP
 Restart -- restarting rlsp -- receive LSP

Connection/Circuit Type St Time last up # Up tran
s
l2vpn-ccc rmt-if Up Nov 25 12:52:11
1
 xe-2/0/0.601 intf Up
 l2vpn-ccc-2 tlsp Up
 l2vpn-ccc-1 rlsp Up

C:\Program Files (x86)\Python\Northstar_Scripts\Working\Juniper\L2VPN_CCC>

 

So that’s all for today.. You can see the possibility of using this framework in so many tasks in your daily networking journey. I hope you like this blog and will try to use it in your network 🙂

Regards

Mohit