Axians Lab: Opensource Network Config Backup

Guest Blog from Alex Gittings: see original post here

Opensource Network Backups for over 120 Network Devices

Oxidised is a network device configuration backup tool which was developed to be a replacement for Rancid. Oxidized is written in the Ruby programming language and is very extensible and supports over 120 operating system types.

Alex will go through running, installing and configuring Oxidized to enable the collection of configs from Juniper EX's Switch's. These configurations will then be stored on a Gitlab account for version control.

Oxidized Highlights:

• Automatically adds/removes threads to meet configured retrieval interval
• Restful API to a move node immediately to head-of-queue (GET/POST /node/next/[NODE])
• Syslog udp+file example to catch config change events (IOS/JunOS) and trigger a config fetch
  • Will signal which IOS/JunOS user made the change, can then be used by output modules (via POST)
  • The git output module uses this info - 'git blame' will show who changed each line, and when
• Restful API to reload list of nodes (GET /reload)
• Restful API to fetch configurations (/node/fetch/[NODE] or /node/fetch/group/[NODE])
• Restful API to show list of nodes (GET /nodes)
• Restful API to show list of version for a node (/node/version[NODE]) and diffs

There are a number of installation methods available for this tool, including building from source, Ruby Gems and also docker containers

For this blog I will be using the Docker container as we run multiple Docker containers within Axians PS Lab.

First we need to create a configuration directory on the host system:

mkdir /home/axians/lab-host/oxidized

We then need to run the container for the first time to initialize the configuration:

Note: this step is only required for creating the Oxidized configuration file and can be skipped if you already have one.

docker run --rm -v /home/axians/lab-host/oxidized:/root/.config/oxidized -p 8888:8888/tcp -t oxidized/oxidized:latest oxidized

You should get the following output:

axians@axians:~/lab-host/oxidized$ docker run --rm -v /home/axians/lab-host/oxidized:/root/.config/oxidized -p 8888:8888/tcp -t oxidized/oxidized:latest oxidized

Unable to find image 'oxidized/oxidized:latest' locally

latest: Pulling from oxidized/oxidized

c64513b74145: Pull complete

01b8b12bad90: Pull complete

c5d85cf7a05f: Pull complete

b6b268720157: Pull complete

e12192999ff1: Pull complete

d39ece66b667: Pull complete

65599be66378: Pull complete

e9876d13469e: Pull complete

950007a6374b: Pull complete

a984980ba6a9: Pull complete

3a0d8dcf44da: Pull complete

09990a0f703c: Pull complete

0db6d82bf294: Pull complete

c6a965a221ba: Pull complete

450d5e516083: Pull complete

bab66af4c4a4: Pull complete

0de7c68519b4: Pull complete

13a57a4e8cc5: Pull complete

941fa46b3540: Pull complete

736f8438e0d0: Pull complete

Digest: sha256:ec63031b64183c0f3486737d18cc6cc0bf3d30e36fff34d6a4ae53c88177b874

Status: Downloaded newer image for oxidized/oxidized:latest

edit ~/.config/oxidized/config

There should now be a config file within your directory which contains the following config:

axians@axians:~/lab-host/oxidized$ cat config

---

username: username

password: password

model: junos

resolve_dns: true

interval: 3600

use_syslog: false

debug: false

threads: 30

timeout: 20

retries: 3

prompt: !ruby/regexp /^([\w.@-]+[#>]\s?)$/

rest: 127.0.0.1:8888

next_adds_job: false

vars: {}

groups: {}

models: {}

pid: "/root/.config/oxidized/pid"

crash:

  directory: "/root/.config/oxidized/crashes"

  hostnames: false

stats:

  history_size: 10

input:

  default: ssh, telnet

  debug: false

  ssh:

    secure: false

  ftp:

    passive: true

  utf8_encoded: true

output:

  default: file

source:

  default: csv

model_map:

  juniper: junos

  cisco: ios

Now that we have our default Oxidized configuration we can prepare Gitlab for the new system.

Create User

If required create a new user for your Gitlab system through Admin > Users > New User and enter the required details.

We will be using an account that is already created so I will be skipping this step.

Add SSH Keys

Once logged in we need to add SSH keys to the Gitlab account so that Oxidized is able to push changes to repositories. If you dont know how to do this please follow the following tutorial from Gitlab on how to create an SSH Key Pair. Link

This can be done in Settings > SSH Keys

Gitlab Project: Create a project in order to store the device configurations.

Projects > New Project

Once this is created, navigate to the project page and copy the project url. This is required later.

Oxidized Configuration

Back on the server, edit the file config that is now in the directory you created.

We need to change multiple parameters in order to be able to have a functioning configuration file.

The configuration file is in a yaml format. There are a number of key elements that make up this configuration file which are:

• interval - How often to collect data (seconds).
• input - The method to get config from devices.
• output - How to store the configuration.
• source - The data souce for the list of devices.
• model_map - Used to map device groups to models.
• groups - Group specific variables
• models - Model specific variables.

Under the source section you will notice the defaut method is csv.

Firstly we need to create a data source file called router.db in our directory.

# /home/axians/lab-host/oxidized/router.db

# name:model:ip

TOR_MGMT_EX2200_B15:junos:192.168.3.35

TOR_MGMT_EX2200_B17:junos:192.168.3.37

TOR_MGMT_EX2200_B18:junos:192.168.3.38

TOR_MGMT_EX2200_B19:junos:192.168.3.39

TOR_MGMT_EX2200_B20:junos:192.168.3.40

TOR_MGMT_EX2200_B21:junos:192.168.9.34

TOR_MGMT_EX4200_B14:junos:192.168.3.34

TOR_MGMT_EX4200_B16:junos:192.168.3.36

I have added some information regarding the information in each column.

We now need to edit the config file so that our source looks like the following:

source:

  default: csv

  csv:

    file: "/root/.config/oxidized/router.db"

    delimiter: !ruby/regexp /:/

    map:

      name: 0

      model: 1

      ip: 2

    gpg: false

Note: the file key value does not match the folders mentioned previously, this is because the final configuration will be run within a docker container to which the configuration file will be mapped.

• file - Location of the source router.db file.
• delimiter - This is a Ruby regular expression to break the csv into columbs based on :.
• map - The map section defines the column number in which to find attributes such as name and IP in the data source.
• gpg - This is to tell Oxidized if the file is encrypted.

To output the configurations to Gitlab we need to alter the configuration file so that it knows how to push the configuration to the repository.

The required configuration looks like the following:

Note: make sure you replace axians-ps-lab-configs.git with your .git repository name as well as changing the user and email.

output:

  default: git

  file:

    directory: "/root/.config/oxidized/configs"

  git:

    single_repo: true

    user: Oxidized

    email: axiansdeveloper@axians.co.uk

    repo: "/root/.config/oxidized/configs/axians-ps-lab-configs.git"

This config instructs Oxidized to create a bare repo on the local host to store the device configurations.

• file - This tells the application where to put the output configuration
• single_repo - If true Oxidized will only create one repo for all device groups if not a different repo will be used for each device group.
• user - User information to attach to the git commit.
• email- User infromation to attach to the git commit.
• repo - The repo that will have the configuration pushed to.

The hooks section allows us to define the details of the Gitlab repo url from the previous step.

The configuration below will attempt to push the remote repo when a diff in a devices configuration is detected.

hooks:

  push_to_remote:

    type: githubrepo

    events: [post_store]

    remote_repo: "git@gitlab.com:axiansdeveloper/axians-ps-lab-configs.git"

    publickey: "/root/.ssh/id_rsa.pub"

    privatekey: "/root/.ssh/id_rsa"

• type - This indicates the hook to execute a git push to a configured remote_repo when the specific event is triggered.
• events - There are many events but post_store is triggered after a node configuration is stored (this is executed only when the configuration has changed).
• remote_repo - The remote repo to be pushed to.
• publickey - public key file path for repository authentication.
• privatekey - private key file path for repository authentication.

This is the time to edit the configuration file to change username passwords etc to make sure it fits your environment.

If you would like to enable the web interface change the rest config to have an IP address of 0.0.0.0 which will accept connections from the network.

Resulting configuration:

---

username: axians

password: P@55w0rd

model: junos

resolve_dns: true

interval: 3600

use_syslog: false

debug: false

threads: 30

timeout: 20

retries: 3

prompt: !ruby/regexp /^([\w.@-]+[#>]\s?)$/

rest: 0.0.0.0:8888

next_adds_job: false

vars: {}

groups: {}

models: {}

pid: "/root/.config/oxidized/pid"

crash:

  directory: "/root/.config/oxidized/crashes"

  hostnames: false

stats:

  history_size: 10

input:

  default: ssh, telnet

  debug: false

  ssh:

    secure: false

  ftp:

    passive: true

  utf8_encoded: true
source:
 default: csv
 csv:
 file: "/root/.config/oxidized/router.db"
 delimiter: !ruby/regexp /:/
 map:
 name: 0
 model: 1
 ip: 2
 gpg: false
output:
 default: git
 file:
 directory: "/root/.config/oxidized/configs"
 git:
 single_repo: true
 user: Oxidized
 email: axiansdeveloper@axians.co.uk
 repo: "/root/.config/oxidized/configs/axians-ps-lab-configs.git"
hooks:
 push_to_remote:
 type: githubrepo
 events: [post_store]
 remote_repo: "git@gitlab.com:axiansdeveloper/axians-ps-lab-configs.git"
 publickey: "/root/.ssh/id_rsa.pub"
 privatekey: "/root/.ssh/id_rsa"
model_map:
 juniper: junos
 cisco: ios
 

Running Oxidized

To run the Oxidized in the docker container using the configuration we have created there are two ways, Docker Run and Docker-Compose. Firstly, we will look at Docker Run

Docker Run

To run our Oxidized container we can use the following command:

docker run -v /home/axians/lab-host/oxidized:/root/.config/oxidized -v /home/axians/.ssh:/root/.ssh -p 8888:8888/tcp -t oxidized/oxidized:latest

Note: change the volume mappings to have your folder locations of the configuration and router.db and also the public and private key folder so that Oxidized is able to them for Gitlab authentication

Docker Compose

To use docker-compose it must be installed.

We then need to create a file called docker-compose.yml

# /home/axians/lab-host/oxidized/docker-compose.yml

oxidized:

  restart: always

  image: oxidized/oxidized:latest

  ports:

    - 8888:8888/tcp

  environment:

    CONFIG_RELOAD_INTERVAL: 600

  volumes:

    - /home/axians/lab-host/oxidized:/root/.config/oxidized

    - /home/axians/.ssh:/root/.ssh

Once saved we can then run this container with docker-compose up.

You will see the following output:

To run this headless use docker-compose up -d.

Verification: Now that the Oxidized is running it will backup every hour.

There will be configurations within your Github.

Note: configs will only change when there is a diff.

Web: If we navigate to the host IP address port 8888 we will be able to see the Oxidized UI.

You can see the changes and diffs within the UI and also gitlab which will allow you to compare versions.

Resources

• https://github.com/ytti/oxidized
• https://gitlab.com/help/ssh/README#generating-a-new-ssh-key-pair