© 2017 Nokia1

Configuration Management- Large Systems challenges

Kari Rossi

Bell Labs Fellow, Head of Technology and Architecture

Network ManagementApplications and Analytics

Nokia

2

• Definitions

• Good software design principles for configuration management

• Infrastructure as Code – software configuration management

• Modeling configurations

• Examples: network management

Contents

33 © Nokia 2016

Definitions

4

• CM is the practice of handling changes systematically so that a system maintains its integrity over time

• CM implements the policies, procedures, techniques, and tools that manage, evaluate proposed changes, track the status of changes, and maintain an inventory of system and support documents as the system changes

• CM programs and plans provide technical and administrative direction to the development and implementation of the procedures, functions, services, tools, processes, and resources required to successfully develop and support a complex system

Configuration Management (CM) definition (Wikipedia)

5

• In software engineering, software configuration management (SCM) is the task of tracking and controlling changes in the software, part of the larger cross-disciplinary field of configuration management

• SCM practices include revision control and the establishment of baselines. If something goes wrong, SCM can determine what was changed and who changed it

Software engineering - Configuration Management

6

• Configuration Management is an Information Technology Infrastructure Library (ITIL) and an IT Service Management (ITSM) process that tracks all of the individual Configuration Items (CI) in an IT system which may be as simple as a single server, or as complex as the entire IT department

• Configuration Management means software for managing the configuration and orchestration of of a large number of computers using a central configuration model. Examples: Ansible, Chef, Puppet

Configuration Management in production systems

• FCAPS is the ISO Telecommunications Management Network model and framework for network management. The goals of Configuration Management include:

• to gather and store configurations from network devices (this can be done locally or remotely)

• to simplify the configuration of the device

• to track changes that are made to the configuration

• to configure ('provision') circuits or paths through non-switched networks

• to plan for future expansion and scaling.

7

• What is a large software development project for you?

• What is large system for you?

• What is large software system for you?

Large projects and systems

8

• Millions of lines of code, lots of components and dependencies• Is part of a bigger system• 100 - 1000+ persons• Persons in several countries, several time zones• Strict non functional requirements (security, availability, …)• New non functional features being the most difficult ones to implement• Continuous integration within and between different systems

• Parallel and continuous release models multiplies the issues!

Some characteristics of large projects and systems

9

9

Good software design principles for configuration management

10

Good design principles

• What are good design principles for software?

• What is special for configuration management?

11

• Application as set of loosely coupled and collaborating services, each service implements a set of narrowly, related micro service

• Stateless design

• Externalize configuration

• Dynamic service registry and lookup with collaborating parties

• Treat backing service as attachable resource

• Use non-blocking asynchronous communication in a loosely coupled architecture

• Scale automatically using proactive and reactive actions cost-optimizes to run efficiently, not wasting resources

Cloud native principles - example

Refer to Twelve Factor App principles https://12factor.net

• Scale automatically using proactive and reactive actions cost-optimizes to run efficiently, not wasting resources

• Upgrades without downtime

• In development, build and run stages are strictly separated

12

• What is a small and big software development project for you?

• What is big system for you?

• What is big software system for you?

Code and configuration

• Why you should separate code from configuration?

13

• Configuration of an application are the installation and run-time parameters which are different in various deployments (SW development, customer environments)

• Examples• Installation parameters, such as number of servers, IP addresses, directories

• Language of an application, appearance (fonts, color), templates

• Resource handles to the database and other backing services

• Credentials to external services such as Amazon S3

• Applications should not store configuration as constants in the code• Configurations are different between different customer deployment, code is not

Separate of code and configuration

14

• Strict separation between the software development and live environments• For example, it is impossible to make changes to the code at runtime, since

there is no way to propagate those changes back to the build stage

Strictly separated software development and live stages

Source: Twelve Factor App principles https://12factor.net

15

15

Infrastructure as CodeSoftware Configuration Management

16

• Infrastructure as code (IaC) is the process of managing and provisioning computer data centers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools

• This Code replaces manual tasks and can be version controlled, automated tested, integrated to Continuous Integration/delivery pipelines

• Code that can be modular, re-usable and provides direct documentation of what really happens to the system

• Examples: Ansible, Chef, Puppet

Definition – infrastructure as Code

17

• An automation engine that automates cloud provisioning, configuration management, and application deployment

• Automation jobs are defined by Playbooks (YAML)• Ansible connects to the nodes, and pushes small programs (modules) to the nodes. The

modules are resource models of the desired state of the system• As Playbooks support multi-machine deployment system they are well suited to

deploying complex applications

• Playbooks can launch tasks synchronously or asynchronously• Playbooks can sequence multi-tier rollouts involving rolling updates, and can delegate

actions to other hosts, interacting with monitoring servers and load balancers along the way

Example - Ansible

18

• What is a small and big software development project for you?

• What is big system for you?

• What is big software system for you?

Complex software updates

• What kind of design principles could you use for designing a system so that you can update the software without downtime?

19

19

Modeling configurations

20

Basic concepts

Managed objects

ManagerClientManagement application

AgentServerDeviceNetwork ElementVirtual Network Function

Resources modelledas managed objects

Actions to managed objectstrigger the action to the corresponding real resources

21

Example

Managed objects

new_config.language=“fi”

manager.connect(host=“Mac Kari”, username=“root”);manager.edit_config(config=new_config);

What happens:- remote connection to “Mac Kari”- system language is set to “fi”

Same as the terminal command:

Mac Kari% sudo languagesetup –langspec 14

module user-settings {…leaf language {

type string;}…

Mac KariManager application

22

Modeling

• Think about examples of managed object classes, instances and operations for laptop management?

23

• The Network Configuration Protocol (NETCONF) is a network management• protocol developed and standardized by the IETF

• NETCONF provides mechanisms to install, manipulate, and delete the configuration of network devices

• NETCONF operations are realized on top of a simple Remote Procedure Call (RPC) layer. The NETCONF protocol uses XML based data encoding for the configuration data as well as the protocol messages

• YANG is a data modeling language for the definition of data sent over the NETCONF• YANG can be used to model both configuration data as well as state data of network

elements

NETCONF and YANG

24

• Installing, manipulate, delete the configurations of network devices

• Support both configuration state (e. g. IP address) and operational state (e. g. error counters)

• Supporting several configurations of a device, e. g. running configuration and configuration to be used next

• Distinguishing the distribution of the configuration from the activation of the configuration

• Reporting configuration change events

• Full configuration dumb and full configuration restore

What can be done with NETCONF?

25

25

Network ManagementExamples

26

Simple network management

x10NE NE NE NE x20

With small number of

network elements only

some automation is needed e.g. by custom scripts

With small number of network

elements operator can manage them by making terminal

connections

27

Network management systems

NE NE NE NE X100+NE NE NE NE

Network Management

System

I cannot manage my job manually and with simple

scripts

Network is changing

constantly, automation is

needed

28

Network management: CM configurations life-cycle

CM Actualconfiguration

CM newconfiguration

NENE Active

configuration

NE Pre-activated

configuration

CM

Upload

Create newconfiguration

Pre-activate(Download)

Activate

29

Examples of technical requirements for CM

• NE integration: auto-discovery vs. manual

• Number of elements: from 100s to 100 000s

• Configuration change response: from seconds to minutes lasting operations• Configuration size: from kilobytes to tens of gigabytes

• Configuration change activation: immediate vs. delayed - separation of transfer, validation and activation

• Operations mode: synchronous vs. asynchronous

• Frequency of data emitted: rare vs. frequent vs. continuous • Origins of changes: external (mgmt. system) vs. ”self-inflicted” (from air interface, SW

itself, …)