provisioning oracle fusion middleware environments with chef and puppet

Provisioning Oracle FMWEnvironments with Chef & PuppetCON7629

Mark Nelson, Edwin BiemondFusion Middleware Platform TeamOctober 02, 2014

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 3

• Mark Nelson– Architect, Oracle


• Edwin Biemond– Consulting Member Technical Staff, Oracle


Agenda

Current State

Experiences

Continuous Delivery

Testing

Wrap Up, Q & A

1

2

3

4

5


What is possible Now!!!


Focus on Puppet & Chef, which are the most popular provisioning tools

• > 10.000.000 nodes• Fastest growing companies in the US• Stand-alone (Client) & Master (Server)• Open Source & Enterprise / Premium– Free < 10 Nodes (Puppet)– Free < 5 Nodes (Chef)

• Manage Linux, Solaris & Windows• Cloud plugins, VMware, Google, Openstack,

Azure & Amazon


Chef components overview


Next major release• SOA architecture• Services• Easy to scale up• Own release lifecycle

• Runs in a JVM

• 3 times faster

• Build with clojure, same as PuppetDB

• JRuby

Puppet component overview


• Administrator / Enterprise focus• Forge.puppetlabs.com > 2700

Modules• Language, Puppet DSL & Ruby• Manifest-> Some Puppet DSL Code • Puppet Modules• Facter• Type/Provider -> Ruby

• Developer focus• Supermarket.getchef.com> 1600

Cookbooks• Language, Chef DSL & Ruby• Recipe -> Some Chef DSL Code• Chef Cookbooks• Ohai• Resource/Provider– Light -> Chef DSL– Heavy -> Ruby


file{'/etc/resolv.conf':

content => template('mod/my_resolv.conf.erb'),

owner => 'root',

group => 'root',

mode => '0644',

}

$files = ['bacon', 'eggs', 'sausage']

file{$files:

content => "#{title} is delicious!",

}

package{'bash':

ensure => latest,

}

template '/etc/resolv.conf' do

source 'my_resolv.conf.erb'

owner 'root'

group 'root'

mode '0644'

end

['bacon', 'eggs', 'sausage'].each do |type|

file "/tmp/#{type}" do

content "#{type} is delicious!"

end

end

package 'bash' do

action :upgrade

end

Some DSL Code


So how does it work!!• We have to create some recipes/manifests which describes the To Be

situation & not how to achieve it • Execute chef-apply or puppet apply• Chef/Puppet retrieves the current state and determines what the necessary

actions should be• Executes the actions (apply only deltas)


Agenda

Current State

Experiences

Continuous Delivery

Testing

Wrap Up, Q & A

1

2

3

4

5

15Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

ExperiencesChef & Puppet challenges & solutions


WebLogic 12c Changes• All jars, no O.S. specific installer– Jars can be used on every Operating System

• Shared oracle_common which is part of WebLogic Infrastructure– ServiceBus installer +/- 400Mb

• Repository Creation Utility is now part of WebLogic Infrastructure– FMW Database repository tightly connected to the domain (because of OPSS)

• WebLogic Restful Management Services


Oracle Tools like OPatch, RCU or BSU are not idempotent• 2nd Times always fails– It breaks Chef or Puppet– Hard to handle this with the standard Puppet/Chef exec resource– Slow, inventory command is always faster then apply and let it crash

• For example OPatch utility– Detect current state with opatch lsinventory• Every utility has a different output ( XML, CSV or text ouput )

– Determine the right action– Skip, Install or Remove the patch with opatch apply– Check the outcome


One Time• Not really the Chef or Puppet way• One big or many WLST scripts• Use exec of Chef or Puppet, almost

impossible to control (Facter/Ohai)• Execute this once or need to catch

the BeanAlreadyExists exception• Re-use your old scripts• Easy to get out of sync

Model driven• WebLogic Chef/Puppet Resources• The Model is always up-to-date• Need to know some Ruby• Map WLST or the WLS Rest

Interface (12.1.3) to Chef or Puppet Attributes• 4 WLST scripts per WLS resource,

retrieve, create/update and destroy• Big effort to have all WLS resources

One time provisioning or Model driven


• BSU, OPatch, AdminServer & ManagedServer Control• Machine, Domain, Server, Channels• (Dynamic) Cluster, Coherence,

Server Template• User, Group, Authentication

providers• Deployment

• Datasource• File Persistence• JMS server, JMS module,

Connection Factory, Queue, Topic, SubDeployment, Quota, Foreign Server• SAF agents, Imported Destination,

Remote Context, Error handler• Workmanagers• Virtualhost

Puppet WebLogic Resources ( Total of 37)


Puppet apply Puppet resource wls_server SoaServer1 --edit

Model driven resource demo


WLST Performance

• Startup of WLST takes times-> even with skipWLSModuleScanning

• Connect() or readDomain(‘xx’) is slow

• With > 10 WLST scripts or with multiple Domains you will have some bad performance

• Don’t schedule a Chef or Puppet run every XX minutes


WLST Domain Daemon process

• For Every domain

• Is already connected to the Domain

• Auto start & destroy in every Puppet/Chef run

• More complexity -> need to know when it is finished or when it fails

• Can’t handle invalid WLST/Python scripts

• It’s Fast -> 2 Domains with 8 clusters and > 100 Queues in just a few minutes ( normal 30-60 minutes)


Naming of WebLogic artifacts• In Chef or Puppet a resource should be unique. There can be only one

Firewall Service or one Package Bash• In WebLogic you can have multiple Domains and a Queue only has to

unique inside a JMS Module.• Need to have a WebLogic Title convention and implement Title pattern in

all the Chef/Puppet Resources


Orchestration of FMW HA Environment• A High Available FMW environments needs at least– A Database configuration with Oracle RAC or Oracle Dataguard ( 10 a 30 min ) –WebLogic AdminServer node which depends on the Database ( 5 a 25 min )– 2 or more WebLogic nodes which depends on the AdminServer ( 5 a 10 min )

• Chef or Puppet can’t orchestrate multi-node environments• Can implement some try / catch together with a repeat & wait – AdminServer should wait for a Database with a FMW Repository–WebLogic Nodes should wait for a Domain pack on the AdminServer

• We need something on top of Chef/Puppet which can handle this


Puppet & Chef Sandbox environments

Oracle VirtualBox

Vagrant

Desktop

Shared Folder Puppet / Chef code

VM

VMTemplate

Puppet/ChefClient

Oracle

JDK

WebLogic

Database

GithubForge

Supermarket

yum / wget

• Out of the box with Chef Test Kitchen• Puppet -> requires some Vagrant

configuration

• Easy to simulate environments• Destroy & Re-Create• Direct feedback, Change Log levels• Remote shell access• Only need some MB of configuration data• Retrieve the latest Modules or Cookbooks


Oracle Big Files

• Download & Extract zip files requires a lot of memory or a big swap file

• Takes a lot of time to download plus extract & requires disk space

• It is slow in every run because Puppet/Chef file resource checks for differences (locally and remote)

• Use a Shared Drive

• Skip the File resource step with some Puppet/Chef Facts


Agenda

Current State

Experiences

Continuous Delivery

Testing

Wrap Up, Q & A

1

2

3

4

5


Continuous Delivery


Separate environment specific variables from your code• Puppet – Facter– Parameter manifest class– Hiera ( Multiple levels Node, Env, Common ) with a YAML, Database, JSON Backend

• CHEF–OHAI– Roles– Environments– Databags– Attribute


Promotion of infrastructure• Versioning (Git Suite -> Gitbucket, Gitlab or Github) for infrastructure

configuration & Development code– Issues– Pull Requests

• Multi environment on the master (Development, Test, Staging, Prod)• Retrieve Modules/Cookbooks from Git or use Forge or Supermarket– Puppet -> Puppet Librarian, R10k– Chef -> Berkshelf, Chef Librarian


Model your environments• Define Building blocks or Roles– Better to understand• just assign to nodes

– Easier to maintain• Consist of small modules• Optimal re-use

– Auto-adapt to the environment• Developer desktop• High Available production environment


Standardize your VM Base ImagePacker.io is a tool for creating identical machine images for multiple platforms from a single source configuration–Works with PXE boot– Download ISO/Net install– Installs Chef or Puppet client– Different post-processors• Vagrant for VMware or Oracle Virtualbox• Amazon• Docker• OpenStack• Google

–Only have to provide a Kickstart file


Continuous Environment Overview – Modules/Cookbooks

Git• Clone

Code• RVM (Parallel)• Chef version

• Lint, RSpec

Smoke Test• Vagrant• Librarian

Approve• Archive• Version

Distribute• Repository


Continuous Environment Overview – Building blocks

Git• Clone

Smoke Test• Vagrant• Librarian

Approve• Archive• Version

Distribute• Repository


Continuous Environment Overview – Test 2 Production

Test

VMs

Cookbooks

RecipesProductionStaging

Test Environment 1

Test Environment 2


Agenda

Current State

Experiences

Continuous Delivery

Testing

Wrap Up, Q & A

1

2

3

4

5


Test/Code Quality frameworks• Rubocup for Ruby coding style

• Chef– Foodcritic for Chef coding style– ChefSpec/Serverspec

• Puppet– Puppet-Lint for Puppet coding style– Rspec

• RVM/Bundler to test the different Ruby & Puppet/Chef combinations• Sublime or IntelliJ plugins


Test against different versionof• Ruby• Chef/Puppet

Test for

• Lint• RSpec Tests• Code Coverage

Opensource

Test in the cloud with Travis CI


Robot Framework• Test your FMW environment if it contains what you expect.– Can’t fully rely on Puppet or Chef output

• With Robot you can add tests in readable and understandable text• Libraries– Selenium for WebLogic Web applications tests– HTTP/Rest Requests for WebLogic Restful Management Services


Test WebLogic applications with Robot


Agenda

Current State

Experiences

Continuous Delivery

Testing

Wrap Up, Q & A

1

2

3

4

5


• JDK7– URandom fix, JCE Security policy

• ORAWLS– Install, Patch, RCU, Domain, Cluster, WLS Native Types

• ORADB, ORA_RAC– Install, ASM, RAC, Patch, Instance, RCU 11g

• ORACLE– DB Native Types like Tablespace, Role, User & Init params

• > 10k Downloads

• Many Contributions/Pull requests

• Many Big Enterprise companies are using it

• Developers with Vagrant/Virtualbox

Download locations

• Forge.puppetlabs.com

• Github.com/biemond

• Github.com/hajee

Opensource Puppet Modules


ConclusionFinally we have total control over FMW Environments without any human interaction.

And experience the provisioning of new FMW environments in minutes and push changes to DTAP


Q & [email protected]@oracle.com

provisioning oracle fusion middleware environments with chef and puppet

Software