ovirt and gluster_hyperconvergence_devconf-2016

Devconf.cz, February 2016

oVirt and Gluster Hyperconvergence

Ramesh NachimuthuSoftware EngineerRed HatDevconf.cz, February 2016


Agenda

● Architecture of oVirt● Hosted Engine● Gluster in a nutshell● Hyper converged infrastructure

● How it looks● Setup● Management

● Recent enhancements● Conclusion


Ovirt Architecture

oVirt is a virtualization platformto manage virtual machines,storage and networks

● Engine (ovirt-engine)Manages the oVirt hosts, and allows system administrators to create and deploy new Vms

● Host Agent (VDSM)oVirt engine communicates with VSDM to manage the Vms, storages and networks

Clu

ster

s

Storage ServerSto

rage

Eng

ine

Storage Server

Cluster Hosts


Ovirt Storage Architecture

● Storage Domain● Centralized storage system

(images, templates, etc.)

● A standalone storage entity

● Stores the images and associated

Metadata

● Only real persistent storage for

VDSM

● Used for synchronization (sanlock)

● Storage Types● NFS, FCP, iSCSI

● Glusterfs

Clu

ster

s

Storage DomainSto

rage

Eng

ine

Storage Domain

VDSM Hosts


Hosted Engine

● Ovirt Engine running inside a

VM

● Highly Available

● Can be migrated to a

different host

● Special Agent for Monitoring

● Bootstrap deployment

neededC

lust

ers

Storage Domain

Sto

rage

Poo

lE

ngin

e

Storage Domain

VDSM Hosts


GlusterFS Architecture

● GlusterFS is a general purpose scale-out distributed file-system

supporting thousands of clients

● Aggregates storage exports over network interconnect to provide

a single unified namespace

● File-system completely in

userspace, runs on

commodity hardware

● Layered on disk file

systems that support

extended attributes


GlusterFS Bricks

● A brick is an export directory located on a specific node

(e.g. host-01:/srv/fs1/brick1)

● Each brick inherits limits of the underlying file-system

● No limit on the number bricks per node (as best-practice each

brick in a cluster should be of the same size)

/srv/fs1/brick1 /srv/fs1/brick2/srv/fs1/brick2

File-System: /srv/fs1 /srv/fs2

Block Device: /dev/dm-1 /dev/dm-2


GlusterFS Volumes

● A volume is a logical collection of bricks

● The mountable entity

● Different types of Volumes

● Distribute, Stripe, Replicate (+ combinations), Quorum

● Type of a volume is specified at the time of volume creation

and determines how and where data is placed

● Accessible via FUSE, NFS, SMB, libgfapi

host01:/srv/fs1/brick1

Volume1

host02:/srv/fs1/brick1 host03:/srv/fs1/brick1

Distribute/Stripe/Replicate


Fuse Access

QEMU

Kernel VFS /dev/fuse

Kernel

Userspace

GlusterFS

Net

File-System

Gluster Brick

NetKernel

Userspace

Client Server

FUSE Access

● More context switches


QEMU libgfapi Support

● GlusterFS exposes APIs for accessing Gluster volumes

● Reduces context switches

QEMU

Kernel

Userspace

Net

File-System

Gluster Brick

NetKernel

Userspace

Client Server

libgfapi Access

But see: https://bugzilla.redhat.com/show_bug.cgi?id=1247933


Hyperconverged oVirt – GlusterFS

● The Data Center nodes are used both for virtualization and serving replicated images from the GlusterFS Bricks

● Engine runs inside a VM (Hosted Engine)

● The boxes can be standardized (hardware and deployment) for easy addition and replacement

● Support for both scaling up, adding more disks, and scaling out, adding more hosts

VM

s an

d S

tora

ge w

ith E

ngin

e in

side

a V

M

GlusterFS Volume

Bricks Bricks Bricks

Experimental?


Improvements needed for HC

● GlusterFS support re-added to hosted-engine setup● With libgfapi and replica 3 volume

● Ovirt-Engine Appliance● Preconfigured VM with oVirt Engine● Cloud-init based customization

● Shared configuration● All nodes see the same configuration data● Upgrade path from oVirt 3.5

● Management GUI for the oVirt-engine VM and HE


HC ingredients

● at least 3 virtualization capable hosts (CentOS 7.1+)● 10 GB of temporary space on the primary host● two separate partitions for data (20GB+) on all hosts● DHCP configured to map a MAC address to a fixed IP● DNS configured with A and PTR names for the IP● oVirt release package installed on all hosts

http://resources.ovirt.org/pub/yum-repo/ovirt-release36.rpm

● Physical console on the primary host or network access and screen package installed

http://resources.ovirt.org/pub/yum-repo/ovirt-release36.rpm


HC Setup - Gluster Volume

● 3 Node Gluster cluster● 2 Volumes with 3 way replication

● One for Hosted Engine VM and other for regular VMs● Enable Cluster Quorum● Sharding for better self-heal and geo-rep performace● Few more tunings

● network.ping-timeout = 10● cluster.data-self-heal-algorithm = full● performance.low-prio-threads = 32● Group = virt● storage.owner-uid, storage.owner-uid = 36


HC Setup – Hosted Engine

Packages Required● ovirt-engine-appliance● ovirt-hosted-engine-setup● Vdsm-gluster

hosted-engine –-deploy● 'glusterfs' storage type● OVA image for hosted-engine VM● Clound-init to customize the VM

● FQDN and Mac for hosted-engine VM

More details at http://blogs-ramesh.blogspot.in/2016/01/ovirt-and-gluster-hyperconvergence.html

http://blogs-ramesh.blogspot.in/2016/01/ovirt-and-gluster-hyperconvergence.html


HC Setup – Final Touch

● Hosted Engine running with single node Cluster● Enable Gluster service● Add remaining nodes● Create and add the main storage domain


Enabling Gluster

● Gluster Service support is located in the Cluster properties

● Deploy Hosts with GlusterFS Server support

● Enable Bricks and Volume Management from oVirt WebAdmin and REST-API

● Engine is not taking in consideration GlusterFS on Virtualization Power-Saving policies and Fencing yet


Adding addtional Nodes

● Simple checkbox during in the Add host dialog

● Host deploy script does everything else auto-magically

Removed at the last moment because of unresolved issues. It can be done through “hosted-engine –deploy” script


Adding Gluster storage

● Gluster volume management using WebAdmin Portal and REST-API

● Volume Pofiling and Capacity monitoring

● Geo Replication

● Volume Snapshot


Add Storage Domain

● Master Storage Domain using gluster volume

● Imports Hosted Engine Storage Domain and Hosted Engine VM autmotically.


Dashboard


Hosted engine VM management

● Support for editing the Hosted Engine VM

● Memory and CPU allocation, network configuration

● Work in progress..

● Distributed to all hosted engine nodes using OVF file on the storage domain

● Hosted engine daemons pick up the configuration when the management VM is restarted


Hosted engine management

● Reporting configuration

– State transitions

– SMTP details● Timeout configuration

– Allowed downtime before forced recovery● Host scoring constants and rules


Recent Enahacements

● Glusterfs Sharding

● Arbiter Volumes

● Replace Brick

● Backup vol file support in libvirt and QEMU-KVM

https://devconfcz2016.sched.org/event/5m1p/improvements-in-gluster-for-virtualization-usecase

https://devconfcz2016.sched.org/event/5m1p/improvements-in-gluster-for-virtualization-usecase


THANK YOU!

http://[email protected]@ovirt.org#ovirt irc.oftc.net

http://www.ovirt.org/Features/GlusterFS-Hyperconvergencehttp://blogs-ramesh.blogspot.in/2016/01/ovirt-and-gluster-hyperconvergence.html

http://www.ovirt.org/

mailto:[email protected]

mailto:[email protected]

http://www.ovirt.org/Features/GlusterFS-Hyperconvergence

http://blogs-ramesh.blogspot.in/2016/01/ovirt-and-gluster-hyperconvergence.html