table of contents - docs.pivotal.iodocs.pivotal.io/archives/mysql-docs-1.7.pdf... , and service...

129

23262730313435394045535557

TableofContents

TableofContentsMySQLforPivotalCloudFoundryReleaseNotesKnownIssuesFrequentlyAskedQuestionsClusterScaling,NodeFailure,andQuorumClusterConfigurationProxyforMySQLforPivotalCloudFoundryCreatingApplicationSecurityGroupsforMySQLMonitoringtheMySQLServiceDeterminingClusterStateBootstrappingaGaleraClusterBackingUpMySQLforPivotalCloudFoundryScalingDownMySQLRotatingMySQLforPCFCredentialsRunningmysql-diag

©CopyrightPivotalSoftwareInc,2013-2017 1of59 1.7

MySQL for Pivotal Cloud FoundryThisisdocumentationfortheMySQLforPivotalCloudFoundry (PCF)tile.

ProductSnapshotCurrentMySQLforPCFDetails

Version:v1.7.32

Release Date:August11,2017

Software component versions:MariaDBv10.1.18,Galerav25.3.17

Compatible Ops Manager version(s):v1.6.xthroughv1.10.x

Compatible Elastic Runtime version(s):v1.6.xthroughv1.10.x

vSphere support?Yes

AWS support?Yes

OpenStack support?Yes

IPsec support?Yes

UpgradingtotheLatestVersionConsiderthefollowingcompatibilityinformationbeforeupgradingMySQLforPCF.

Formoreinformation,seethefullProductCompatibilityMatrix .

Ops Manager VersionSupported Upgrades from Imported MySQL InstallationFrom To

v1.3.xv1.2 v1.3

v1.3.2 v1.4.0

v1.4.x and v1.5.x v1.3.2v1.4.0

v1.5.0

v1.4.x - v1.10.x

v1.4.0 v1.5.0

v1.5.0 v1.6.1–v1.6.26

v1.6.1–v1.6.25Nextv1.6.xrelease–v1.6.26

v1.7.0–v1.7.32

v1.7.0–v1.7.31 Nextv1.7.xrelease–v1.7.32

(*)Notethereisaknownissueupgradingsomereleasesofp-mysqlv1.6andv1.7onOpsManagerv1.6.

ReleaseNotesForinformationaboutchangesbetweenversionsofthisproduct,seetheReleaseNotes.

OverviewTheMySQLforPCFproductdeliversafullymanaged,“DatabaseasaService”toCloudFoundryusers.Wheninstalled,thetiledeploysandmaintainsasingleorthree-nodeclusterrunningarecentreleaseofMariaDB ,SQLProxiesforsuper-fastfailover,andServiceBrokersforCloudFoundryintegration.Weworkhardtoshiptheserviceconfiguredwithsanedefaults,followingtheprincipleofleastsurpriseforageneral-userelationaldatabaseservice.

MySQL for PCF 1.7 is no longer supported.Thesupportperiodforversion1.7hasexpired.Tostayuptodatewiththelatestsoftwareandsecurityupdates,pleaseplantoupdatetomorerecentreleasesofMySQLforPCF.

*

*


https://network.pivotal.io/products/pivotal-cf

http://docs.pivotal.io/compatibility-matrix.pdf

http://www.mariadb.org

Wheninstalled,developerscanattachadatabasetotheirapplicationsinaslittleastwocommands, cfcreate-service

and cfbind-service

.Connection

credentialsareautomaticallyprovidedinthestandardmanner .Developerscanselectfromamenuofserviceplansoptions,whichareconfiguredbytheplatformoperator.

Twoconfigurationsaresupported:

Single Highly Available

MySQL 1node 3-nodecluster

SQL Proxy 1node 2nodes

Service Broker 1node 2nodes

HighAvailability - Yes

Multi-AZSupport - Yes*

RollingUpgrades - Yes

AutomatedBackups Yes Yes

CustomizablePlans Yes Yes

CustomizableVMInstances Yes Yes

PlanMigrations Yes Yes

EncryptedCommunication Yes✝ Yes✝

EncryptedDataat-rest - -

Long-livedCanaries - -

(*)vSphereonly,v1.7andearlier(✝)RequiresIPSECBOSHplug-in

LimitationsSingleandthree-nodeclustersaretheonlysupportedtopologies.OpsManagerwillallowtheOperatortosetthenumberofinstancestoothervalues,onlyoneandthreeareadvised.Formoreinformation,seeAvoidanevennumberofnodesintheClusterScaling,NodeFailure,andQuorumtopic.

AlthoughtwoProxyinstancesaredeployedbydefault,thereisnoautomationtodirectclientsfromonetotheother.Toaddressthis,configurealoadbalancerasdescribedintheProxysection.

OnlytheInnoDBstorageengineissupported;itisthedefaultstorageenginefornewtables.Useofotherstorageengines(includingMyISAM)mayresultindataloss.

Alldatabasesaremanagedbyshared,multi-tenantserverprocesses.Althoughdataissecurelyisolatedbetweentenantsusinguniquecredentials,applicationperformancemaybeimpactedbynoisyneighbors.

Round-triplatencybetweendatabasenodesmustbelessthanfiveseconds;ifthelatencyishigherthanthis,nodeswillbecomepartitioned.Ifmorethanhalfofclusternodesarepartitioned,theclusterwilllosequorumandbecomeunusableuntilmanuallybootstrapped.

SeealsothelistofKnownLimitations inMariaDBcluster.

KnownIssuesForinformationaboutissuesincurrentreleasesofMySQLforPCF,seeKnownIssues.

Installation1. DownloadtheproductfilefromPivotalNetwork .

2. NavigatetotheOpsManagerInstallationDashboard.


https://docs.pivotal.io/pivotalcf/devguide/deploy-apps/environment-variable.html#VCAP-SERVICES

https://mariadb.com/kb/en/mariadb/mariadb-galera-cluster-known-limitations/

https://network.pivotal.io/products/p-mysql

3. ClickImport a ProducttouploadtheproductfiletoyourOpsManagerinstallation.

4. ClickAddnexttotheuploadedproductdescriptionintheAvailableProductsviewtoaddthisproducttoyourstagingarea.

5. ClickthenewlyaddedtiletoreviewconfigurableSettings.


6. ClickApply Changestodeploytheservice.

Settings

ServicePlanAsingleserviceplanenforcesquotasof100megabytesofstorageperdatabaseand40concurrentconnectionsperuserbydefault.UsersofOperationsManagercanconfiguretheseplanquotas.Changestoquotaswillapplytoallexistingdatabaseinstancesaswellasnewinstances.Incalculatingstorageutilization,indexesareincludedalongwithrawtabulardata.

Thenameoftheplanis100mb-devbydefaultandisautomaticallyupdatedifthestoragequotaismodified.Thus,ifthestoragequotaischangedto1024megabytes,thenewdefaultplannamewillbe1024mb-dev.

Note:Afterchangingaplan’sdefinition,allinstancesoftheplanmustbeupdated.Foreachplan,eithertheoperatorortheusermustruncfupdate-serviceSERVICE_INSTANCE-pNEW_PLAN_NAME

onthecommandline.

Further Note:ThisfeaturedoesnotworkproperlyinversionsofMySQLforPCFv1.6.3andearlier.SeetheentryinKnownIssuesfortherecommendedworkaround.

ProvisioningaserviceinstancefromthisplancreatesaMySQLdatabaseonamulti-tenantserver,suitablefordevelopmentworkloads.Bindingapplicationstotheinstancecreatesuniquecredentialsforeachapplicationtoaccessthedatabase.

ProxyTheproxytierisresponsibleforroutingconnectionsfromapplicationstohealthyMariaDBclusternodes,evenintheeventofnodefailure.

ApplicationsareprovidedwithahostnameorIPaddresstoreachadatabasemanagedbytheservice.Formoreinformation,seeApplicationBinding .Bydefault,theMySQLservicewillprovideboundapplicationswiththeIPofthefirstinstanceintheproxytier.Evenifadditionalproxyinstancesaredeployed,clientconnectionswillnotberoutedthroughthem.Thismeansthefirstproxyinstanceisasinglepointoffailure.

In order to eliminate the first proxy instance as a single point of failure, operators must configure a load balancer to route clientconnections to all proxy IPs, and configure the MySQL service to give bound applications a hostname or IP address that resolves tothe load balancer.

ConfiguringaLoadBalancer

Inolderversionsoftheproduct,applicationsweregiventheIPofthesingleMySQLserverinbindcredentials.Whenupgradingtov1.5.0,existingapplicationswillcontinuetofunction,but,totakeadvantageofhighavailabilityfeatures,theymustbereboundtoreceiveeithertheIPofthefirstproxyinstanceortheIP/hostnameofaloadbalancer.

InordertoconfigurealoadbalancerwiththeIPsoftheproxytierbeforev1.5.0isdeployedandpreventapplicationsfromobtainingtheIPofthefirstproxyinstance,theproductenablesanoperatortoconfiguretheIPsthatwillbeassignedtoproxyinstances.ThefollowinginstructionsappliestotheProxysettingspagefortheMySQLproductinOperationManager.

IntheProxy IPsfield,enteralistofIPaddressesthatshouldbeassignedtotheproxyinstances.TheseIPaddressesmustbeintheCIDRrangeconfiguredintheDirectortileandnotbecurrentlyallocatedtoanotherVM.LookattheStatuspagesofothertilestoseewhatIPaddressesareinuse.

IntheBinding Credentials Hostnamefield,enterthehostnameorIPaddressthatshouldbegiventoboundapplicationsforconnectingto


http://docs.pivotal.io/pivotalcf/devguide/services/index.html#application-binding

databasesmanagedbytheservice.ThishostnameorIPaddressshouldresolvetoyourloadbalancerandbeconsideredlong-lived.Whenthisfieldismodified,applicationsmustbereboundtoreceiveupdatedcredentials.

ConfigureyourloadbalancertorouteconnectionsforahostnameorIPtotheproxyIPs.Asproxyinstancesarenotsynchronized,werecommendconfiguringyourloadbalancertosendalltraffictooneproxyinstanceatatimeuntilitfails,thenfailovertoanotherproxyinstance.Formoreinformation,seeKnownIssues.

Important:Toconfigureyourloadbalancerwithahealthcheckormonitor,useTCPagainstport1936.Unauthenticatedhealthchecksagainstport3306willcausetheservicetobecomeunavailable,andwillrequiremanualinterventiontofix.

AddingaLoadBalancerafteranInitialDeploy

Ifv1.5.0isinitiallydeployedwithoutaloadbalancerandwithoutproxyIPsconfigured,aloadbalancercanbesetuplatertoremovetheproxyasasinglepointoffailure.However,thereareseveralimplicationstoconsider:

ApplicationswillhavetobereboundtoreceivethehostnameorIPthatresolvestotheloadbalancer.Torebind:unbindyourapplicationfromtheserviceinstance,binditagain,thenrestageyourapplication.Formoreinformation,seeManagingServiceInstanceswiththeCLI .Inordertoavoidunnecessaryrebinding,werecommendconfiguringaloadbalancerbeforedeployingv1.5.0.

InsteadofconfiguringtheproxyIPsinOperationsmanager,usetheIPsthatweredynamicallyassignedbylookingattheStatuspage.ConfigurationofproxyIPsaftertheproductisdeployedwithdynamicallyassignedIPsisnotwellsupported;seeKnownIssues.

ApplicationServiceGroupsYoumustcreateappropriateApplicationSecurityGroups (ASGs)fortheMySQLforPCFinorderforapplicationstohaveaccesstotheservice.

SeeCreatingApplicationSecurityGroupsforMySQLforinstructions.

LifecycleErrandsTwolifecycleerrandsarerunbydefault:thebroker registrarandthesmoke test.ThebrokerregistrarerrandregistersthebrokerwiththeCloudControllerandmakestheserviceplanpublic.Thesmoketesterrandrunsbasicteststovalidatethatserviceinstancescanbecreatedanddeleted,andthatapplicationspushedtoElasticRuntimecanbeboundandwritetoMySQLserviceinstances.BotherrandscanbeturnedonoroffontheLifecycleErrandspageundertheSettingstab.

ResourceConfig

InstanceCapacity

Anoperatorcanconfigurehowmanydatabaseinstancescanbeprovisioned(instancecapacity)byconfiguringtheamountofpersistentdiskallocatedtotheMySQLservernodes.Thebrokerwillprovisionarequesteddatabaseifthereissufficientunreservedpersistentdisk.ThiscanbemanagedusingthePersistentDiskfieldfortheMySQLServerjobintheResourceConfigsettingpageinOperationsManager.Notallpersistentdiskwillbeavailableforinstancecapacity;about2-3GBisreservedforserviceoperation.Addingnodestotheclusterincreasesdurability,notcapacity.Multiplebackendclusters,toincreasecapacityorforisolation,arenotyetsupported.

Indetermininghowmuchpersistentdisktomakeavailablefordatabases,operatorsshouldalsoconsiderthatMariaDBserversrequiresufficientCPU,RAM,andIOPStopromptlyrespondtoclientrequestsforalldatabases.

ProvisioningandBindingviaCloudFoundry

Note:WithoutASGs,theservicewillnotbeusable.

Note:Youmightalsonoticeabroker-deregistrarerrand.Do not run this errand unless instructed to do so by Support.Broker-deregistrarisapartoftheautomationusedbyOpsManagerwhiledeletingatile.Runningthiserrandunderanyothercircumstancewilldeleteuserdata.


http://docs.pivotal.io/pivotalcf/devguide/services/managing-services.html

http://docs.pivotal.io/pivotalcf/adminguide/app-sec-groups.html

AspartofinstallationtheproductisautomaticallyregisteredwithPivotalCloudFoundry ElasticRuntime(seeLifecycleErrands).Onsuccessfulinstallation,theMySQLserviceisavailabletoapplicationdevelopersintheServicesMarketplace,viatheweb-basedDeveloperConsoleor cf

marketplace.

Developerscanthenprovisioninstancesoftheserviceandbindthemtotheirapplications:

$cfcreate-servicep-mysql100mb-devmydb$cfbind-servicemyappmydb$cfrestartmyapp

Formoreinformationabouttheuseofservices,seetheServicesOverview .

ExampleApplicationTohelpapplicationdevelopersgetstartedwithMySQLforPCF,wehaveprovidedanexampleapplication,whichcanbedownloadedhere.InstructionscanbefoundintheincludedREADME.

ServiceInstanceDashboardCloudFoundryuserscanaccessadashboardforeachMySQLserviceinstancesviaSSOfromAppsManager.Thedashboarddisplayscurrentstorageutilizationofthedatabaseandtheplanquotaforstorage.OntheSpacepageinAppsManager,userswiththeSpaceDeveloperrolewillfindaManagelinknexttotheinstance.ClickingthislinkwilllogusersintotheservicedashboardviaSSO.

ConnecttoyourDatabasewiththeMySQLPluginYoucanusetheCloudFoundryCommandLineInterface(cfCLI)MySQLplugintoconnecttotheMySQLdatabasesusedbyyourCloudFoundryapps.Thepluginsupportsthefollowingactions:

Inspectingdatabasesfordebuggingpurposes.

Manuallyadjustingdatabaseschemaorcontentsindevelopmentenvironments.

Dumpingandrestoringdatabases.

Formoreinformation,seethecf-mysql-plugin repository.

ProxyDashboardTheserviceprovidesadashboardwhereadministratorscanobservehealthandmetricsforeachinstanceintheproxytier.Metricsincludethenumberofclientconnectionsroutedtoeachbackenddatabaseclusternode.

Thedashboardforeachproxyinstancecanbefoundat: http://proxy-<jobindex>.p-mysql.<system-domain>

.Jobindexstartsat0soifyouhavetwoproxy

instancesdeployedandyoursystem-domainis example.com ,dashboardswouldbeaccessibleat http://proxy-0.p-mysql.example.com andhttp://proxy-1.p-mysql.example.com .

Basicauthcredentialsarerequiredtoaccessthedashboard.ThesecanbefoundintheCredentialstaboftheMySQLproductinOperationsManager.

FormoreinformationaboutSwitchBoard,readtheproxydocumentation.

SeeAlsoClusterConfiguration

BackingUpMySQLNote:ForinformationaboutbackingupyourPCFinstallation,seeBackingUpandRestoringPivotalCloudFoundry .

DeterminingClusterState

ClusterScaling,NodeFailure,andQuorum

BootstrappingaCluster


https://network.pivotal.io/products/pivotal-cf

http://docs.pivotal.io/pivotalcf/devguide/services/

http://docs.pivotal.io/p-mysql/1-7/mysql-example-app.tgz

https://github.com/andreasf/cf-mysql-plugin

http://docs.pivotal.io/pivotalcf/customizing/backup-restore/index.html

ScalingDownMySQL


Release Notes

v1.7.32ReleaseDate:11August,2017

Change the Interruptor’s default setting to OFF.Forayear ,MySQLforPCFhasincludedtheInterruptor.It’saprotectivemechanismwhichstopsanodefromautomaticallyrejoiningtheclusterifdoingsomaydeleteapplicationdata.WealsoupgradedtoMariaDB10.1 andprovidedtheReplicationCanarytofurtherprotectapplicationdata.TherehavebeenzeroinstanceswheretheInterruptorhasbeenneededtoprotectapplicationdata.

Inthisrelease,wearedisablingtheInterruptorbecauseitisdisruptivetonormalclusterfunction,andrequiresmanualOperatoractiontorestoreavailability.WefeelconfidentthatdisablingtheInterruptorinallbutthemostcriticalenvironmentsisasafeandconvenientchoice.

IfyouwishtocontinueusingtheInterruptor,makesurethat“Preventnodeautore-join”ischeckedinthe“AdvancedOptions”configurationpane,thenhitApply Changes.

Upgradesseveraldependenciesincluding nokogiri1.8.0 , golang1.8.3 , xtrabackup2.4.5 , boost1.59.0 ,and python2.7.13

Updatedstemcellto3312.32.Thissecurityupgraderesolvesthefollowing:

USN-3265-2

Formoreinformation,seepivotal.io/security .

v1.7.31ReleaseDate:June22,2017

Newconfigurationpaneforsyslog:Previously,MySQLforPCFusedthesameconfigurationsettingsasElasticRuntime.However,someuserswanttosendMySQLforPCFlogstodestinationsotherthanElasticRuntimelogs.Thus,MySQLforPCFnowhasseparateconfiguration,similartoRabbitMQforPCFandRedisforPCF.Actionrequired:DuringinstallationorupgradeofMySQLforPCF,youmustconfigureordisablesysloggingintheSyslogsettingspane.


USN-3334-1 .


v1.7.30ReleaseDate:June2,2017


USN-3291-3


v1.7.29ReleaseDate:May19,2017

BugfixestoaddressissueswithMySQLforPCFwhentheIPsecadd-on isalsoinstalled:

Bug fix:WhileinstallingMySQLforPCFwithIPsecinstalled,theproductmightfailtodeploy.ThismaybeduetoanissuewherethedefaultprobetimeoutistoolongwhilerunningunderIPsec,andshouldbereduced.Version1.7.29ofMySQLforPCFallowsyoutoreducetheNewClusterProbe

This is the last planned release of MySQL for PCF 1.7.Thesupportperiodforversion1.7hasexpired.Tostayuptodatewiththelatestsoftwareandsecurityupdates,pleaseplantoupgradetomorerecentreleasesofMySQLforPCF.


https://mariadb.com/kb/en/mariadb/what-is-mariadb-101/

http://www.ubuntu.com/usn/USN-3265-2/

https://pivotal.io/security





https://docs.pivotal.io/addon-ipsec/

TimeoutintheMySQLserverconfigurationpage.Formoreinformation,seeOptionsandFeatures inthev1.8documentation.Bug fix:WealsomadeasmallchangeinthewaythatMySQLnodesshutdown,whichshouldbetterallownodestoleavetheclustergracefullywhileIPsecisinstalled.

v1.7.28ReleaseDate:April27,2017


USN-3265-2

Bug fix:AddressedanissuewherebackupswereunabletostorebackupsonAWSS3regionsthatrequirethev4signature.

Bug fix:Addressedanissuewherenodesmayfailtorejointheclusterafterrestart.SeetheRejoinUnsafeFailsKnownIssueformoredetails.

Note:Thetitleofthetilenowappearsas“MySQLforPCF,”notsimply“MySQL.”

Additionalinformationcanbefoundathttps://pivotal.io/security

v1.7.27ReleaseDate:2017April3

Seebelow,sameupdateasv1.6.26

v1.6.26ReleaseDate:2017April3

Updatednokogiritov1.7.1.Thisisasecurityupgradethatresolvesthefollowing:

USN-3235-1

Updatedstemcellto3263.22.Thisisasecurityupgradethatresolvesthefollowing:

USN-3249-2


v1.7.26ReleaseDate:2017March23

UpdateddependencyofGolanguagetov1.7.Thischangedoesnotimpactfunctionality,itisdoneonlytokeepuptodatewithsupportedversionsofGo.


Bug fix:Changedthevalueof wsrep_max_ws_rows to0topreventMariaDBbugMDEV-11817 fromaffectingDDLs.




https://docs.pivotal.io/p-mysql/1-8/index.html#options_and_features



https://www.ubuntu.com/usn/usn-3235-1/



https://jira.mariadb.org/browse/MDEV-11817

Updatedstemcellto3263.21.Thisisasecurityupgradethatresolvesthefollowing:

USN-3220-2


v1.7.24ReleaseDate:2017February24



Updatedstemcelltov3263.20toresolvethefollowing:

USN-3208-2


v1.7.23ReleaseDate:26January2017



Updatedstemcelltov3263.17,whichisaroutinepatchupdatetoaddressmediumandlowsecurityvulnerabilities.


v1.7.22ReleaseDate:21December2016

Resolvesanissuewhichpreventsupgradingpreviousinstallationsofp-mysqltov1.7.19throughv1.7.22onOpsManagerv1.6andearlier.


Updatedstemcelltov3263.14.

Resolvesanissuewhichpreventsupgradingpreviousinstallationsofp-mysqltov1.6.20andv1.6.21onOpsManagerv1.6andearlier.






https://www.ubuntu.com/usn/USN-3208-2/



USN-3156-1



USN-3156-1






USN-3151-2


v1.7.19ReleaseDate:14November2016

Updatedstemcelltov3233.4toaddressstandardsecurityupdates.

UpdatedMariaDBtov10.1.18toresolveavarietyofunspecifiedsecurityvulnerabilities.

Updatedmysql-backup-releasetov1.27.3.Whenbackupsaredisabled,thebackupserveroneachnodeisalsodisabled.

v1.7.18ReleaseDate:26October2016

Updatedstemcelltov3233.3,sameupdateasv1.6.19.

Security:Updatetheservicebrokertopreventloggingofservicecredentials.


UpdatedMariaDBtov10.1.18toresolveavarietyofunspecifiedsecurityvulnerabilities.

Updatedstemcelltov3233.3.Thisisasecurityupgradethatresolvesthefollowing:

USN-3106-2














USN-3099-2



Includesstabilityandbugfixes.





USN-3087-2

UpgradestheLinuxv3.19kerneltov4.4.


v1.7.14ReleaseDate:28September2016

Seebelow,sameupdateasv1.6.16.



USN-3087-2










Note:UpdatedMariaDBtov10.1.17.Theupgradeisautomatic,andifdeployedinHAconfigurationwillnotcausedowntimeforapplications.

Updatedstemcelltov3232.19.Thisisaroutinesecurityupgradethatresolveslowandmediumvulnerabilities.

UpdatedRubyandRailssoftwaretoadditionalresolvesecurityvulnerabilities.

Bug fix:Introducedafixtothereplicationcanarywhichreducesthepossibilityoffalsepositives.


Updatedstemcelltov3232.19.Thisisaroutinesecurityupgradethatresolveslowandmediumvulnerabilities.

UpdatedRubyandRailssoftwaretoadditionalresolvesecurityvulnerabilities.


v1.7.12Seebelow,sameupdateasv1.6.14.

Bug fix:AddressesabuginOpsManagerv1.7.0,inwhichupgradingtoarecentversionofPivotalMySQLcausesOpsManagertoissueaninternalservererror.

v1.6.14ReleaseDate:24August2016


USN-3064-1

USN-3048-1

USN-3060-1

USN-3061-1

USN-3065-1


v1.7.11ReleaseDate:27July2016

We’vediscoveredarareconditionwhereaMySQLclusterexperiencesafaultinreplicationthatcanresultinsomedataloss.Whenthisoccurs,previousreleasesdonotlogtherootcauseofthebug.Inordertobestaddressthisissue,v1.7.11containssignificantadditionaltelemetryandseveraldefensivefeatureswhichwillaccountforthefailureconditionandpreventdataloss.

Introducing the Replication CanaryWe’veincludedanewlong-runningmonitor,theReplication Canary.TheReplicationCanarycontinuallymonitorstheMySQLcluster,watchingforinstancesinwhichcross-clusterreplicationhasfailed.Itisenabledbydefault,andrequiresane-mailaddressintheAdvancedOptionsconfigurationpane.

If any of these protections activate, it is critical that you contact Pivotal support immediately.Supportwillworkwithyoutodeterminethenatureofthecluster’sfailure,andadviseasuggestedresolution.Additionally,contactingSupportwillprovideuswithevidencethatwillenableustoidentifyandaddresstherootcauseinthefuture.









Intheeventthatreplicationhasfailed,theCanaryperformstwoactions:

E-mailtheOperator:PartoftheReplicationCanary’sconfigurationisane-mailaddress,whichcanbedirectedtoanyOperatore-mailaddress,oranescalationsystemsimilartoPagerDuty.DenyAccess:Whenreplicationhasfailed,theReplicationCanarywillautomaticallydisableuserandapplications’abilitytoaccesstheclusterviatheProxies.

Youmustsetthe Monitoring jobto1intheResourceConfigpane,ortheReplicationCanarywillnotbeenabled,regardlessofconfiguration.YoumustalsoconfirmthattheElasticRuntimetileisproperlyconfiguredtosende-mail.ThesesettingsarenecessaryforanystandardCloudFoundryconfiguration.

Ensurethatthe Notifications errandhasbeenenabled.Ensurethat SMTP Config hasbeenproperlyconfigured.

Ifeitherofthesearenotset,configureandApply Changesbeforedeployingv1.7.11.

FormoreinformationabouttheReplicationCanary,seethemonitoringdocumentation.

Introducing the InterruptorTheMySQLnodeshavenewlogicthat,whenenabled,willpreventanodefromre-joiningaclusterundercertainconditions.Thisisasecondlevelofprotectionagainstthepossibilityofdataloss.

FormoreinformationabouttheInterruptor,seethemonitoringdocumentation.

New feature:BackingupallnodesIntheBackupsconfigurationpane,there’snowanoptiontotakebackupsfromallMySQLnodes.Thisfeatureprotectsyourusersfromdatalossinthecasethatsomenodeshavedifferentdatathantheothers.

LoggingChanges

MySQLjoblogsarekeptlocalontheVM,inadditiontosenttosyslogifconfigured.Binarylogsarenowenabledandrotatedautomaticallybythesystem.Plusahostofdebuglogchangeshavebeenaddedtoaidindiagnosisefforts.

XATransactionsarenowdisallowed.

XATransactionsarenotcompatiblewithourHAtechnology.

QuotaEnforcerisnowconfigurable.

Maximumopenfiledescriptorsnowdefaultto65Kforlargedatabases.

Security fix:NowincludesMariaDB10.0.23

Avoidsapossiblecredentialleak.

v1.7.10Seebelow,sameupdateasv1.6.13



USN-3020-1

Additionalinformationcanbefoundathttps://pivotal.io/security .

v1.7.9

Note:Duetotheseriousnatureofafailureinreplication,bothbehaviorsareenabledbydefault.Duringconfiguration,youmayelecttosettheReplicationCanarytonotify-onlymode,butthisisnotrecommended.


http://www.ubuntu.com/usn/usn-3020-1/



v1.6.12ReleaseDate:16June2016


USN-3001-1



v1.6.11ReleaseDate:18May2016

Updatedstemcelltov3232.4Thisisasecurityupgradethatresolvesthefollowing:

USN-2977-1


v1.6.10ReleaseDate:06May2016

Updatedstemcelltov3146.11Thisisasecurityupgradethatresolvesthefollowing:

USN-2959-1

Bug fix:UpdatedacceptanceteststopassonPCFv1.7.

Bug fix:Update broker-registrar toavoidrunawayCPUconditiononbrokerVMs.





USN-2929-1













USN-2910-1




USN-2900-1 ,acriticalGNUClib(glibc)CVEUSN-2897-1

USN-2896-1





USN-2882-1

USN-2879-1

USN-2875-1

USN-2874-1

USN-2871-1

USN-2868-1

USN-2865-1

USN-2861-1



v1.6.5



http://www.ubuntu.com/usn/usn-2900-1













ReleaseDate:18January2016


USN-2869-1

CVE-2016-0715 .




Updatedstemcelltov3146.2.ThisisasecurityupgradethatresolvesthefollowingUbuntuSecurityNotices:

USN-2857-1,USN-2842-1,USN-2842-2,USN-2836-1,USN-2834-1,USN-2830-1,andUSN-2829-1

v1.7.0.4Seebelow,sameupdateasv1.6.3.4

v1.6.3.4ReleaseDate:04December2015

Addressesanissuewherechangingthemaximumnumberofallowedconnectionsintheserviceplandoesnotaffectthemaximumnumberofallowedconnectionsinserviceinstances,neworexisting.NotethattheKnownIssueforChangingServicePlanDefinitionstillapplies;you’llstillneedtorunthemanualworkaroundforexistinginstances.PleaselookforimprovementsinafuturereleaseofMySQLforPivotalCloudFoundry(PCF),wearesorryfortheinconvenience.

Updatedstemcelltov3146.ThisisasecurityupgradethatresolvesthefollowingUbuntuSecurityNotices:

[USN-2821-1 ]GnuTLSvulnerability


v1.6.3.3ReleaseDate:02December2015

Updatedstemcelltov3144.ThisisaregularsecurityupgradethatresolvesthefollowingUbuntuSecurityNotices:

[USN-2815-1 ]libpngvulnerabilities[USN-2812-1 ]libxml2vulnerabilities[USN-2810-1 ]Kerberosvulnerabilities




https://pivotal.io/security/cve-2016-0715






v1.6.3.2ReleaseDate:11November2015

Updatedstemcelltov3130.Thisisaregularsecurityupgradethatresolvesthefollowingissues:

[USN-2806-1 ]Linuxkernel(VividHWE)vulnerability[USN-2798-1 ]Linuxkernel(VividHWE)vulnerabilities


v1.6.3.1ReleaseDate:03November2015


[USN-2778-1 ]Linuxkernel(VividHWE)vulnerabilities


New Feature:AutomatedOperator-configureddatabasebackupsforDisasterRecovery.

UpdatedMariaDBtov10.0.21 whichalsoincludesupdatesfromMariaDBv10.0.20 .

Updatedstemcelltov3100.

Security:FixesforCVE-2015-3900,aman-in-the-middlerubygemsvulnerability.

Bugfix:SwitchboardfailstofindrecreatedmysqlnodewhenARPcachelockedbyhangingSYN_SENT.

Bugfix:EveryinstanceofSwitchboardregistrarstheroute proxy-0.p-mysql. ratherthanchangingbasedonAZindex.

Bugfix:MySQLforPCFnowsupportstheElasticRuntimesettingtorestrictHAProxytraffictoHTTPSonly.UsersofMySQLforPCFv1.6andearliermustupgradetoMySQLforPCFv1.7orlaterinordertousethisfeature.

NowhonorsOpenIDConnectIDtokenswheninteractingwithUAA.

Thiswillpreventblankpageswhenclicking‘Manage’inAppsManagerwithmorerecentversionsofElasticRuntime.



[USN-2765-1 ]Linuxkernel(VividHWE)vulnerability



[USN-2694-1]PCREvulnerabilities[USN-2698-1]SQLitevulnerabilities[USN-2710-1]OpenSSHvulnerabilities[USN-2710-2]OpenSSHregression




http://people.canonical.com/~ubuntu-security/cve/2015/CVE-2015-5156.html

https://mariadb.com/kb/en/mariadb/mariadb-10021-release-notes/

https://mariadb.com/kb/en/mariadb/mariadb-10020-release-notes/


[USN-2718-1]Linuxkernel(VividHWE)vulnerability

KnownIssuesExperimentalfeatureHTTPStraffictoHAProxydoesnotwork;itwillbefixedinanupcomingrelease.


Updatedstemcelltov3026toresolveCVE-2015-3290

v1.6.0NowincludesMariaDBv10.0.19andGalerav5.5.43(releasenotes )

IncludesseveraldefaultconfigurationchangestobettermanageMariaDB’smemoryanddiskusageduringperiodsofheavyuse.

Improved stability:Thisversionincludesanall-newQuotaEnforcerforenhancedstabilityandinpreparationfornewfeaturesinfuturereleases.

Improved stability:Nowprovidesgreaterstabilityduringclusterrecoverybyusingthextrabackup-v2replicationmechanism.

UpdatestobothServiceandProxydashboardstosupporttheexperimentalHTTPS-onlyfeatureinElasticRuntimev1.5

NowusestheMariaDBconnectorratherthanadditionallyincludingtheMySQLconnector.

Security:TheMySQLdeploymentnowrunsasuservcap,notroot.

Security:UpgradedRubyandRailscomponentstoaddressvariousCVEs.

Bug fix:Onceoverquota,writeprivilegesarenotrestoredbydroppingalltables.

Bug fix:Thebroker-deregistrarerrandnowsucceedsevenwhenaMySQLserviceisbroken.

Bug fix:ServiceBrokerdashboardshouldnotreturn500ifOAuthaccesstokenexpires.

Upgrade support:Thisproductcanbeautomaticallyupgradedfromv1.5.0

Documentationnowincludesseveralnewsections:

NotesonclusterconfigurationDeterminingMySQLclusterstateBackgroundonClusterScaling,NodeFailure,andQuorumBootstrappinganailingMySQLcluster

Note:BOSHStemcellv3026isrequired;thisstemcellisprovidedbyOpsManagerv1.5.1.


AWS support:TheclustereddatabaseservicecannowbedeployedonAmazonWebServicesfromtheOperationsManagerWebUI.

DeploymentislimitedtoasingleAvailabilityZone.Lookformulti-AZinfuturereleases.SingleavailabilityzoneisalimitationonAWS.OperationsManageronvSpherecontinuestosupportdeploymenttomultipleavailabilityzones.ThedefaultinstancetypefortheclusternodesonAWSism3.large.AlljobsaredeployedwithSSDforephemeralandpersistentdisk.

IaaS agnostic

ThesameproductcanbedeployedtobothAWSandvSpherePrecompiledpackagesarenolongerincludedMySQLforPCFv1.5.0requiresOpsManagerv1.4.0

New proxy tier

Improvedavailability:Wehaveentirelyre-writtentheproxytoeliminatesituationswhereclientscouldhangwhenaclusternodewasunhealthy.


https://mariadb.com/kb/en/mariadb-galera-cluster-10019-release-notes/

Adashboardthatclearlydisplaysnodehealthinrealtime

Upgrade support:Thisproductcanbeautomaticallyupgradedfromv1.3.2orv1.4.0

Cluster node resources increased for vSphere:Thedefaultresourcesarenow4GBRAM,2CPU,10GBpersistentdisk

Faster compilation:DefaultresourceforthecompilationjobsonvSpherearenow4GBRAM,4CPU,20GBpersistentdisk

Bug fix:Fixbroker-deregistrarerrandtosucceedevenwhenMySQLserviceisbroken

Bug fix:Quotaenforcercouldfailwhenbrokerhasn’tfinishedinitializing

Known issues:

OnAWS,thisversionsupportsdeploymentsintheUS-Eastregion.Multi-regionsupportiscominginafuturerelease.

TheexperimentalHTTPS-onlyfeatureinElasticRuntimev1.5maycauseissueswiththisversionoftheproduct.FullsupportforHTTPS-onlytrafficiscominginafuturerelease.

Note:BOSHStemcellv2865.1isrequiredforinstallationonOpsManagerv1.5.xandabove.


High Availability:databaseserverisnowclusteredandsynchronouslyreplicatedusingMariaDBGaleraCluster.Acopyofeachdatabaseresidesonallclusternodes,andwritestoanydatabasearereplicatedtoallcopies.Allclientconnectionsareroutedtoaprimaryclusternode,andintheeventofanodefailuretheproxytiermanagesfailover,routingclientconnectionstoahealthyclusternode.MySQLserver,proxy,andbrokerjobscanallbescaledouthorizontallyforincreasedavailability,eliminatingsinglepointsoffailure.

Improved logging and monitoring:route-registrationonthebrokerisnowanindependentprocess

Bug fix:calculationofstorageutilizationforthepurposesofquotaenforcementwhenmultipleappsarebound

Bug fix:formatofjdbcUrlconnectionstring(foundinVCAP_SERVICESonbind)

NotesonHighAvailabilityWhenupgradingfromanolderversion,applicationsmustbereboundtotakeadvantageofhighavailabilityfeatures.Torebind:unbindyourapplicationfromtheserviceinstance,binditagain,thenrestageyourapplication.FormoreinformationseeManagingServiceInstanceswiththeCLI.

Eliminationoftheproxyasasinglepointoffailurerequiresconfigurationofanexternalloadbalancertorouteconnectionstoproxyinstances.Fordetails,seeProxySettings.

SeeKnownIssues.


Updated stemcell addresses bash-shellshock vulnerabilities:resolvesCVEsdiscussedhere andhere .

v1.3.0Syslog forwarding:SyslogsarenowstreamedtothesamehostandportconfiguredinElasticRuntimesettings

Dynamic instance capacity management:Previouslyoperatorshadtomanuallyconfigurethemaximumnumberofserviceinstancespermittedbytheserver.Thisrequiredmanualcalculationandaknowledgeofrequiredsystemheadroom.Adminscannowmanageinstancecapacitysimplybyadjustingpersistentdiskallocatedtomysqlnodes.Remaininginstancecapacityisdetermineddynamicallybysubtractingasafeestimateforsystemheadroomandreservedstorageforprovisionedinstances.

Trusty stemcell:ServerandbrokerarenowdeployedonUbuntu“Trusty”v14.04LTSstemcells,providingimprovedsecurity,performance,andasmallerresourcefootprint.

Least necessary privileges:TheMySQLservicedashboardusesanew,limitedpermissionOAuthscopetodeterminewhetherausercurrentlyhasaccesstoaserviceinstance.Thedashboardnolongerhasfullreadaccesstoauser’saccount.


http://docs.pivotal.io/pivotalcf/devguide/services/managing-services.html

http://docs.pivotal.io/p-mysql/1-7/#proxy

http://www.pivotal.io/security/CVE-2014-6271

http://www.pivotal.io/security/CVE-2014-7186

Precompiled packages:Mostpackageshavebeenprecompiledforthetargetedstemcell.Thiswilllowerinitialdeploymenttimes,atthecostofalargerdownload.

v1.2.0Productrenamedto'MySQLforPivotalCF’

Planattributesareconfigurable:maxstorageperdatabase,maxconcurrentconnectionsperuser,andmaxdatabases

Plannameisdetermineddynamicallybasedonconfiguredstoragequota

Planfeaturesincludedisclaimerthattheserviceisnotforproductionuse

DeveloperscanSSOtoaservicedashboardthatdisplaysstorageutilization

SecurityfixesincludingupdatestoRails

ServicebrokerisregisteredbyURL(ratherthanbyIP).Typicallyhastheformat https://p-mysql.<cf-domain> .

Lifecycleerrandsareusedtoregisterthebrokerandrunteststhatverifythedeployment.

Improvedlogginginservicebroker

Thefollowingcomponentswillbere-deployed:cf-mysql-broker

mysql

Newcomponents:broker-registrar

broker-deregistrar

acceptance-tests

v1.1.0Updatedtheformatofmetadatafieldsinthebrokercatalogendpointandaddedadditionalfields.Formoreinformation,seeCatalogMetadata.

UpdatedRubytov2.0.0p353tofixavulnerabilityinv1.9.3p448.

Requeststodeleteaserviceinstanceorbindingnowgeta200responsewithanemptyJSONbodyinsteadofa204.

Thebrokernowreturnsaclearerrorwhenthereisnomorecapacityforadditionalinstancesduringaprovisionrequest.Theresponsehasstatuscode507 .Theuser-facingerrormessageis“Serviceplancapacityhasbeenreached.”

Thefollowingcomponentswillbere-deployed:cf-mysql-broker

mysql


Known Issues

UnabletoUpgradefromPreviousVersionsonOpsManagerv1.6MySQLforPivotalCloudFoundry(PCF)v1.6.20throughv1.6.21andv1.7.19throughv1.7.21arenotabletoupgradefrompreviousdeploymentsofMySQLonOpsManagerv1.6andearlier.Thishasbeencorrectedinv1.6.22andv1.7.22.

ThisisasampleerrorfromOpsManagerwhenattemptingtoupgradetooneoftheaffectedversions:

Product'MySQLforPivotalCloudFoundry'couldnotbeupgradedfrom'1.7.11'to'1.7.21'.PleasecontactyourPivotalrepresentative.

MySQLBackupstoAWSS3LimitedtoStandardRegionInMySQLforPCFv1.7,backupsareonlysenttoAWSS3bucketsthathavebeencreatedintheUSStandard region,“us-east-1.”Thislimitationhasbeenresolvedinv1.8.0-Edge.2andlater.

ElasticRuntimeHTTPS-onlyFeatureSupportfortheExperimentalHTTPS-onlyfeatureisbrokeninMySQLforPCFv1.6.xandearlier.TheHTTPS-onlyfeatureworksasdesignedinMySQLforPCFv1.7.0andlater.

AccidentalDeletionofaServicePlanIfandonlyiftheOperatordoesallofthesestepsinsequence,aplanwillbecome“unrecoverable”:

1. Clickthetrash-caniconintheServicePlanscreen.

2. Enteraplanwiththeexactsamename.

3. ClickSaveonthesamescreen.

4. ReturntotheOpsManagertop-level,andclickApply Changes.

AfterclickingApply Changes,thedeploywilleventuallyfailwiththeerror:

Servererror,statuscode:502,errorcode:270012,message:Servicebrokercatalogisinvalid:Plannamesmustbeuniquewithinaservice

Thisunfortunatesituationisunavoidable;aftertheOperatorhascommittedwithApply Changes,theoriginalplancannotberecovered.Foraslongasserviceinstancesofthatplanexist,youmaynotenteranewplanofthesamename.Atthispoint,theonlyworkaroundistocreateanewplanwiththesamespecifications,butspecifyadifferentname.Existinginstanceswillcontinuetoappearundertheoldplanname,butnewinstanceswillneedtobecreatedusingthenewplanname.

Ifyouhavecommittedsteps1and2,butnot4,noproblem.Donothitthe‘Save’button.SimplyreturntotheInstallationDashboard.Anyaccidentalchangeswillbediscarded.

Ifyouhavecommittedsteps1,2and3,donotclick'ApplyChanges.’Instead,returntotheInstallationDashboardandclickthe’Revert’button.Anyaccidentalchangeswillbediscarded.

ChangingServicePlanDefinitionInMySQLforPCFv1.7.0andearlier,thereisonlyoneserviceplan.Changingthedefinitionofthatplan,thenumberofmegabytes,numberofconnections,orboth,willmakeitsothatanynewserviceinstanceswillhavethosecharacteristics.

ThereisabuginMySQLforPCFv1.7andearlier.Changingtheplandoesnotchangeexistingserviceinstances.Existingplanswillcontinuetobegovernedbytheplanconstraintseffectiveatthetimetheywerecreated.Thisistrueregardlessofwhetherornotanoperatorruns cfupdate-service .


http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region

Thereisaworkaroundforthisbug,whichwillberesolvedinfuturereleasesofMySQLforPCF.Inorderforthechangetobeeffectiveforexistingplans,youmusttriggerthisbyinteractingdirectlywiththeservicebroker:curl-v-k-XPATCHhttps://BROKER_CREDS_USERNAME:[email protected]/v2/service_instances/SERVICE_INSTANCE_ID?plan_id=UNIQUE_ID

SYSTEM.DOMAINisdefinedinOpsManager,underElasticRuntime’sSettingstab,inthe Cloud Controller entry.

BROKER_CREDS_USERNAMEandBROKER_CREDS_PASSWORDaredefinedinOpsManager,underMySQLforPCF’sCredentialstab,intheBroker Auth Credentials entry.

TogettheUNIQUE_IDfortheserviceplan:1. Run cf curl /v2/services2. Intheoutput,findthedataobjectwiththe entity > label and description fortheservicewiththechangedplan,andrecordits

metadata > guid value.3. Run cf curl /v2/services/SERVICE-GUID/service_plans withtheserviceGUIDfromthelaststep.4. Intheoutput,findthedataobjectwiththe entity > label and description forthechangedserviceplan,andrecordits unique_id value.

ThisistheUNIQUE_IDvaluetopasstothe plan_id argumentabove;donotusetheplan’sGUID.

TogeteachSERVICE_INSTANCE_ID,run cf service INSTANCE --guid .Youshouldseeoutputlikethisexample:

$cfserviceacceptDB--guid4cae3a5e-66b1-4c9a-8536-feaff25237bf

Runthis curl commandforeachserviceinstancetobeupdated.

Furthermore,ifyouhavechangedthemaxnumberofconnectionsconstraint,thenyouneedtoupdateeachboundapplication’ssettingdirectlyfromtheMySQLconsole.Followthesesteps:

1. SSHintoyourOpsManagerDirectorusingtheseinstructions .

2. Run bosh deployments todiscoverthenameofyourMySQLforPCFdeployment.

3. Run bosh ssh usingyourMySQLforPCF’sdeploymentname.Example: bosh ssh mysql-partition-9d32f5601988152e869b/0

4. Run /var/vcap/packages/mariadb/bin/mysql -u root -p .Therootuser’spasswordisdefinedinOpsManager,underMySQLforPCF’sCredentialstab.

5. IssuethisMySQLcommand:UPDATE mysql.user SET mysql.user.max_user_connections=NEW_MAX_CONN_VALUE WHERE mysql.user.User NOT LIKE '%root%' ;

Makesuretochange NEW_MAX_CONN_VALUE towhatevernewsettingyou’vechosen.

6. exit;

ProxiesMayWritetoDifferentMySQLMastersAllproxyinstancesusethesamemethodtodetermineclusterhealth.However,certainconditionsmaycausetheproxyinstancestoroutetodifferentnodes,forexampleafterbriefclusternodefailures.

Thiscouldbeanissuefortablesthatreceivemanyconcurrentwrites.Multipleclientswritingtothesametablecouldobtainlocksonthesamerow,resultinginadeadlock.Onecommitwillsucceedandallotherswillfailandmustberetried.Thiscanbepreventedbyconfiguringyourloadbalancertorouteconnectionstoonly one proxy instance at a time.

NumberofProxyInstancesCannotbeReducedAftertheproductisdeployedwithoperator-configuredproxyIPs,thenumberofproxyinstancescannotbereduced,norcantheconfiguredIPsberemovedfromtheProxy IPsfield.IfinsteadtheproductisinitiallydeployedwithoutproxyIPs,IPsaddedtotheProxy IPsfieldwillonlybeusedwhenaddingadditionalproxyinstances,scalingdownisunpredictablypermitted,andthefirstproxyinstancecanneverbeassignedanoperator-configuredIP.

BackupsMetadataInMySQLforPCFv1.7.0,both compressed and encrypted showas N inthebackupmetadatafile.ThisisduetothefactthatMySQLforPCFimplementscompressionandencryptionoutsideofthetoolusedtogeneratethefile.Thisisaknowndefect,andwillbecorrectedinfuturereleases.


https://docs.pivotal.io/pivotalcf/customizing/trouble-advanced.html#prepare

MyISAMTablesTheclusteringpluginusedinthisrelease(Galera)doesnotsupportreplicationofMyISAMTables.However,theservicedoesnotpreventthecreationofMyISAMtables.WhenMyISAMtablesarecreated,thetableswillbecreatedoneverynode(DDLstatementsarereplicated),butdatawrittentoanodewon’tbereplicated.Ifthepersistentdiskislostonthenodewheredataiswrittento(forMyISAMtablesonly),datawillbelost.TochangeatablefromMyISAMtoInnoDB,followthisguide .

MaxUserConnectionsWhenupdatingthe max_user_connections propertyforanexistingplan,theconnectionscurrentlyopenwillnotbeaffected.Forexample,ifyouhavedecreasedfrom20to40,userswith40openconnectionswillkeepthemopen.Toforcethechangesuponuserswithopenconnections,anoperatorcanrestarttheproxyjob.Thiswillcausetheconnectionstoreconnectandstaywithinthelimit.Otherwise,ifanyconnectionabovethelimitisreset,itwon’tbeabletoreconnect,sothenumberofconnectionswilleventuallyconvergeonthenewlimit.

LongSSTTransfersWeprovidea database_startup_timeout inourmanifestwhichspecifieshowlongtowaitfortheinitialSSTtocomplete(defaultis150seconds).IftheSSTtakeslongerthanthisamountoftime,thejobwillreportasfailing.Versionsbefore cf-mysql-releasev23 haveaflawinourstartupscriptwhereitdoesnotkillthemysqldprocessinthiscase.Whenmonitrestartsthisprocess,itseesthatmysqlisstillrunningandexitswithoutwritinganewpidfile.Thismeansthejobwillcontinuetoreportasfailing.TheonlywaytofixthisistoSSHontothefailingnode,killthemysqldprocess,andre-run monitstart

mariadb_ctrl.

LoadBalancerTimeoutInterruptsLong-RunningQueriesTosomeloadbalancers,aconnectionthatwaitsforresultsappearstobeanidleconnection.Theselong-runningqueriesmaybeinterruptediftheyexceedtheidletimeoutoftheloadbalancer.Thefollowingerroristypicalofsuchaninterruption:

LostconnectiontoMySQLserverduringquery

Forexample,theAWSElasticLoadBalancer hasadefaultidletimeoutof60seconds.Ifaquerytakeslongerthanthisduration,theELBseverstheMySQLconnectionandreturnsanerror.

Topreventthesetimeouts,increasetheidletimeoutdurationaccordingly.


http://dev.mysql.com/doc/refman/5.5/en/converting-tables-to-innodb.html

http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/config-idle-timeout.html

Frequently Asked Questions

ManyreplicationerrorsinthelogsIseelotsofreplicationerrorsinmylogs!Istheclusterbroken?

UnlesstheGRAfilesshowaclearexecutionerror(e.g.,outofdiskspace)thisisanormalbehavior,andit’snothingtoworryabout.Wewillbeworkingonmoreadvancedmonitoringtodetectthefailurecase,andalertOperatorsinthefuture.

Occasionally,you’llseereplicationerrorsintheMySQLlogsthatwilllooksomethinglikethis:1603189:25:16[Warning]WSREP:RBRevent1Queryapplywarning:1,169924561603189:25:16[Warning]WSREP:IgnoringerrorforTOisolatedaction:source:abcd1234-abcd-1234-abcd-1234abcd1234version:3local:0state:APPLYINGflags:65conn_id:246804trx_id:-1seqnos(l:865022,g:16992456,s:16992455,d:16992455,ts:2530660989030983)1603189:25:16[ERROR]SlaveSQL:Error'Duplicatecolumnname'number''onquery.Defaultdatabase:'cf_0123456_1234_abcd_1234_abcd1234abcd'.Query:'ALTERTABLE...'

Whatthisissayingisthatsomeone(probablyanapp)issuedan“ALTERTABLE”commandthatfailedtoapplytothecurrentschema.Moreoftenthannot,thisisusererror.

ThenodethatreceivestherequestprocessesitasanyMySQLserverwill,ifitfails,itjustspitsthatfailurebacktotheapp,andtheappneedstodecidewhattodonext.Thatpartisnormal.HOWEVER,inaGaleracluster,allDDLisreplicated,andallreplicationfailuresarelogged.Sointhiscase,thebadALTERTABLEcommandwillberunbybothslavenodes,andifitfails,thoseslavenodeswilllogitasa“replicationfailure”sincetheycan’ttellthedifference.

It’sreallyhardtogetavalidDDLtoworkonsomenodes,yetfailonothers.Usuallythosecasesarelimitedtooutofdiskspaceorworkingmemory.Wehaven’tduplicatedthatyet.

ButIfoundablogarticlethatsuggeststhattheschematacangetoutofsync?

https://www.percona.com/blog/2014/07/21/a-schema-change-inconsistency-with-galera-cluster-for-mysql/

ThekeythingaboutthispostisthathehadtodeliberatelyswitchanodetoRSU,whichMySQLforPivotalCloudFoundry(PCF)neverdoesexceptduringSST.Sothisisademonstrationofwhatispossible,butdoesnotexplainhowacustomermayactuallyexperiencethisinproduction.

MySQLhasblacklisteditsownproxy?Whatdoestheerror, blockedbecauseofmanyconnection

errorsmean?

TherearetimeswhenMySQLwillblacklistitsownproxies:OUT07:44:02.070[paasEnv=MYPASSorgName=MYORGspaceName=MYSPACEappName=dc-routingappId=0123456789][http-nio-8080-exec-5]ERRORo.h.e.jdbc.spi.SqlExceptionHelper-Host'192.0.2.15'isblockedbecauseofmanyconnectionerrors;unblockwith'mysqladminflush-hosts'

YoucansolvethisbyrunningthefollowingonanyoftheMySQLjobVMS:

/var/vcap/jobs/mysql/packages/mariadb/bin/mysqladminflush-hosts

Thisisanartifactofanautomaticpolling-protectionfeature builtintoMySQLandMariaDB.ItisahistoricalfeatureintendedtoblockDenialofServiceattacks.ItisusuallytriggeredbyaLoadBalancerorSystemMonitoringsoftwareperformingempty“portchecks”againsttheMySQLproxies.ThisiswhyitisimportanttoconfigureanyLoadBalancertoperformTCPchecksagainsttheproxyhealth-checkport,default1936.Repeatedportchecksagainst3306willcauseanoutageforallMySQLforPivotalCloudFoundry(PCF)users.

Note:ThisissuehasbeendisabledasofMySQLforPivotalCloudFoundry(PCF)v1.8.0-edge.4.


https://www.percona.com/blog/2014/07/21/a-schema-change-inconsistency-with-galera-cluster-for-mysql/

https://mariadb.com/kb/en/mariadb/server-system-variables/#max_connect_errors

Cluster Scaling, Node Failure, and QuorumDocumentedherearescenariosinwhichthesizeofaclustermaychange,howtheclusterbehaves,andhowtorestoreservicefunctionwhenimpacted.GaleraCluster isusedtomanagetheMariaDB clusterinourrelease.

HealthyClusterGaleradocumentationreferstonodesinahealthyclusterasbeingpartofaprimarycomponent .Thesenodeswillrespondnormallytoallqueries,reads,writes,anddatabasemodifications.

Ifanindividualnodeisunabletoconnecttotherestofthecluster(ex:networkpartition)itbecomesnon-primary(stopsacceptingwritesanddatabasemodifications).Inthiscase,therestoftheclustershouldcontinuetofunctionnormally.Anon-primarynodemayeventuallyregainconnectivityandrejointheprimarycomponent.

Ifmorethanhalfofthenodesinaclusterarenolongerabletoconnecttoeachother,alloftheremainingnodeslosequorumandbecomenon-primary.Inthiscase,theclustermustbemanuallyrestarted,asdocumentedinthebootstrappingdocs.

GracefulremovalofanodeShuttingdownanodewithmonit(ordecreasingclustersizebyone)willcausethenodetogracefullyleavethecluster.

Clustersizeisreducedbyoneandmaintainshealthystate.Clusterwillcontinuetooperate,evenwithasinglenode,aslongasothernodesleftgracefully.

AddingnewnodesWhennewnodesareaddedtoorremovedfromaMySQLservice,atop-levelpropertyisupdatedwiththenewnodes’IPaddresses.AsBOSHdeploys,itwillupdatetheconfigurationandrestartallofthemysqlnodesandtheproxynodes(toinformthemofthenewIPaddressesaswell).Restartingthenodeswillcauseallconnectionstothatnodetobedroppedwhilethenoderestarts.

Scalingthecluster

Scalingupfrom1toNnodes

WhenanewMariaDbnodecomesonline,itreplicatesdatafromtheexistingnodeinthecluster.Oncereplicationiscomplete,thenodewilljointhecluster.Theproxywillcontinuetorouteallincomingconnectionstotheprimarynodewhileitremainshealthy.

Iftheproxydetectsthatthisnodebecomesunhealthy ,itwillseverexistingconnections,androuteallnewconnectionstoadifferent,healthynode.IftherearenohealthyMariaDbnodes,theproxywillrejectallsubsequentconnections.

Whiletransitioningfromonenodetoacluster,therewillbeanundeterminedperiodofperformancedegradationwhilethenewnodesyncsalldatafromtheoriginalnode.

Note:IfyouareplanningtoscaleupMariaDbnodes,itisrecommendedtodosoindifferentAvailabilityZonestomaximizeclusteravailability.AnAvailabilityZoneisanetwork-distinctsectionofagivenRegion.FurtherdetailsareavailableinAmazon’sdocumentation .

ScalingdownfromNto1node

WhenscalingfrommultiplenodestoasingleMariaDbnode,theproxywilldeterminethatthesoleremainingnodeistheprimarynode(provideditremainshealthy).TheproxyroutesincomingconnectionstotheremainingMariaDbnode.

Rejoiningthecluster(existingnodes)Existingnodesrestartedwithmonitshouldautomaticallyjointhecluster.Ifanexistingnodefailstojointhecluster,itmaybebecauseitstransactionrecord’s( seqno )ishigherthanthatofthenodesintheclusterwithquorum(akatheprimarycomponent).


http://galeracluster.com

https://mariadb.com/kb/en/mariadb/what-is-mariadb-galera-cluster/

http://galeracluster.com/documentation-webpages/glossary.html#term-primary-component

https://github.com/cloudfoundry/cf-mysql-release/blob/release-candidate/docs/proxy.md#unhealthy

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html

Ifthenodehasahigher seqno itwillbeapparentintheerrorlog /var/vcap/sys/log/mysql/mysql.err.log .

Ifthehealthynodesofaclusterhavealowertransactionrecordnumberthanthefailingnode,itmightbedesirabletoshutdownthehealthynodesandbootstrapfromthenodewiththemorerecenttransactionrecordnumber.Seethebootstrappingdocsformoredetails.

Manualrecoverymaybepossible,butiserror-proneandinvolvesdumpingtransactionsandapplyingthemtotherunningcluster(outofscopeforthisdoc).

Abandoningthedataisalsoanoption,ifyou’reokwithlosingtheunsynchronizedtransactions.Followthefollowingstepstoabandonthedata(asroot):

Stoptheprocesswith monit stop mariadb_ctrl .Deletethegalerastate( /var/vcap/store/mysql/grastate.dat )andcache( /var/vcap/store/mysql/galera.cache )filesfromthepersistentdisk.Restartingthenodewith monit start mariadb_ctrl .

StateSnapshotTransfer(SST)Whenanewnodeisaddedtotheclusterorrejoinsthecluster,itsynchronizesstatewiththeprimarycomponentviaaprocesscalledSST.Asinglenodefromtheprimarycomponentischosentoactasastatedonor.BydefaultGalerausesrsynctoperformSST,whichblocksforthedurationofthetransfer.However,MySQLforPivotalCloudFoundry(PCF)isconfiguredtouseXtrabackup ,whichallowsthedonornodetocontinuetoacceptreadsandwrites.

QuorumInorderfortheclustertocontinueacceptingrequests,aquorummustbereachedbypeer-to-peercommunication.Morethanhalfofthenodesmustberesponsivetoeachothertomaintainaquorum.

Ifmorethanhalfofthenodesareunresponsiveforaperiodoftimethenodeswillstoprespondingtoqueries,theclusterwillfail,andbootstrappingwillberequiredtore-enablefunctionality.

AvoidanevennumberofnodesItisgenerallyrecommendedtoavoidanevennumberofnodes.Thisisbecauseapartitioncouldcausetheentireclustertolosequorum,asneitherremainingcomponenthasmorethanhalfofthetotalnodes.

A2nodeclustercannottoleratethefailureofsinglenodefailureasthiswouldcauselossofquorum.Assuch,theminimumnumberofnodesrequiredtotoleratesinglenodefailureis3.

Unresponsivenode(s)Anodecanbecomeunresponsiveforanumberofreasons:

networklatencymysqlprocessfailurefirewallrulechangesvmfailure

Unresponsivenodeswillstoprespondingtoqueriesand,aftertimeout,leavethecluster.

Nodeswillbemarkedasunresponsive(inactive)either:

Iftheyfailtorespondtoonenodewithin15secondsORIftheyfailtorespondtoallothernodeswithin5seconds

Unresponsivenodesthatbecomeresponsiveagainwillrejointhecluster,aslongastheyareonthesameIPwhichispre-configuredinthegcommaddressonalltheotherrunningnodes,andaquorumwasheldbytheremainingnodes.

Allnodessuspendwritesoncetheynoticesomethingiswrongwiththecluster(writerequestshang).Afteratimeoutperiodof5seconds,requeststonon-quorumnodeswillfail.Mostclientsreturntheerror: WSREP has not yet prepared this node for application use .Someclientsmayinsteadreturn unknown error .Nodeswhohavereachedquorumwillcontinuefulfillingwriterequests.

Ifdeployedusingaproxy,acontinuallyinactivenodewillcausetheproxytofailover,selectingadifferentmysqlnodetoroutenewqueriesto.


http://www.percona.com/doc/percona-xtrabackup

Re-bootstrappingtheclusterafterquorumislostThestartscriptwillcurrentlybootstrapnode0onlyoninitialdeploy.Ifbootstrappingisnecessaryatalaterdate,itmustbedonemanually.Formoreinformationaboutmanuallybootstrappingacluster,seeBootstrappingGalera.

Ifthesinglenodeisbootstrapped,itwillcreateanewone-nodeclusterthatothernodescanjoin.

SimulatingnodefailureTosimulateatemporarysinglenodefailure,use kill -9 onthepidofthemysqlprocess.Thiswillonlytemporarilydisablethenodebecausetheprocessisbeingmonitoredbymonit,whichwillrestarttheprocessifitisnotrunning.

Tomorepermanentlydisabletheprocess,execute monit unmonitor mariadb_ctrl before kill -9 .

Tosimulatemulti-nodefailurewithoutkillinganodeprocess,communicationcanbeseveredbychangingtheiptablesconfigtodisallowcommunication:

iptables-F&&#optional-flushexistingrules\iptables-AINPUT-ptcp--destination-port4567-jDROP&&\iptables-AINPUT-ptcp--destination-port4568-jDROP&&\iptables-AINPUT-ptcp--destination-port4444-jDROP&&\iptables-AINPUT-ptcp--destination-port3306&&\iptables-AOUTPUT-ptcp--destination-port4567-jDROP&&\iptables-AOUTPUT-ptcp--destination-port4568-jDROP&&\iptables-AOUTPUT-ptcp--destination-port4444-jDROP&&\iptables-AOUTPUT-ptcp--destination-port3306

Torecoverfromthis,dropthepartitionbyflushingallrules: iptables-F


Cluster ConfigurationThispagedocumentsthevariousconfigurationdecisionsthathavebeenmadeinrelationtoMariaDBandGaleraincf-mysql-release.

SSTmethodGalerasupportsmultiplemethodsforStateSnapshotTransfer .The rsync methodisusuallyfastest.The xtrabackup methodhastheadvantageofkeepingthedonornodewriteableduringSST.Wehavechosentouse xtrabackup .

InnoDBLogFilesOurclusterdefaultsto1GBforlogfilesizetosupportlargerblob.

MaxUserConnectionsToensureallusersgetfairaccesstosystemresources,wehavecappedeachuser’snumberofconnectionsto40.

SkipExternalLockingSinceeachVirtualMachineonlyhasonemysqldprocessrunning,wedonotneedexternallocking.

MaxAllowedPacketWeallowblobsupto256MB.Thissizeisunlikelytolimitauser’squery,butisalsomanageableforourInnoDBlogfilesize.

InnodbFilePerTableInnodballowsusingeitherasinglefiletorepresentalldata,oraseparatefileforeachtable.Wechosetouseaseparatefileforeachtableasthisprovidesmoreflexibilityandoptimization.Forafulllistofprosandcons,seeMySQL’sdocumentationforInnoDBFile-Per-TableMode .

InnodbFileFormatTotakeadvantageofalltheextrafeaturesavailablewiththe innodb_file_per_table=

ONoption,weusethe Barracuda fileformat.

TemporaryTablesMySQLisconfiguredtoconverttemporaryin-memorytablestotemporaryon-disktableswhenaqueryEITHERgeneratesmorethan16millionrowsofoutputorusesmorethan32MBofdataspace.UserscanseeifaqueryisusingatemporarytablebyusingtheEXPLAINcommandandlookingfor“Usingtemporary,”intheoutput.Iftheserverprocessesverylargequeriesthatuse/tmpspacesimultaneously,itispossibleforqueriestoreceivenospacelefterrors.


http://www.percona.com/doc/percona-xtradb-cluster/5.5/manual/state_snapshot_transfer.html

http://dev.mysql.com/doc/refman/5.5/en/innodb-multiple-tablespaces.html

Proxy for MySQL for Pivotal Cloud FoundryInMySQLforPivotalCloudFoundry(PCF),Switchboard isusedtoproxyTCPconnectionstohealthyMariaDBnodes.

AproxyisusedtogracefullyhandlefailureofMariaDBnodes.Useofaproxypermitsveryfast,unambiguousfailovertoothernodeswithintheclusterintheeventofanodefailure.

Whenanodebecomesunhealthy,theproxyre-routesallsubsequentconnectionstoahealthynode.Allexistingconnectionstotheunhealthynodeareclosed.

ProxyDashboardTheserviceprovidesadashboardwhereadministratorscanobservehealthandmetricsforeachinstanceintheproxytier.Metricsincludethenumberofclientconnectionsroutedtoeachbackenddatabaseclusternode.

Thedashboardforeachproxyinstancecanbefoundat: http://proxy-<jobindex>-p-mysql.<system-domain> .Thejobindexstartsat0.Forexample,ifyouhavetwoproxyinstancesdeployedandyoursystem-domainis example.com ,dashboardswouldbeaccessibleat:

http://proxy-0-p-mysql.example.com


Basicauthcredentialsarerequiredtoaccessthedashboard.ThesecanbefoundintheCredentialstaboftheMySQLproductinOperationsManager.

ConsistentRoutingAtanygiventime,Switchboardwillonlyroutetooneactivenode.Thatnodewillcontinuetobetheonlyactivenodeuntilitbecomesunhealthy.

IfmultipleSwitchboardproxiesareusedinparallel(ex:behindaload-balancer)thereisnoguaranteethattheproxieswillchoosethesameactivenode.Thiscanresultindeadlocks,whereinattemptstoupdatethesamerowbymultipleclientswillresultonecommitsucceedingandtheotherfails.Thisisaknownissue,withexplorationofmitigationoptionsontheroadmapforthisproduct.Toavoidthisproblem,useasingleproxyinstanceoranexternalfailoversystemtodirecttraffictooneproxyinstanceatatime.

NodeHealth

HealthyTheproxyqueriesanHTTPhealthcheckprocess,co-locatedonthedatabasenode,whendeterminingwheretoroutetraffic.

IfthehealthcheckprocessreturnsHTTPstatuscodeof200,thenodeisaddedtothepoolofhealthynodes.

Aresurrectednodewillnotimmediatelyreceiveconnections.Theproxywillcontinuetorouteallconnections,neworexisting,tothecurrentlyactivenode.Inthecaseoffailover,allhealthynodeswillbeconsideredascandidatesfornewconnections.

UnhealthyIfthehealthcheckreturnsHTTPstatuscode503,thenodeisconsideredunhealthy.

Thishappenswhenanodebecomesnon-primary,asspecifiedbythecluster-behaviordocs.

Theproxywillseverallexistingconnectionstonewlyunhealthynodes.Clientsareexpectedtohandlereconnectingonconnectionfailure.Theproxywillroutenewconnectionstoahealthynode,assumingsuchanodeexists.

UnresponsiveIfnodehealthcannotbedeterminedduetoanunreachableorunresponsivehealthcheckendpoint,theproxywillconsiderthenodeunhealthy.ThismayhappenifthereisanetworkpartitionoriftheVMcontainingthehealthcheckandMariaDBnodedied.


https://github.com/cloudfoundry-incubator/switchboard



ProxycountIftheoperatorsetsthetotalnumberofproxiesto0hostsinOpsManagerorBOSHdeploymentmanifest,thenapplicationswillconnectdirectlytoonehealthyMariaDBnodemakingthatnodeasinglepointoffailureforthecluster.

Therecommendednumberofproxiesare2;thisprovidesredundancyshouldoneoftheproxiesfail.

RemovingtheproxyasaSPOFTheproxytierisresponsibleforroutingconnectionsfromapplicationstohealthyMariaDBclusternodes,evenintheeventofnodefailure.

BoundapplicationsareprovidedwithahostnameorIPaddresstoreachadatabasemanagedbytheservice.Bydefault,theMySQLservicewillprovideboundapplicationswiththeIPofthefirstinstanceintheproxytier.Evenifadditionalproxyinstancesaredeployed,clientconnectionswillnotberoutedthroughthem.Thismeansthefirstproxyinstanceisasinglepointoffailure.

In order to eliminate the first proxy instance as a single point of failure, operators must configure a load balancer to route clientconnections to all proxy IPs, and configure the MySQL service to give bound applications a hostname or IP address that resolves tothe load balancer.

ConfiguringloadbalancerConfiguretheloadbalancertoroutetrafficforTCPport3306totheIPsofallproxyinstancesonTCPport3306.Next,configuretheloadbalancer’shealthchecktousetheproxyhealthport.ThisisTCPport1936bydefaulttomaintainbackwardscompatibilitywithpreviousreleases.Thisportisnotconfigurable.

ConfiguringMySQLforPCFtogiveapplicationstheaddressoftheloadbalancerToensurethatboundapplicationswillusetheloadbalancertoreachbounddatabases,navigatetotheMySQLforPCFtileinOperationsManager,thentheResourceConfigconfigurationscreenwithinit.On AWS only,enteryourloadbalancer’shostnameinthe“ELBNames”columnfortheProxyrow.

AWSRoute53TosetupaRoundRobinDNSacrossmultipleproxyIPsusingAWSRoute53,followthefollowinginstructions:

1. LogintoAWS.

2. ClickRoute53.

3. ClickHostedZones.

4. Selectthehostedzonethatcontainsthedomainnametoapplyroundrobinroutingto.

5. Click‘GotoRecordSets’.

6. Selecttherecordsetcontainingthedesireddomainname.

7. Inthevalueinput,entertheIPaddressesofeachproxyVM,separatedbyanewline.

Finally,updatethemanifestproperty properties.mysql_node.host forthecf-mysql-brokerjob,asdescribedabove.

APITheproxyhostsaJSONAPIat proxy-<boshjobindex>-p-mysql.<systemdomain>/v0/ .

TheAPIprovidesthefollowingroute:

Request:


Method:GET

Path: /v0/backends

Params:~

Headers:BasicAuth

Response:

[{"name":"mysql-0","ip":"1.2.3.4","healthy":true,"active":true,"currentSessionCount":2},{"name":"mysql-1","ip":"5.6.7.8","healthy":false,"active":false,"currentSessionCount":0},{"name":"mysql-2","ip":"9.9.9.9","healthy":true,"active":false,"currentSessionCount":0}]

FormoreinformationaboutSwitchBoard,readtheproxydocumentation


Creating Application Security Groups for MySQLThistopicdescribeshowtocreateApplicationSecurityGroups (ASGs)forMySQLforPivotalCloudFoundry(PCF).

ToallowsmoketeststorunwhenyouinstalltheMySQLforPCFserviceandallowappstoaccessMySQLforPCFafteritisinstalled,youmustcreateanappropriateASGandbindittotheservice.

Inaddition,applicationcontainersthataccessinstancesofthisservicerequireanoutboundnetworkconnectiontotheloadbalancerconfiguredfortheMySQLforPCFservice.

TocreateASGsfortheMySQLforPCFservice,performthefollowingsteps:

1. CreateaJSONfilewiththefollowingcontentscalled p-mysql-security-group.json :

[{"ports":"3306","protocol":"tcp","destination":"REPLACEWITHTHEP-MYSQLLOADBALANCERIP,RANGEORCIDR"}]

Inthe destination field,addtheIPaddress,range,orCIDRoftheloadbalancerthatyouconfiguredfortheMySQLforPCFservice.

2. LogintoyourPCFdeploymentasanadministrator,andcreateanASGcalled p-mysql-service .

#afterlogginginasanadministrator$cfcreate-security-groupp-mysql-servicep-mysql-security-group.json

3. BindthenewASGtothe default-running ASGsettoallowallapplicationstoaccesstheservice.

$cfbind-running-security-groupp-mysql-service

Iftheserviceshouldonlybemadeavailabletospecificspaces,bindtheASGdirectlytothosespaces.

$cfbind-security-groupp-mysql-serviceORGANIZATION_NAMESPACE_NAME

Note:WithoutanASG,theserviceisnotinstallableorusable.


http://docs.pivotal.io/pivotalcf/adminguide/app-sec-groups.html

Monitoring the MySQL ServiceThisdocumentdescribeshowtousetheReplicationCanaryandInterruptortomonitoryourMySQLcluster.

ReplicationCanaryMySQLforPivotalCloudFoundry(PCF)isaclusteredsolutionthatusesreplicationtoprovidebenefitssuchasquickfailoverandrollingupgrades.Thisismorecomplexthanasinglenodesystemwithnoreplication.MySQLforPCFincludesaReplicationCanarytohelpwiththeincreasedcomplexity.TheReplicationCanaryisalong-runningmonitorthatvalidatesthatreplicationisworkingwithintheMySQLcluster.

HowitWorksTheReplicationCanarywritestoaprivatedatasetinthecluster,andattemptstoreadthatdatafromeachnode.Itpausesbetweenwritingandreadingtoensurethatthewritesetshavebeencommittedacrosseachnodeofthecluster.Theprivatedatasetdoesnotuseasignificantamountofdiskcapacity.

Whenreplicationfailstoworkproperly,theCanarydetectsthatitcannotreadthedatafromallnodes,andimmediatelytakestwoactions:

E-mailsapre-configuredaddresswithamessagethatreplicationhasfailed.Seethesamplebelow.

Disablesclientaccesstothecluster.

SampleNotificationE-mailIftheCanarydetectsareplicationfailure,itimmediatelysendsane-mailthroughtheElasticRuntimenotificationservice.Seethefollowingexample:

Subject:CFNotification:p-mysqlReplicationCanary,alert417

Thismessagewassentdirectlytoyouremailaddress.

{alert-code417}Thisisane-mailtonotifyyouthattheMySQLservice'sreplicationcanaryhasdetectedanunsafeclusterconditioninwhichreplicationisnotperformingasexpectedacrossallnodes.

ClusterAccessEachtimetheCanarydetectsclusterreplicationfailure,itinstructsallproxiestodisableconnectionstothedatabasecluster.Ifthereplicationissueresolves,theCanarydetectsthisandautomaticallyrestoresclientaccesstothecluster.

IfyoumustrestoreaccesstotheclusterregardlessoftheReplicationCanary,contactSupport.

DetermineProxyState

YoucandetermineiftheCanarydisabledclusteraccessbyusingtheProxyAPI.Seethefollowingexample:

ubuntu@ip-10-0-0-38:~$curl-kuadmin:PASSWORD_FROM_OPSMGR-XGEThttps://proxy-0-p-mysql.SYSTEM-DOMAIN/v0/cluster;echo{"currentBackendIndex":0,"trafficEnabled":false,"message":"Disablingclustertraffic","lastUpdated":"2016-07-27T05:16:29.197754077Z"}

EnabletheReplicationCanaryToenabletheReplicationCanary,followtheinstructionsbelowtoconfigureboththeElasticRuntimetileandtheMySQLforPCFtile.

Note:Malfunctioningreplicationexposestheclustertothepossibilityofdataloss.Becauseofthis,bothbehaviorsareenabledbydefault.ItiscriticalthatyoucontactPivotalsupportimmediatelyinthecaseofreplicationfailure.Supportwillworkwithyoutodeterminethenatureoftheclusterfailureandprovideguidanceregardingasolution.


ConfiguretheElasticRuntimeTile

1. IntheSMTP Configsection,enteraFrom EmailthattheReplicationCanarycanusetosendnotifications,alongwiththeSMTPserverconfiguration.

2. IntheErrandssection,selecttheNotificationserrand.

ConfiguretheMySQLforPCFTile1. IntheAdvanced Optionssection,selectEnable replication canary.

2. IfyouwanttotheReplicationCanarytosende-mailbutnotdisableaccessattheproxy,selectNotify only.

3. YoucanoverridetheReplication canary time period.TheReplication canary time periodsetshowfrequentlythecanarychecksforreplicationfailure,inseconds.Thisaddsasmallamountofloadtothedatabases,butthecanaryreactsmorequicklytoreplicationfailure.Thedefaultis30seconds.

4. YoucanoverridetheReplication canary read delay.TheReplication canary read delaysetshowlongthecanarywaitstoverifydataisreplicatingacrosseachMySQLnode,inseconds.Clustersunderheavyloadexperiencesomesmallreplicationlagaswritesetsarecommittedacrossthenodes.TheDefaultis20seconds.

5. EnteranE-mail addresstoreceivemonitoringnotifications.Useacloselymonitorede-mailaddressaccount.ThepurposeoftheCanaryistoescalatereplicationfailureasquicklyaspossible.

6. IntheResource Configsection,ensuretheMonitoringjobhasoneinstance.

Note:InatypicalPCFdeployment,thesesettingsarealreadyconfigured.

Note:Pivotalrecommendsleavingthischeckboxunselectedduetothepossibilityofdatalossfromreplicationfailure.


DisabletheReplicationCanaryIfyoudonotneedtheReplicationCanary,forinstanceifyouuseasingleMySQLnode,followthisproceduretodisableboththejobandtheresourceconfiguration.

1. IntheAdvanced OptionssectionoftheMySQLforPCFtile,selectDisable Replication Canary.

2. IntheResource Configpane,settheMonitoringjobtozeroinstances.

InterruptorTherearerarecasesinwhichaMySQLnodesilentlyfallsoutofsyncwiththeothernodesofthecluster.TheReplicationCanarycloselymonitorstheclusterforthiscondition.However,iftheReplicationCanarydoesnotdetectthefailure,theInterruptorprovidesasolutionforpreventingdataloss.

HowitWorksIfthenodereceivingtrafficfromtheproxyfallsoutofsyncwiththecluster,itgeneratesadatasetthattheothernodesdonothave.Ifthesamenodelaterreceivesatransactionthatisnotcompatiblewiththedatasetsoftheothernodes,itdiscardsitslocaldatasetandadoptsthedatasetsoftheothernodes.Thisisgenerallydesiredbehavior,unlessdatareplicationisnotfunctioningacrossthecluster.Thenodecoulddestroyvaliddatabydiscardingitslocaldataset.Whenenabled,theInterruptorpreventsthenodefromdestroyingitslocaldatasetifthereisariskoflosingvaliddata.

Anout-of-syncnodeemploysoneoftwotwomodes tocatchupwiththecluster:

Incremental State Transfer (IST):Ifanodehasbeenoutoftheclusterforarelativelyshortperiodoftime,suchasareboot,thenodeinvokesIST.Thisisnotadangerousoperation,andtheInterruptordoesnotinterfere.

State Snapshot Transfer (SST):Ifanodehasbeenunavailableforanextendedamountoftime,suchasahardwarefailurethatrequiresphysicalrepair,thenodemayinvokeSST.Incasesoffailedreplication,SSTcancausedataloss.Whenenabled,theInterruptorpreventsthismethodofrecovery.

SampleNotificationE-mailTheInterruptorsendsanemailthroughtheElasticRuntimenotificationservicewhenitpreventsanodefromrejoiningacluster.Seethefollowingexample:

Subject:CFNotification:p-mysqlalert100

Thismessagewassentdirectlytoyouremailaddress.

{alert-code100}Hello,justwantedtoletyouknowthattheMySQLnode/clusterhasgonedownandhasbeendisallowedfromre-joiningbytheinterruptor.

Note:IfyoureceiveanotificationthattheInterruptorhasactivated,itiscriticalthatyoucontactPivotalsupportimmediately.Supportwillworkwithyoutodeterminethenatureofthefailure,andprovideguidanceregardingasolution.


http://galeracluster.com/documentation-webpages/statetransfer.html

InterruptorLogsYoucanconfirmthattheInterruptorhasactivatedbyexamining /var/vcap/sys/log/mysql/mysql.err.log onthefailingnode.Thelogcontainsthefollowingmessage:

WSREP_SST:[ERROR]#####################################################################################(2016061004:33:21.338)WSREP_SST:[ERROR]SSTdisabledduetodangerofdataloss.Verifydataandruntherejoin-unsafeerrand(2016061004:33:21.340)WSREP_SST:[ERROR]#####################################################################################(2016061004:33:21.341)

ForceaNodetoRejointheClusterIngeneral,iftheInterruptorhasactivatedbuttheReplicationCanaryhasnottriggered,itissafeforthenodetorejointhecluster.

1. Followtheseinstructionstochoosethep-mysqlmanifestwiththeBOSHCLI.

2. Run boshrunerrandrejoin-unsafe toforceanodetorejointhecluster:

$boshrunerrandrejoin-unsafe[...][stdout]Startedrejoin-unsafeerrand...Successfullyrepairedclusterrejoin-unsafeerrandcompleted

[stderr]None

Errand`rejoin-unsafe'completedsuccessfully(exitcode0)

DisabletheInterruptorTheInterruptorisenabledbydefault.TodisabletheInterruptor:

IntheAdvanced Optionssection,underEnable optional protections,un-checkPrevent node auto re-join.

Note:ThistopicrequiresyoutoruncommandsfromtheOpsManagerDirector usingtheBOSHCLI.RefertotheAdvancedTroubleshootingwiththeBOSHCLI topicformoreinformation.


http://docs.pivotal.io/pivotalcf/customizing/index.html

http://docs.pivotal.io/pivotalcf/customizing/trouble-advanced.html#prepare

Determining Cluster StateConnecttoeachMySQLnodeusingamysqlclientandcheckitsstatus.

$mysql-hNODE_IP-uroot-pPASSWORD-e'SHOWSTATUSLIKE"wsrep_cluster_status";'+----------------------+---------+|Variable_name|Value|+----------------------+---------+|wsrep_cluster_status|Primary|+----------------------+---------+

Ifallnodesareinthe Primary component,youhaveahealthycluster.Ifsomenodesareina Non-primary component,thosenodesarenotabletojointhecluster.

Seehowmanynodesareinthecluster.

$mysql-hNODE_IP-uroot-pPASSWORD-e'SHOWSTATUSLIKE"wsrep_cluster_size";'+--------------------+-------+|Variable_name|Value|+--------------------+-------+|wsrep_cluster_size|3|+--------------------+-------+

Ifthevalueof wsrep_cluster_size isequaltotheexpectednumberofnodes,thenallnodeshavejoinedthecluster.Otherwise,checknetworkconnectivitybetweennodesanduse monitstatus toidentifyanyissuespreventingnodesfromstarting.

Formoreinformation,seetheofficialGaleradocumentationforCheckingClusterIntegrity .


http://galeracluster.com/documentation-webpages/monitoringthecluster.html#checking-cluster-integrity

Bootstrapping a Galera ClusterPage last updated:

ThistopicdescribestheprocedureforrecoveringaterminatedElasticRuntimeclusterusingaprocessknownasbootstrapping.

WhentoBootstrapYoumustbootstrapaclusterthatlosesquorum.Aclusterlosesquorumwhenlessthanhalfofthenodescancommunicatewitheachotherforlongerthantheconfiguredgraceperiod.Ifaclusterdoesnotlosequorum,individualunhealthynodesautomaticallyrejointheclusterafterresolvingtheerror,restartingthenode,orrestoringconnectivity.

Youcandetectlostquorumthroughthefollowingsymptoms:

Allnodesappear“Unhealthy”ontheproxydashboard,viewableat proxy-BOSH-JOB-INDEX.p-mysql.YOUR-SYSTEM-DOMAIN :

Allresponsivenodesreportthevalueof wsrep_cluster_status as non-Primary :

mysql>SHOWSTATUSLIKE'wsrep_cluster_status';+----------------------+-------------+|Variable_name|Value|+----------------------+-------------+|wsrep_cluster_status|non-Primary|+----------------------+-------------+

Allresponsivenodesrespondwith ERROR1047 whenqueriedwithmoststatementtypes:

mysql>select*frommysql.user;ERROR1047(08S01)atline1:WSREPhasnotyetpreparednodeforapplicationuse

SeetheClusterScaling,NodeFailure,andQuorumtopicformoredetailsaboutdeterminingclusterstate.

Followthestepsbelowtorecoveraclusterthathaslostquorum.

Step1:ChoosetheCorrectManifest

1. LogintotheBOSHdirectorbyrunning boshtargetDIRECTOR-URL followedby boshloginUSERNAMEPASSWORD .

2. Run boshdeployments .

$boshdeploymentsActingasuser'director'on'p-bosh-30c19bdd43c55c627d70'

+-------------------------+-------------------------------+----------------------------------------------+--------------+|Name|Release(s)|Stemcell(s)|CloudConfig|+-------------------------+-------------------------------+----------------------------------------------+--------------+|cf-e82cbf44613594d8a155|cf-autoscaling/28|bosh-aws-xen-hvm-ubuntu-trusty-go_agent/3140|none|||cf-mysql/23|||||cf/225|||||diego/0.1441.0|||||etcd/18|||||garden-linux/0.327.0|||||notifications-ui/10|||||notifications/19|||||push-apps-manager-release/397|||+-------------------------+-------------------------------+----------------------------------------------+--------------+|p-mysql|p-mysql|||+-----------------------------------------------------------------------------------------------------------------------+

3. Downloadthemanifest.

Note:ThistopicrequiresyoutoruncommandsfromtheOpsManagerDirector usingtheBOSHCLI.RefertotheAdvancedTroubleshootingwiththeBOSHCLI topicformoreinformation.


http://docs.pivotal.io/p-mysql/1-8/cluster-behavior.html

http://docs.pivotal.io/pivotalcf/customizing/index.html

http://docs.pivotal.io/pivotalcf/customizing/trouble-advanced.html#prepare

$boshdownloadmanifestp-mysql/tmp/p-mysql.ymlActingasuser'director'ondeployment[...]Deploymentmanifestsavedto`/tmp/p-mysql.yml'

4. SetBOSHtousethedeploymentmanifestyoudownloaded.

$boshdeployment/tmp/p-mysql.yml

Step2:BootstrapTodeterminewhichsetofinstructionstofollow,youmustdeterminethestateofyourVirtualMachines(VMs).

Iftheoutputof bosh instances showsthestateofthejobsas failing ,thenfollowthestepsinScenario1below.

Iftheoutputof bosh instances showsthestateofthejobsas unknown/unknown ,thenfollowthestepsinScenario2.

Scenario1:VirtualMachinesRunning,ClusterDisruptedInthisscenario,nodesareupandrunning,buttheclusterhasbeendisrupted.

1. SSHtoeachnodeintheclusterand,asroot,shutdownthe mariadb process.ToSSHintoBOSH-deployedVMs,seetheAdvancedTroubleshootingwiththeBOSHCLItopic.

$monitstopmariadb_ctrl

Re-bootstrappingtheclusterwillnotbesuccessfulunlessallothernodeshavebeenshutdown.

2. Chooseanodetobootstrapbylocatingthenodewiththehighesttransactionsequencenumber( seqno ).Youcanobtainthe seqno ofastoppednodeinoneoftwoways:

Ifanodeshutdowngracefully,the seqno isintheGalerastatefileofthenode.

$cat/var/vcap/store/mysql/grastate.dat|grep'seqno:'

Ifthenodecrashedorwaskilled,the seqno intheGalerastatefileofthenodeis -1 .Inthiscase,the seqno mayberecoverablefromthedatabase.

1. Runthefollowingcommandtostartupthedatabase,logtherecoveredsequencenumber,andexit.

$/var/vcap/packages/mariadb/bin/mysqld--wsrep-recover

2. Scantheerrorlogfortherecoveredsequencenumber.Thelastnumberafterthegroupid( uuid )istherecovered seqno :

$grep"Recoveredposition"/var/vcap/sys/log/mysql/mysql.err.log|tail-115022518:09:42mysqld_safeWSREP:Recoveredpositione93955c7-b797-11e4-9faa-9a6f0b73eb46:15

Ifthenodeneverconnectedtotheclusterbeforecrashing,itmaynothaveagroupid( uuid in grastate.dat ).Inthiscase,youcannotrecoverthe seqno .Unlessallnodescrashedthisway,donotchoosethisnodeforbootstrapping.

3. Choosethenodewiththehighest seqno valueasthebootstrapnode.Ifallnodeshavethesame seqno ,youcanchooseanynodeasthebootstrapnode.

4. Onthebootstrapnode,updatethestatefileandrestartthe mariadb process.

$echo-n"NEEDS_BOOTSTRAP">/var/vcap/store/mysql/state.txt$monitstartmariadb_ctrl

Note:Thefollowingstepsarepronetousererrorandcanresultinlostdataiffollowedincorrectly.

Note:Onlyperformthesebootstrapcommandsonthenodewiththehighest seqno .Otherwise,thenodewiththehighest seqno willbeunabletojointhenewclusterunlessitsdataisabandoned.Its mariadb processwillexitwithanerror.SeetheClusterScaling,NodeFailure,andQuorumtopicformoredetailsonintentionallyabandoningdata.


http://docs.pivotal.io/p-mysql/pivotalcf/customizing/trouble-advanced.html

http://docs.pivotal.io/p-mysql/1-8/cluster-behavior.html

5. Checkthatthe mariadb processhasstartedsuccessfully.

$watchmonitsummary

Itcantakeuptotenminutesfor monit tostartthe mariadb process.

6. Oncethebootstrappednodeisrunning,startthe mariadb processontheremainingnodes:

a. IftheInterruptorisenabled,runthefollowingcommandbeforestarting mariadb :

$touch/var/vcap/sys/run/galera-healthcheck/enable_sst

b. Startthemariadbprocessusing monit :

$monitstartmariadb_ctrl

7. Verifythatthenewnodeshavesuccessfullyjoinedthecluster.Thefollowingcommanddisplaysthetotalnumberofnodesinthecluster:

mysql>SHOWSTATUSLIKE'wsrep_cluster_size';

Scenario2:VirtualMachinesTerminatedorLostInthisscenario,severecircumstancessuchaspowerfailurehaveterminatedallofyourVMs.YouneedtorecreatetheVMsbeforeyoucanrecoverthecluster.

1. IfyouenabledtheVMResurrector inOpsManager,thesystemdetectstheterminatedVMsandautomaticallyattemptstorecreatethem.Runboshtasksrecent--no-filter toseethe scanandfix jobrunbytheVMResurrector.

$boshtasksrecent--no-filter+-----+------------+-------------------------+----------+--------------------------------------------+---------------------------------------------------+|#|State|Timestamp|User|Description|Result|+-----+------------+-------------------------+----------+--------------------------------------------+---------------------------------------------------+|123|queued|2016-01-0800:18:07UTC|director|scanandfix||

IfyouhavenotenabledtheVMResurrector,runtheBOSHcloudcheckcommand boshcck todeleteanyplaceholderVMs.Whenprompted,chooseDeleteVMreference byentering 3 .


http://docs.pivotal.io/pivotalcf/customizing/resurrector.html#enabling

$boshcck

Actingasuser'director'ondeployment'cf-e82cbf44613594d8a155'on'p-bosh-30c19bdd43c55c627d70'Performingcloudcheck...

Directortask34Startedscanning22vmsStartedscanning22vms>CheckingVMstates.Done(00:00:10)Startedscanning22vms>19OK,0unresponsive,3missing,0unbound,0outofsync.Done(00:00:00)Donescanning22vms(00:00:10)

Startedscanning10persistentdisksStartedscanning10persistentdisks>Lookingforinactivedisks.Done(00:00:02)Startedscanning10persistentdisks>10OK,0missing,0inactive,0mount-infomismatch.Done(00:00:00)Donescanning10persistentdisks(00:00:02)

Task34done

Started2015-11-2601:42:42UTCFinished2015-11-2601:42:54UTCDuration00:00:12

Scaniscomplete,checkingifanyproblemsfound.

Found3problems

Problem1of3:VMwithcloudIDì-afe2801f'missing.1.Skipfornow2.RecreateVM3.DeleteVMreferencePleasechoosearesolution[1-3]:3

Problem2of3:VMwithcloudIDì-36741a86'missing.1.Skipfornow2.RecreateVM3.DeleteVMreferencePleasechoosearesolution[1-3]:3

Problem3of3:VMwithcloudIDì-ce751b7e'missing.1.Skipfornow2.RecreateVM3.DeleteVMreferencePleasechoosearesolution[1-3]:3

Belowisthelistofresolutionsyou'veprovidedPleasemakesureeverythingisfineandconfirmyourchanges

1.VMwithcloudIDì-afe2801'missing.DeleteVMreference

2.VMwithcloudIDì-36741a86'missing.DeleteVMreference

3.VMwithcloudIDì-ce751b7e'missing.DeleteVMreference

Applyresolutions?(type'yes'tocontinue):yesApplyingresolutions...

Directortask35StartedapplyingproblemresolutionsStartedapplyingproblemresolutions>missing_vm11:DeleteVMreference.Done(00:00:00)Startedapplyingproblemresolutions>missing_vm27:DeleteVMreference.Done(00:00:00)Startedapplyingproblemresolutions>missing_vm26:DeleteVMreference.Done(00:00:00)Doneapplyingproblemresolutions(00:00:00)

Task35done

Started2015-11-2601:47:08UTCFinished2015-11-2601:47:08UTCDuration00:00:00Cloudcheckisfinished

2. Run boshinstances andexaminetheoutput.TheVMstransitionfrom unresponsiveagent to starting .Ultimately,twoappearas failing .DonotproceedtothenextstepuntilallthreeVMsareinthe starting or failing state.


$boshinstances[...]+--------------------------------------------------+----------+------------------------------------------------+------------+|mysql-partition-e97dae91e44681e0b543/0|starting|mysql-partition-e97dae91e44681e0b543|192.0.2.60||mysql-partition-e97dae91e44681e0b543/1|failing|mysql-partition-e97dae91e44681e0b543|192.0.2.61||mysql-partition-e97dae91e44681e0b543/2|failing|mysql-partition-e97dae91e44681e0b543|192.0.2.62|+--------------------------------------------------+----------+------------------------------------------------+------------+

3. CompletethestepsforScenario1.

4. Run boshinstances andexaminetheoutputtoconfirmthatthebootstrapwassuccessful.Someinstancesmaystillappearas failing .


Backing Up MySQL for Pivotal Cloud FoundryThistopicdescribeshowtoenable,configure,andusebackupsinMySQLforPivotalCloudFoundry(PCF).

OverviewAutomatedbackupshavethefollowingfeatures:

Periodicallycreateanduploadbackupartifactssuitableforrestoringthecompletesetofdatabaseinstancesallocatedintheservice

Nolocks,nodowntime

TheonlyeffectontheservingsystemsistheamountofI/OrequiredtocopythedatabaseandlogfilesoffoftheVM

Includesametadatafilethatcontainsthecriticaldetailsofthebackupartifact,includingtheeffectivecalendartimeofthebackup

BackupartifactsareencryptedwithintheMySQLforPCFclusterofVMs;unencrypteddataisnevertransportedoutsideoftheMySQLforPCFdeployment

EnableAutomatedBackupsYoucanconfigureMySQLforPCFtoautomaticallybackupitsdatabasestoexternalstorage.

How and Where:Therearetwooptionsforhowautomatedbackupstransferbackupdataandwherethedatasavesoutto:

MySQLforPCFrunsan scp commandthatsecure-copiesbackupfilestoaVMorphysicalmachineoperatingoutsideofPCF.TheoperatorprovisionsthebackupmachineseparatelyfromtheirPCFinstallation.Thisisthemostefficientoption.MySQLforPCFrunsanS3 clientthatsavesbackupstoanAmazonS3bucket,Ceph storagecluster,orotherS3-compatibleendpointcertifiedbyPivotal.

When:Backupsfollowaschedulethatyouspecifywithacron expression.

What:Youcanbackupjusttheprimarynode,orallnodesinthecluster.

Toenableautomatedbackupsandconfigurethemforoptionsabove,performthefollowingsteps:

1. NavigatetotheMySQLforPivotalCloudFoundrytileontheOpsManagerInstallationDashboard.

2. ClickBackups.

3. UnderBackups,clickEnable Backups.

4. ForCron Schedule,enteracronscheduleforthebackups.Thesyntaxissimilartotraditionalcron,withadditionalfeaturessuchas @every 1d ,whichspecifiesdailybackups.SeethecronGolibrarydocumentation formoreinformation.

5. Ifyouwanttobackupallnodes,selecttheBack up all nodescheckbox.

6. ToenablebackupsusingCeph orAWS,continuetotheCephorAWSsection.ToenablebackupsusingSCP,continuetotheSCPsection.


https://aws.amazon.com/documentation/s3/

http://docs.ceph.com/docs/master/

http://godoc.org/github.com/robfig/cron

https://godoc.org/github.com/robfig/cron

http://docs.ceph.com/docs/master/

CephorAWSTobackupyourdatabaseonCephorAmazonWebServices(AWS)S3,performthefollowingsteps:

1. SelectCeph or Amazon S3.

2. EnteryourS3 Endpoint URL.Forinstance, https://s3.amazonaws.com .

3. EnteryourS3 Bucket Name.Donotincludean s3:// prefix ,atrailing / ,orunderscores.Ifthebucketdoesnotalreadyexist,itwillbecreatedautomatically.

4. ForBucket Path,specifyafolderwithinthebuckettoholdyourMySQLbackups.Donotincludeatrailing / .Ifthefolderdoesnotalreadyexist,itwillbecreatedautomatically.

5. ForAWS Access Key IDandAWS Secret Access Key,enteryourCephorAWScredentials.ForAWS,PivotalrecommendscreatinganIAM

credentialthatonlyhasaccesstothisbucket.

6. ClickSave.

SCPTobackupyourdatabaseusingSCP,performthefollowingsteps:

Note:Youmustusethisfolderexclusivelyforthiscluster’sbackupartifacts.Mixingthebackupartifactsfromdifferentclusterswithinasinglefoldercancauseconfusionandpossibleinadvertentlossofbackupartifacts.


https://aws.amazon.com/iam/

1. SelectSCP to a Remote Host.

2. EntertheUsername,Hostname,andDestination Directoryforthebackups.

3. ForPrivate Key,pasteintheprivatekeythatwillbeusedtoencrypttheSCPtransfer.

4. EntertheSCP Port.SCPrunsonport22bydefault.

5. ClickSave.

DisableAutomatedBackupsTodisableautomatedbackups,performthefollowingsteps:


Note:PivotalrecommendsusingaVMnotwithinthePCFdeploymentforthedestinationofSCPbackups.SCPenablestheoperatortouseanydesiredstoragesolutiononthedestinationVM.


2. ClickBackups.

3. UnderBackups,clickDisable Backups.

4. UnderBackup Destination,clickNo Backups.

5. ClickSave.

6. Intheleftnavigation,clickResource Config.

7. ChangethenumberofinstancesforBackup Prepare Nodefrom 1 to 0 .

8. ClickSave.

9. ReturntotheOpsManagerInstallationDashboardandclickApply Changes.

ToconfigureautomatedbackupsforMySQLforPCF,performthefollowingsteps:


2. ClickBackups.

UnderstandBackupsThesectionsbelowdescribetheprocessthatMySQLforPCFcomponentjobsfollowwhenperformingautomatedbackups,andtheformatforthemetadatafilethatrecordsinformationabouteachbackup.

BackupProcessOperatorsuseOpsManagertoconfigurethescheduleforautomatedbackupsandthelocationandcredentialsneededtostorebackupartifacts.

ThediagrambelowshowstheprocessthroughwhichMySQLforPCFjobsinitiateandrunautomatedbackups.

sequenceDiagramparticipantBlobstoreparticipantServiceBackupjobNoteoverServiceBackupjob:Triggeredbytimer,followingscheduleconfiguredinOpsManagerServiceBackupjob->>StreamingBackupclient:RequestbackupStreamingBackupclient->>StreamingBackuptool:RequestbackupStreamingBackuptool->>MySQLserver:RequestbackupNoteoverMySQLserver:FlushtableswithreadlockMySQLserver->>StreamingBackuptool:DataStreamingBackuptool->>StreamingBackupclient:DataStreamingBackupclient->>ServiceBackupjob:DataNoteoverServiceBackupjob:CompressandencryptServiceBackupjob->>Blobstore:BackupartifactNoteoverBlobstore:Storebackupartifact,usingcredsconfiguredinOpsManagerBlobstore-->>ServiceBackupjob:ConfirmartifactstoredNoteoverServiceBackupjob:Cleanuplocalstorage

TwoMySQLforPCFcomponentVMshostthejobslistedaboveasfollows:

Job Job name in the code Host VM


ServiceBackup service-backup BackupPrepareVMStreamingBackupclient streaming-backup-client

StreamingBackuptool streaming-backup-tool

MySQLVMMySQLserver mysql

BackupMetadataAlongwitheachbackupartifact,MySQLforPCFuploadsa mysql-backup-XXXXXXXXXX.txt metadatafile.

Thecontentsofthemetadatafileresemblethefollowing:

compact=Nencrypted=Ntool_version=2.4.5server_version=10.1.20-MariaDBend_time=2017-05-0523:26:19binlog_pos=filename'mysql-bin.000016',position'7000000',GTIDofthelastchange'0-1-30000'incremental=Nformat=tarcompressed=Nuuid=30000000-3000-1000-9000-40000000000fname=lock_time=0innodb_from_lsn=0innodb_to_lsn=6286393partial=Ntool_command=--user=admin--password=...--stream=tartmp/ibbackup_version=2.4.5tool_name=innobackupexstart_time=2017-05-0523:26:17

Withinthisfile,themostimportantitemsarethe start_time andthe server_version entries.Transactionsthathavenotbeencompletedatthestartofthebackupeffortarenotpresentintherestoredartifact.

RestoreaBackupArtifactMySQLforPCFkeepsatleasttwocompletecopiesofthedata.Inmostcases,ifaclusterisstillabletoconnecttopersistentstorage,youcanrestoreaclustertohealthusingthebootstrapprocess.Beforeresortingtoadatabaserestore,contactPivotalSupport toensureyourexistingclusterisbeyondhelp.

ThedisasterrecoverybackupsfeatureofMySQLforPCFisprimarilyintendedasawaytorecoverdatatothesamePCFdeploymentfromwhichthedatawasbackedup.Thisprocessreplaces100%ofthedataandstateofarunningMySQLforPCFcluster.Thisisespeciallyrelevantwithregardtoserviceinstancesandbindings.

Intheeventofatotalclusterloss,theprocesstorestoreabackupartifacttoaMySQLforPCFclusterisentirelymanual.Performthefollowingstepstousetheoffsitebackupstorestoreyourclustertoitspreviousstate:

1. DiscovertheencryptionkeysintheCredentialstaboftheMySQLforPCFtile.

2. Ifnecessary,installthesameversionoftheMySQL for PCFproductintheOpsManagerInstallationDashboard.

3. PerformthefollowingstepstoreducethesizeoftheMySQLforPCFclustertoasinglenode:

a. FromtheOpsManagerInstallationDashboard,clicktheMySQL for PCFtile.b. ClickResource Config.c. SetthenumberofinstancesforMySQL Serverto1.d. ClickSave.e. ReturntotheOpsManagerInstallationDashboardandclickApply Changes.

Note:Both compressed and encrypted showas N inthisfile,yettheartifactuploadedbyMySQLforPCFisbothcompressedandencrypted.Thisisaknownbug.

Note:Becauseofhowservicesinstancesaredefined,youcannotrestoreaMySQLforPCFdatabasetoadifferentPCFdeployment.


https://support.pivotal.io

4. Afterthedeploymentfinishes,performthefollowingstepstopreparethefirstnodeforrestoration:

a. SSHintotheOpsManagerDirector.Formoreinformation,seetheSSHintoOpsManager sectioninthetopic.

b. RetrievetheIPaddressfortheMySQLserverbynavigatingtotheMySQL for PCFtileandclickingtheStatustab.c. RetrievetheVMcredentialsfortheMySQLserverbynavigatingtotheMySQL for PCFtileandclickingtheCredentialstab.d. FromtheOpsManagerDirectorVM,usetheBOSHCLItoSSHintothefirstMySQLjob.Formoreinformation,seetheBOSHSSH sectioninthe

topic.e. OntheMySQLserverVM,becomesuperuser:

$sudosu

f. Pausethelocaldatabaseserver:

$monitstopall

g. Confirmthatalljobsarelistedas not monitored :

$watchmonitsummary

h. DeletetheexistingMySQLdatathatisstoredondisk:

$rm-rf/var/vcap/store/mysql/*

5. Performthefollowingstepstorestorethebackup:

a. Movethecompressedbackupfiletothenodeusing scp .b. Decryptandexpandthefileusing gpg ,sendingtheoutputtotar:

$gpg--decryptmysql-backup.tar.gpg|tar-C/var/vcap/store/mysql-xvf-

c. Changetheownerofthedatadirectory,becauseMySQLexpectsthedatadirectorytobeownedbyaparticularuser:

$chown-Rvcap:vcap/var/vcap/store/mysql

d. Startallserviceswith monit :

$monitstartall

e. Watchthesummaryuntilalljobsarelistedas running :

$watchmonitsummary

f. ExitoutoftheMySQLnode.

6. Performthefollowingstepstoincreasethesizeoftheclusterbacktothree:

a. FromtheOpsManagerInstallationDashboard,clicktheMySQL for PCFtile.b. ClickResource Config.c. SetthenumberofinstancesforMySQL Serverto 3 .d. ClickSave.e. ReturntotheOpsManagerInstallationDashboardandclickApply Changes.

PerformManualBackupIfyoudonotwanttousetheautomatedbackupsincludedinMySQLforPCF,youcanperformbackupsmanually.

RetrieveIPAddressandCredentialsPerformthefollowingstepstoretrievetheIPaddressandcredentialsrequiredforamanualbackup:

1. FromtheOpsManagerInstallationDashboard,clicktheMySQL for PCFtile.


http://docs.pivotal.io/pivotalcf/customizing/trouble-advanced.html#ssh

http://docs.pivotal.io/pivotalcf/1-7/customizing/trouble-advanced.html#bosh-ssh

2. ClicktheStatustab.

3. LocatetheIPaddressfortheMySQLnodeunderMySQL Server.

4. IntheCredentialstab,fromtheMySQL ServerjobandMysql Admin Passwordname,obtaintheadminpassword.

ManualBackupBackupyourdatamanuallywithmysqldump .Thisbackupacquiresaglobalreadlockonalltables,butdoesnotholditfortheentiredurationofthedump.

TobackupalldatabasesintheMySQLdeployment:

$mysqldump-uadmin-p-h$MYSQL_NODE_IP--all-databases--single-transaction>user_databases.sql

Tobackupasingledatabase,specifythedatabasename:

$mysqldump-uadmin-p-h$MYSQL_NODE_IP$DB_NAME--single-transaction>user_databases.sql

ManualRestoreTheprocedureforrestoringfromabackupisthesamewhetheroneormultipledatabaseswerebackedup.ExecutingtheSQLdumpwilldrop,recreate,andrefillthespecifieddatabasesandtables.

Preparetorestore:

IfrunninginHAconfiguration,reducethesizeoftheMySQLforPCFclustertoasinglenode,followingtherestoreinstructionsabove.

LocatetheMySQLAdmincredentialsintheCredentialstab,asabove.

UsetheMySQLpasswordandIPaddresstoenablethecreationoftablesusinganystorageengine.

$mysql-uadmin-p-h$MYSQL_NODE_IP-e"SETGLOBALenforce_storage_engine=NULL"

UsetheMySQLpasswordandIPaddresstorestoretheMySQLdatabasesbyrunningthefollowingcommand.

$mysql-uadmin-p-h$MYSQL_NODE_IP<user_databases.sql

UsetheMySQLpasswordandIPaddresstorestoreoriginalstorageenginerestriction.

WARNING:Restoringadatabasedeletesalldatathatexistedinthedatabasebeforetherestore.Restoringadatabaseusingafullbackupartifact,producedby mysqldump--all-

databasesforexample,replacesalldataanduserpermissions.


https://mariadb.com/kb/en/mariadb/mysqldump/

$mysql-uadmin-p-h$MYSQL_NODE_IP-e"SETGLOBALenforce_storage_engine='InnoDB'"

TorestoreHAmode,re-configureMySQLforPCFtorunusingthreenodesinthesamewayastherestoringinstructionsabove.

IfnotrunningHAmode,it’simportanttorestartthedatabaseserver.ThisstepisnotnecessaryifscalingbacktothreeMySQLnodes.

$monitstopmariadb_ctrl$monitstartmariadb_ctrl

Formoreexamplesofmanualbackupandrestoreprocedures,seetheMariaDBdocumentation .


http://mariadb.com/kb/en/mariadb/mysqldump/#examples

Scaling Down MySQLThistopicdescribeshowtosafelyscaledownyourMySQLforPivotalCloudFoundry(PCF)clustertoasinglenode.

BydefaultMySQLforPCFisasinglenode.TotakeadvantageofthehighavailabilityfeaturesofMySQLforPCF,youmayhavescaledtheconfigurationuptothreenodes.

ChecktheHealthofYourClusterBeforescalingdownyourMySQLcluster,performthefollowingactionstoensuretheclusterishealthy.

1. ObtaintheIPaddressesofyourMySQLserverbyperformingthefollowingsteps:

a. FromthePivotalCloudFoundry(PCF)Installation Dashboard,clicktheMySQL for Pivotal Cloud Foundrytile.b. ClicktheStatustab.c. RecordtheIPaddressesforallinstancesoftheMySQL Serverjob.

2. ObtaintheadmincredentialsforyourMySQLserverbyperformingthefollowingsteps:

a. FromtheMySQLtile,clicktheCredentialstab.b. LocatetheMysql Admin PasswordentryintheMySQL ServersectionandclickLink to Credential.c. Recordthevaluesfor identity and password .

3. SSHintotheOpsManagerVM.BecausetheproceduresvarybyIaaS,reviewtheSSHintoOpsManager sectionoftheAdvancedTroubleshootingwiththeBOSHCLItopicforspecificinstructions.

4. FromtheOpsManagerVM,placesomedatainthefirstnodebyperformingthefollowingsteps,replacing FIRST-NODE-IP-ADDRESS withtheIPaddressofthefirstnoderetrievedaboveand YOUR-IDENTITY withthe identity valueobtainedabove.Whenpromptedforapassword,providethepassword valueobtainedabove.

a. Createadummydatabaseinthefirstnode:

$mysql-hFIRST-NODE-IP-ADDRESS-uYOUR-IDENTITY-p-e"createdatabaseverify_healthy;"

b. Createadummytableinthedummydatabase:

$mysql-hFIRST-NODE-IP-ADDRESS-uYOUR-IDENTITY-p-Dverify_healthy-e"createtabledummy_table(idintnotnullprimarykeyauto_increment,infotext)engine='InnoDB';"

c. Insertsomedataintothedummytable:

$mysql-hFIRST-NODE-IP-ADDRESS-uYOUR-IDENTITY-p-Dverify_healthy-e"insertintodummy_table(info)values('dummydata'),('moredummydata'),('evenmoredummydata');"

d. Querythetableandverifythatthethreerowsofdummydataexistonthefirstnode:

mysql-hFIRST-NODE-IP-ADDRESS-uYOUR-IDENTITY-p-Dverify_healthy-e"select*fromdummy_table;"Enterpassword:+----+----------------------+|id|info|+----+----------------------+|4|dummydata||7|moredummydata||10|evenmoredummydata|+----+----------------------+

5. VerifythattheothernodescontainthesamedummydatabyperformingthefollowingstepsforeachoftheremainingMySQLserverIPaddressesobtainedabove:

a. Querythedummytable,replacing NEXT-NODE-IP-ADDRESS withtheIPaddressoftheMySQLserverinstanceand YOUR-IDENTITY withtheidentity valueobtainedabove.Whenpromptedforapassword,providethe password valueobtainedabove.

$mysql-hNEXT-NODE-IP-ADDRESS-uYOUR-IDENTITY-p-Dverify_healthy-e"select*fromdummy_table;"

b. Examinetheoutputofthe mysql commandandverifythatthenodecontainsthesamethreerowsofdummydataastheothernodes.

Note:IfyouareonlyrunningtheMySQLclusterwithasinglenode,youdonotneedtoperformthesesteps.


http://docs.pivotal.io/pivotalcf/customizing/trouble-advanced.html#ssh

+----+----------------------+|id|info|+----+----------------------+|4|dummydata||7|moredummydata||10|evenmoredummydata|+----+----------------------+

6. IfeachMySQLserverinstancedoesnotreturnthesameresult,contactPivotalSupport beforeproceedingfurtherormakinganychangestoyourdeployment.IfeachMySQLserverinstancedoesreturnthesameresult,thenyoucansafelyproceedtoscalingdownyourclustertoasinglenodebyperformingthestepsinthefollowingsection.

ScaleDownYourCluster1. Deletethedummydatabase,replacing FIRST-NODE-IP-ADDRESS withtheIPaddressofthefirstMySQLservernodeand YOUR-IDENTITY withthe

identity valueobtainedabove.Whenpromptedforapassword,providethe password valueobtainedabove.

$mysql-hFIRST-NODE-IP-ADDRESS-uYOUR-IDENTITY-p-e"dropdatabaseverify_healthy;"

2. FromthePCFInstallation Dashboard,clicktheMySQL for Pivotal Cloud Foundrytile.

3. ClicktheSettingstab.

4. ClickResource Configandusethedrop-downmenutochangetheInstancescountforMySQL Serverto 1 .

5. ClickSavetoapplythechanges.


https://support.pivotal.io/

Rotating MySQL for PCF CredentialsPage last updated:

ThistopicdescribeshowtorotatecredentialsforMySQLforPivotalCloudFoundry(MySQLforPCF).IfyouarealsousingElasticRuntimeMySQL,reviewthenotesinthisprocedureinordertorotatecredentialsforbothproducts.

PrerequisitesToperformthestepsbelow,youneedtoobtainthefollowing:

1. YourrootCAcertificateina .crt file.ToretrievetherootCAcertificateofyourdeployment,runthefollowingcommand:

$curl"https://YOUR-OPSMAN-IP-ADDRESS/api/v0/security/root_ca_certificate"

2. YourMySQLforPCFrootpassword.ToretrieveyourMySQLforPCFrootpassword,navigatetotheOpsManagerInstallationDashboardandselectMySQL for Pivotal Cloud Foundry > Credentials.YourMySQLforPCFrootpasswordiscalled MysqlAdminPassword .

RotateYourMySQLforPCFCredentials1. InstalltheUserAccountandAuthentication(UAA)CommandLineInterface(UAAC).

$geminstallcf-uaac

2. Makesure uaac gemisinstalled.

$whichuaac/Users/pivotal/.gem/ruby/2.3.0/bin/uaac

3. TargetyourOpsManagerUAAandprovidethepathtoyourrootCAcertificate.

$uaactargethttps://YOUR-OPSMAN-FQDN/uaa/--ca-certYOUR-ROOT-CA.crtTarget:https://YOUR-OPSMAN-FQDN/uaa/

4. Getyourtokenwith uaactokenownerget .Enter opsman for Client ID .Pressenterfor Client secret toleaveitblank.UsetheusernameandpasswordyouusedabovetologintotheOpsManagerwebinterfacefor User name and Password .

$uaactokenownergetClientID:opsmanClientsecret:Username:adminPassword:*********Successfullyfetchedtokenviaownerpasswordgrant.Target:https://YOUR-OPSMAN-FQDN/uaaContext:admin,fromclientopsman

5. RunthefollowingcommandtodisplaytheusersandapplicationsauthorizedbytheUAAserver,andthepermissionsgrantedtoeachuserand

Note:TheOpsManagerAPIreturnsthecertificateinJSONformatwith \n foreverynewline.Removealloccurrencesof \n whenyoucopythecertificateintoa .crt file.

Note:IfyouuseElasticRuntimeMySQL,youalsoneedyourElasticRuntimeMySQLrootpassword.ToretrieveyourElasticRuntimeMySQLrootpassword,navigatetotheOpsManagerInstallation DashboardandselectMySQL > Credentials.YourElasticRuntimeMySQLrootpasswordiscalled MysqlAdminCredentials .


application.

$uaaccontext[1][https://YOUR-OPSMAN-FQDN/uaa]skip_ssl_validation:trueca_cert:/Users/pivotal/.ssh/YOUR-ROOT-CA.crt[0]*[admin]user_id:75acfdfa-9449-4497-a093-ce40ded250acclient_id:opsmanaccess_token:LONG_ACCESS_TOKEN_STRINGtoken_type:bearerrefresh_token:LONG_REFRESH_TOKEN_STRINGexpires_in:43199scope:clients.readopsman.useruaa.adminscim.readopsman.adminclients.writescim.writejti:8419c793d377429aa40eea07fb6e7686

6. Createafilecalled uaac-token thatcontainsonlythe LONG_ACCESS_TOKEN_STRING fromtheoutputabove.

7. Use curl tomakearequesttotheOpsManagerAPI.Authenticatewiththecontentsofthe uaac-token fileandpipetheresponseintoinstallation_settings_current.json .

$curl-skH"Authorization:Bearer$(catuaac-token)"https://YOUR-OPSMAN-FQDN/api/installation_settings>installation_settings_current.json

8. ChecktoseethattheMySQLforPCFrootpasswordisinthecurrentinstallationsettingsfile:

$grep-cYOUR-MYSQL-FOR-PCF-ROOT-PASSWORDinstallation_settings_current.json

9. Removetherootpasswordfromtheinstallationsettingsfile.

$sed-e's/"value":{"identity":"root","password":"[^"]*"},$"identifier":"mysql_admin$/\1/g'installation_settings_current.json>installation_settings_updated.json

10. Validatethattherootpasswordhasbeenremovedfromthe installation_settings_updated.json file.

$grep-cYOUR-MYSQL-FOR-PCF-ROOT-PASSWORDinstallation_settings_updated.json0

11. Uploadtheupdatedinstallationsettings.

$curl-skXPOST-H"Authorization:Bearer$(catuaac-token)""https://YOUR-OPSMAN-FQDN/uaa/api/installation_settings"-F'installation[file]=@installation_settings_updated.json'{}

12. NavigatetotheOpsManagerInstallation DashboardandclickApply Changes.

13. Oncetheinstallationhascompleted,validatethattheMySQLforPCFrootpasswordhasbeenchanged.RetrievethenewpasswordfromMySQL >Credentials.UsetheIPaddressfortheMySQL ProxylocatedintheStatustab.

$mysql-uroot-p-h198.51.100.1Enterpassword:WelcometotheMariaDBmonitor.Commandsendwith;or\g.[...]

Note:IfyouuseElasticRuntimeMySQL,youshouldalsorunthefollowingcommand: $grep-cYOUR-ERT-MYSQL-ROOT-PASSWORD

installation_settings_current.json

Note:IfyouuseElasticRuntimeMySQL,youshouldalsorunthefollowingcommand: $grep-cYOUR-ERT-MYSQL-ROOT-PASSWORD

installation_settings_updated.json

Note:IfyouuseElasticRuntimeMySQL,youshouldalsovalidatethattheElasticRuntimeMySQLrootpasswordhasbeenchanged.RetrievethenewpasswordfromElastic Runtime > Credentials.UsetheIPaddressfortheMySQL Proxy,locatedintheStatustab.


Running mysql-diagThistopicdiscusseshowtousethe mysql-diag toolinMySQLforPivotalCloudFoundry(PCF). mysql-diag relaysthestateofyourMySQLserviceandsuggestsstepstotakeintheeventofanodefailure.InconjunctionwithPivotalSupport,thistoolhelpsexpeditethediagnosisandresolutionofproblemswithMySQLforPCF.

InMySQLforPCF1.9.0andlater, mysql-diag isautomaticallyinstalledandconfigured.IfyouarerunningMySQLforPCF1.8.xandearlierthenyouwillneedtocreateaconfigurationfileinordertouse mysql-diag .

PrepareYourEnvironmentMySQLforPCF1.9.0andlatershipswiththe mysql-diag toolandcomeswithanautomaticallygeneratedconfigurationfile.Inversions1.9.0.andlater,youcanfind mysql-diag onthe mysql-monitor node.

IfyouarerunningMySQLforPCF1.8.xorearlierthenyoumustdownload mysql-diag andcreateaconfigurationfile.Ifyoudonothaveamonitornode,asisthecasewithsomeolderversionsofthesoftware,Pivotalrecommendsthatyouuseoneofthemysqlclusternodesinstead.

OnlycompletethedownloadandconfigurationinstructionsbelowifyouareonMySQLforPCF1.8.xorearlier.

DownloadandRunmysql-diagTodownload mysql-diag :

1. Downloadthefilelabeledmysql-diag.confattachedtotheDiagnosingproblemswithElasticRuntimeMySQLorthePivotalMySQLTile

KnowledgeBasearticle.

2. Copythatbinarytothe mysql-monitor VMwiththefollowingcommand: boshscpJOB-NAMEJOB-INSTANCE-NUMBER--uploadLOCAL-FILE-PATHREMOTE-

FILE-PATH

Runningthe boshinstances commandwilldisplaytheinformationneededtoinserttheJOB-NAMEandJOB-INSTANCE-NUMBERoptions.Formoreinformationonthe boshistances command,seetheboshdocumentation onsystemadministrationtasks.TheLOCAL-FILE-PATHoptionisthepathtowhereyouwanttolocatethemysql-diag.conffile.TheREMOTE-FILE-PATHoptionistheinitiallocationofthemysql-diag.conffile.

1. Executethemysql-diag.conffilewiththefollowingcommand:

mysql-diag-c./mysql-diag.conf

Configuremysql-diagToconfigure mysql-diag :

1. PastetheConfigurationFileTemplatebelowintoatexteditor

{"mysql":{"username":"repcanary","password":"password","port":3306,"nodes":[{"host":"10.244.7.4",},{"host":"10.244.8.4",},{"host":"10.244.9.4",}]}}

2. ReplacethepasswordswiththevaluesthatyoufindinOpsManwithintheCredentialstab.


https://discuss.pivotal.io/hc/en-us/articles/115001980967-Diagnosing-problems-with-Elastic-Runtime-MySQL-or-the-Pivotal-MySQL-Tile

https://bosh.io/docs/sysadmin-commands.html

3. CopythecompletedtemplateintothesameVMthatyoudownloadedthe mysql-diag tool,usingthe boshscp command.

4. Movetheconfigurationfiletothesamedirectoryasthe mysql-diag tool.

5. Runthefollowingcommandinordertostartthetool:

$mysql-diag-c./diag-config.json

mysql-diag-agentMySQLforPCF1.9.0andlaterwillhavethe mysql-diag-agent present.Versions1.8.xandearlierofMySQLforPCFdonothavethe mysql-diag-agent .Ifthemysql-diag-agent isnotavailable,youroutputfromthe mysql-diag toolwillnotincludethepercentageofPersistentandEphemeralDiskspaceusedbya

Host.

ExampleHealthyOutputThereplicationcanaryinversions1.7.10andearlierofMySQLforPCFdoesnotprovideareplicationAPI.ForthoseversionsofPivotalMySQL, mysql-diag

willnotbeabletodetermineifyourcanarystatusis“healthy”or“unhealthy.”

Checkingcanarystatus...Gethttp://127.0.0.1:8111/api/v1/status:dialtcp127.0.0.1:8111:getsockopt:connectionrefused

Hereisasample mysql-diag outputafterthetoolhasidentifiedahealthyclusterinaMySQLforPCFversionthatdoesnotcontainthemysql-diag-agent :

Checkingclusterstatusofmysql/a1at10.0.16.44...Checkingclusterstatusofmysql/c3at10.0.32.10...Checkingclusterstatusofmysql/b2at10.0.16.45...Checkingclusterstatusofmysql/a1at10.0.16.44...doneCheckingclusterstatusofmysql/c3at10.0.32.10...doneCheckingclusterstatusofmysql/b2at10.0.16.45...done+------------+-----------+-------------------+----------------------+--------------------+|HOST|NAME/UUID|WSREPLOCALSTATE|WSREPCLUSTERSTATUS|WSREPCLUSTERSIZE|+------------+-----------+-------------------+----------------------+--------------------+|10.0.16.44|mysql/a1|Synced|Primary|3||10.0.32.10|mysql/c3|Synced|Primary|3||10.0.16.45|mysql/b2|Synced|Primary|3|+------------+-----------+-------------------+----------------------+--------------------+Idon'tthinkbootstrapisnecessaryCheckingdiskstatusofmysql/a1at10.0.16.44...Checkingdiskstatusofmysql/c3at10.0.32.10...Checkingdiskstatusofmysql/b2at10.0.16.45...Checkingdiskstatusofmysql/a1at10.0.16.44...dialtcp10.0.16.44:getsockopt:connectionrefuseCheckingdiskstatusofmysql/c3at10.0.32.10...dialtcp10.0.16.44:getsockopt:connectionrefuseCheckingdiskstatusofmysql/b2at10.0.16.45...dialtcp10.0.16.44:getsockopt:connectionrefuse

ExampleUnhealthyOutputIntheeventofabrokencluster,running mysql-diag outputsactionablestepsmeanttoexpeditetherecoveryofthecluster.Belowisasample mysql-diag

outputafterthetoolidentifiedanunhealthyclusterinaMySQLforPCFversionthatdoesnotcontainthe mysql-diag-agent :


Checkingclusterstatusofmysql/a1at10.0.16.44...Checkingclusterstatusofmysql/c3at10.0.32.10...Checkingclusterstatusofmysql/b2at10.0.16.45...Checkingclusterstatusofmysql/a1at10.0.16.44...dialtcp10.0.16.44:getsockopt:connectionrefusedCheckingclusterstatusofmysql/c3at10.0.32.10...dialtcp10.0.32.10:getsockopt:connectionrefusedCheckingclusterstatusofmysql/b2at10.0.16.45...dialtcp10.0.16.45:getsockopt:connectionrefused

+------------+-----------+-------------------+----------------------+--------------------+|HOST|NAME/UUID|WSREPLOCALSTATE|WSREPCLUSTERSTATUS|WSREPCLUSTERSIZE|+------------+-----------+-------------------+----------------------+--------------------+|10.0.16.44|mysql/a1|N/A-ERROR|N/A-ERROR|N/A-ERROR||10.0.16.45|mysql/b2|N/A-ERROR|N/A-ERROR|N/A-ERROR||10.0.32.10|mysql/c3|N/A-ERROR|N/A-ERROR|N/A-ERROR|+------------+-----------+-------------------+----------------------+--------------------+

Checkingdiskstatusofmysql/a1at10.0.16.44...Checkingdiskstatusofmysql/c3at10.0.32.10...Checkingdiskstatusofmysql/b2at10.0.16.45...Checkingdiskstatusofmysql/a1at10.0.16.44...dialtcp10.0.16.44:getsockopt:connectionrefusedCheckingdiskstatusofmysql/c3at10.0.32.10...dialtcp10.0.32.10:getsockopt:connectionrefusedCheckingdiskstatusofmysql/b2at10.0.16.45...dialtcp10.0.16.45:getsockopt:connectionrefused

[CRITICAL]Thereplicationprocessisunhealthy.Writesaredisabled.

[CRITICAL]Runthedownload-logscommand:$download-logs-d/tmp/output-n10.0.16.44-n10.16.45-n10.0.32.10Forfullinformationabouthowtodownloadandusethedownload-logscommandseehttps://discuss.pivotal.io/hc/en-us/articles/221504408

[WARNING]DonotperformthefollowingunlessinstructedbyPivotalSupport:-Donotscaledowntheclustertoonenodethenscaleback.Thisputsuserdataatrisk.-Avoid“boshrecreate”and“boshcck”.TheseoptionsremovelogsontheVMsmakingithardertodiagnoseclusterissues.


table of contents - docs.pivotal.iodocs.pivotal.io/archives/mysql-docs-1.7.pdf... , and service...

Documents