Slide 1

Everyday DBA for Progress OpenEdgeWhat You Need to Know, and When to Call for HelpDavid EddyLinx Tailored Development Services2010 Linx ADS Pty Ltd 2What is the DBA responsible for?AvailabilityUsers must be able to connect to, read from, and update the database whenever they need to.2010 Linx ADS Pty Ltd The database must exist and be up to date and consistentIt must be on-line when users need to connectUsers must be able to connectDB operations should be fast enough, and other applications should not be unduly impacted3Sounds simple... What does it mean?2010 Linx ADS Pty Ltd Ensuring user actions are monitored and/or controlledProtecting the database from unauthorised accessComplying with legislative requirements or other standards4It may also involve...2010 Linx ADS Pty Ltd ExistenceUptimeConnectivitySecurityAuditingPerformance5Agenda2010 Linx ADS Pty Ltd Some tasks need to be done weekly, daily, or sometimes even hourlyYou should know how to do these, and when to do themSome tasks only need to be done occasionallyFor most users, its easy to forget how to do theseCall in the expertsDont panicProgress databases largely require no administration once set up (for small to medium users).6Hang on! Thats quite a bit!2010 Linx ADS Pty Ltd The database must exist, be current, and be usable.7Existence2010 Linx ADS Pty Ltd Your database must existIf it goes away, you must be able to get it backYour database must be currentLost data costs moneyYour application must exist and be currentWhat challenges are there and how will you address them?8ExistenceWhere is your database today?2010 Linx ADS Pty Ltd Backup everybody does itOn-line, offline, mirror splitRestore can you do it?Have you actually tested your backup recently?Clear, available documentationWhat if youre sick and the restore has to be done by someone else?What about the application?Or the system data? User password database?9ExistenceThe tender art of backing up2010 Linx ADS Pty Ltd For small and medium sites:KISS: Use probkup (online or offline) probkup [online] dbname backupfilename comThen back up backup file, applications, and critical system files to removable media (and take them offsite)Large sites may wish to use mirror split (i.e. array-based backup)proquiet dbname C enable (pauses update activity)Mirror split or snapshot commenceproquiet dbname C disable10Backup techniques2010 Linx ADS Pty Ltd Technique will correspond with backupFor probkup-based backups:Restore DB backup file (and possibly other stuff) from removable media into scratch areaRestore database:prorest dbname backupfileCopy any other needed parts (e.g. application code) to proper locations11Restoring a backup2010 Linx ADS Pty Ltd Application corresponds with databaseChanges to DB structure, or application code changes, may mean restored DB will not run with current application codeNot usually an issue in practiseBut needed anyway in case of total disk loss12Why application?2010 Linx ADS Pty Ltd If you have to restore from backup, youve lost everything thats happened sincePaper-based operations? No problem (just $ to rekey)Telephone or web-based? Youre in trouble!Solution: After-image loggingTrivial to implementSelf-managing in OpenEdge 10 and higherNo excuse not to!13Existence II: Preventing data lossThe case for after-imaging2010 Linx ADS Pty Ltd Add after-image extents to DB if you need to5 variable-sized extents ok for most small-medium sitesCreate a directory to hold archived after-image dataEnable after-imaging14Implementing after-imaging (AI)Its easy!2010 Linx ADS Pty Ltd Create a file e.g. ai.st to describe AI extents (one line per extent needed)Add to database

15Implementing AIAdding AI extents# AI extents for somedba /path/to/directorya /path/to/directory...a /path/to/directoryprostrct {add|addonline} dbname ai.st -validateprostrct {add|addonline} dbname ai.st2010 Linx ADS Pty Ltd 16Implementing AIEnabling after-imaging in the database# Shut down DB

# Enable after-imaging and AI log managementrfutil dbname C mark backeduprfutil dbname C aimage beginrfutil dbname C aiarchiver enable

# Modify db startup to include argument# -aiarcdir /path/to/aiarchive/dir

# Restart DB2010 Linx ADS Pty Ltd What planning do you need to make for disasters?What if theres a fire at the office?Or total loss of electrical power?If youre a multi-site operation, you should consider DRWould being able to run on an off-site system benefit the business?17Existence III: Up in smokePlanning for business continuance2010 Linx ADS Pty Ltd Key part of Business Continuance planningBut only a partOpenEdge ReplicationTrigger-based replicationArray-based replicationLog shippingOften an area needing expert advice18ReplicationA complex topic2010 Linx ADS Pty Ltd Plans are of no use if nobody knows of themYou might be sick or injured (or in the Bahamas)Clear, written checklists and instructionsStore a copy with your off-site backupsInclude exact commands to use in case must be performed by unfamiliar personnel19Information vacuum sucks2010 Linx ADS Pty Ltd Take backups (and manage them properly)Test restoreTest fact that you can read your backupsTest documented procedure in case youre not thereDocument how long restore takesDocument disaster scenarios and recovery proceduresSocialise existence of documentation20DBA tasks for: Existence2010 Linx ADS Pty Ltd UptimeThe system needs to be available whenever the users need it.But it need not be available when they dont need it.21Uptime2010 Linx ADS Pty Ltd Most businesses have modest needs, e.g. 5x16 or 6x24e.g Mon-Fri, 7am-11pm or Mon 6am -> Sat 9pmVery few business are genuinely 24x7Exceptions are typically web-response systemsLarge businesses24-hour (petrol station, convenience store, Crown Casino)Nobody will care if you bust a gut making the system available when they dont need itUptime costs money how much do you need?22Uptime requirementsIdentify your needs2010 Linx ADS Pty Ltd Do you need to be able to remotely administer your database?Use free OpenEdge Explorer if 10.1C+Should your database auto-restart if it goes down?Use OpenEdge Management or bespoke scripting if needed/appropriateWhat non-database services are needed?AppServers, WebSpeed, batch procedures, interfaces e.g. Sonic23Other uptime requirements2010 Linx ADS Pty Ltd Consider using AdminServer for database administrationUse is required anyway for WebSpeed, AppServer, etc so you may as well use it.Database configuration stored in $DLC/properties/conmgr.propertiesAdministered using dbman commandsDatabase have nicknames instead of pathnamesPreferred method if using OpenEdge Explorer or Management (i.e. Web-based admin interface)Default for OpenEdge 10.2B+24AdminServer2010 Linx ADS Pty Ltd Human errorLack of DB free spaceLack of log free spaceLack of filesystem free spaceReaching DB extent size limitsReaching area size limitsHard kill of client processes (sometimes)Silly stuff:Removing .lk fileChanging host nameSystem crash25What causes uptime problems2010 Linx ADS Pty Ltd Space/size issues are your primary uptime riskAddress by combination of:Risk mitigationMonitoring 26Free space management2010 Linx ADS Pty Ltd Enable large file handling (for Enterprise Server)Ensure all storage areas have variable extentTest whether on-line extent add works for youMight not if clients dont have DB extent file permissionsEstablish a usage baselineMake sure AI logs are archived/emptiedMake sure BI log doesnt get too bigMonitor free space at all levelsBe aware of area maximum size given recs/block setting (_AreaThreshold VST)27Free space: Risk mitigation strategies2010 Linx ADS Pty Ltd Monitor database free space for each storage area independently (including log areas)prostrct statistics_AreaStatus VSTCheck for variable extent size and growthMonitor filesystem free spaceIn filesystems containing variable-size extents28Free space: Monitoring2010 Linx ADS Pty Ltd Just knowing kB free is often uselessHow many kB per week are consumed?Establish a usage baselineDescribe free space in terms of number of weeksLong-term average consumption7-day moving average consumptionAlso monitor if 7-day average moves too far from long-term average29Monitor usefully2010 Linx ADS Pty Ltd Automated monitoring is most useful Especially when combined with exception reportingCan be hard to do if youre not systems-literateCall in the experts to set it up for youLearn to interpret output30Automate2010 Linx ADS Pty Ltd Less than 2 weeks free space in any data areaRemember: When one area fills, the whole DB is full regardless of how much free space there is in other areasVariable extent >1GBUnless you have large file handling enabledVery little (4GB) can have extended crash recovery timeOnly one empty AI log extent31Bad things!Hot-spots to watch out for2010 Linx ADS Pty Ltd Long-running transactionsCause BI log growthNo empty AI log extentsBackups will fail until correctedLog space exhaustion prevents updates and shuts DB2 or more locked AI log extentsReplication has probably stoppedUnreplicated data prevents AI log space reuseMeasure with rfutil db C aimage extent list32More bad things...a little more indirect, this time2010 Linx ADS Pty Ltd Supported in most V9 and all OE10 portsEnterprise Server only (not Workgroup or Personal)Filesystems need LFH also (beware AIX!)How to tell whether its enabled:You have an extent larger than 2GBUse proutil db C describe (10.1C+)Look in the log file (after multi-user session begin) for large-file handling enabled33Large file handling for OpenEdge2010 Linx ADS Pty Ltd Monitor free space in:All DB areasFilesystemsMonitor size of BI logMonitor number of empty AI log extentsWatch for variable-size extent growthKnow how to add extentsOn-line, ideally34DBA tasks for: Uptime2010 Linx ADS Pty Ltd The database should have enough slots for users to be able to connect.35Connectivity2010 Linx ADS Pty Ltd Remote vs. Local connectionsMaximum total DB connectionsServer poolsLock table size36ConnectivityIs primarily concerned with...2010 Linx ADS Pty Ltd You should use local connections where feasibleBest performanceMinimum resource usageRemote connections are useful for:Off-system clientsEnsuring database stability if client termination causes problemsSegregating users into pools (access denial, user count control, performance)Performance in specialised circumstances (e.g. highly-asymmetric NUMA)37Remote vs. Local connections2010 Linx ADS Pty Ltd Presence of S parameter in causes remote connectionBeware, this parameter is needed on DB server to allow remote connectionsUsage of common parameter lists for client & server is therefore a Bad Idea38Remote connectionsWhat makes a connection remote2010 Linx ADS Pty Ltd Number of connections vs. MaximumGlobal for databasePer server poolNumber of locks vs. Maximum39Monitoring connectivityWhat to monitor2010 Linx ADS Pty Ltd PromonR&D, 1, 13: Overall connections, overall lock usageR&D, 1, 2: Users per server, number of free slots per server, number of free server slotsVSTS: _Connect, _Server, _DbStatus40Monitoring connectivityHow to monitor2010 Linx ADS Pty Ltd By default, 1 server pool onlyMultiple pools recommended if you frequently use SQL-92Multiple pool setup can be hard to getStraightforward but frequently misunderstood and only occasionally neededAn example where calling the experts can save you some headaches41Server pools2010 Linx ADS Pty Ltd Monitor usage of constrained resources Global connections, server connections, locksMake sure there is sufficient headroom to allow for burst usageConnections: At least 10% freeLocks: Limit ideally at least 1.3x max usageYou may have connection limit policy to enforceConnections are not necessarily licences42DBA tasks for: Connectivity2010 Linx ADS Pty Ltd OpenEdge Management (http://web.progress.com/en/openedge/openedge-management.html)Monitoring, trending, alerting, job/report execution, remote monitoring, files, cpu, disk, networkProTop (http://www.greenfieldtech.com/articles/protop.shtml)Free, monitoring onlyMonitoring infrastructure kits E.g. Nagios, Hyperic43Monitoring ToolsSome people prefer to buy their tools2010 Linx ADS Pty Ltd Many things to many people44Security2010 Linx ADS Pty Ltd Authentication (identification)Authorization (limitation of action)Encryption (prevention of snooping)Integrity (detection/prevention of attacks)Key part of larger concernsAuditingLegislative requirements45Security means many things2010 Linx ADS Pty Ltd Know how legislation or Best Practise affects youAus: TFN managementUSA: HIPPA, SoX, P.I. Data (Mass.)Europe: Personal identification dataCan be part of a competitive positionWeb commerce

462010 Linx ADS Pty Ltd Filesystem permissions for DB containersDisallow deletionsUse _User table for key user identification e.g. DBATake advantage of pre-compiled permissions to limit interactive accessibility of DB47Basic tasks2010 Linx ADS Pty Ltd Use CLIENT-PRINCIPAL and use it to assert trusted identity in AppServersSingle sign-onEncryption (field & table level)SSL plumbingSystem hardening

Many options, often with substantial tradeoffsGet expert advice

48More advanced stuff2010 Linx ADS Pty Ltd Mostly about setup, procedure, and sometimes programmingLittle maintenance neededDifficult to maintain skill49DBA tasks for security2010 Linx ADS Pty Ltd Who did what and when... inarguably50Auditing2010 Linx ADS Pty Ltd Recording of changes in DBOften triggered by legal requirementsSometimes pure paranoiaMaintenance of queryable archiveInvestigate prior questionable activitySpot checking by auditorsNon-repudiableMight have to stand up to challengeAlso useful for preventing black hats hiding tracks51AuditingAuditing is...2010 Linx ADS Pty Ltd Tradeoff between completeness and performanceFully-enriched audit trail requires code modificationsPlanning is required to guide recording of eventsNeed to think about how archive might be queried or reported upon52Complications2010 Linx ADS Pty Ltd Keep audit data from building up in live DBMove to audit archive DB regularlyMaintain auditing rolesPlanning of events and contextImplementing changes to audit policySpecialised help may be needed in some areasSSO integrationKey techniques e.g. copying identity to AppServersHelp with planning events and context dataSetup of archive database53DBA tasks for auditing2010 Linx ADS Pty Ltd How fast to you want to go?54Performance2010 Linx ADS Pty Ltd Database tuningSystem tuning and configurationSchema designQuery formation and index usagePerformance impact of DB structural elementsUsage patternsCode efficiencyClient tuning and parametersConnection modeHardware and network configurationBusiness changesUser expectations55Performance is multi-facetedAn incomplete list...2010 Linx ADS Pty Ltd 56A complex systemA complex system is a system composed of interconnected parts that as a whole exhibit one or more properties not obvious from the properties of the individual parts.Snowballing declineDatabases can exhibit other counterintuitive behaviours2010 Linx ADS Pty Ltd Most aspects of performance need expert adviceSome tasks can be usefully managed by DBA:Gradually increasing execution times and/or sustained record reads >50,000/sec: Sign of bad indexing (code fault)High I/O = too little buffer memory (DB config)If adding memory doesnt fix it, call the experts it can be a complex topic!57DBA tasks in performance2010 Linx ADS Pty Ltd HourlyLong-running transactions and BI size/free spaceConnections, lock table usageDailyStorage area and filesystem free space (expressed as weeks worth of consumption)Growth of variable-sized extentsBackupAI extent archive managementAs neededPerform test of backup restore processNumerous configuration and setup tasksDocument your DR plans and promulgate58Summary of TasksTheres quite a bit you can and should do2010 Linx ADS Pty Ltd Investigate performance and capacity issuesTake baseline load data for long-term trend analysisReview your system for best practiseHelp plan implementation of new featuresHelp solve automation problemsExecute seldom-performed tasksTrain or mentor you59Call the experts to...2010 Linx ADS Pty Ltd Know your business!Understand business requirements of systemsPlan to meet objectivesImplement appropriate measures to suitKnow what you need, and what you dont need60Most importantly...2010 Linx ADS Pty Ltd This presentation largely canvassed topics rather than detailIf you want detailed instruction, talk to me about training and mentoringIf theres demand, might develop a 1 or 2 day course covering the mechanics of these topics61A Final Thought2010 Linx ADS Pty Ltd Questions and Answers62

2010 Linx ADS Pty Ltd 62Thanks for your time

632010 Linx ADS Pty Ltd So what do you think ?

Action plan schedule next meeting63