1

Automating the Cube Build/Publish ProcessUsing CognosScript

Automating the Cube Build/Publish ProcessUsing CognosScript

Brian Morris,Brian Morris, CBIP

BI-340 – Cutting Costs in Half by Automating the Cube Build Process Cognos Forum 2004 (June 27 -30, Orlando, Florida)

This session is Automating the Cube Build/Publish process using Cognos Script

Presented by Brian Morris

CBIP – Certified Business Intelligence Professional – TDWI/ICCP (The Data Warehousing Institute / Institute for the Certification of Computing Professionals)

2

Brian MorrisBrian Morris

• Data Warehousing Advisor

• Kerr-McGee Shared Services, LLC.

• Over 1,100 Cognos users & growing

• Building & Publishing 60+ Cubes Daily

• Cognos Products in Production: PPES, IWR, CQ, Viz, NoticeCast, CMM, Web Services, Planning, Consolidation. ReportNet in Dev

• Data Warehousing Advisor

• Kerr-McGee Shared Services, LLC.

• Over 1,100 Cognos users & growing

• Building & Publishing 60+ Cubes Daily

• Cognos Products in Production: PPES, IWR, CQ, Viz, NoticeCast, CMM, Web Services, Planning, Consolidation. ReportNet in Dev

My name is Brian Morris and I work as a Data Warehousing Advisor for Kerr-McGee within the Data Warehouse group. My key responsibility is to work with all things Cognos. My unofficial title is “Cognos Evangelist” as I am cons tantly looking for new opportunities to add value using our corporate standard information delivery toolset.

We are currently Building and Publishing 64 cubes to our production environment each day.

My current environment has over 1,100 named users running PowerPlay Web (PPES), IWR, CQ, Viz, CMM, NoticeCast, Web Services, Planning and Consolidation all running on the Microsoft Windows 2000 platform.

3

RequirementsRequirements

Automated Cube Build / Publish

• 24/7 BI Production Environment

• Build & Publish Cubes based on:• Scheduled Date & Time• Flag Files • SQL Criteria

• Email Notification of:• Success• Called-Off• Failure

Automated Cube Build / Publish

• 24/7 BI Production Environment

• Build & Publish Cubes based on:• Scheduled Date & Time• Flag Files • SQL Criteria

• Email Notification of:• Success• Called-Off• Failure

Plan for Success! - After the first six cubes were developed and deployed to Production we experienced a dozens of new request for development. While we could manage the daily building and publishing of our initial six cubes we knew that managing more than a few more would quickly max out our existing support staff. We needed an automated solution.

Based on what we knew from our current environment and the request we had received we created the following minimum requirements:

•24/7 BI Production Environment – With BI customers around the world we needed to provide updated information as soon as it was available. Waiting until US based support staff came into the office was not an option so an automated solution would have to be delivered.•Build & Publish Cubes based on:

•Scheduled Date & Time – For a very limited set of cubes a simple date/time schedule could be established. (i.e. every morning at 2:30am).•Flag Files – The ability to kick off the cube build as soon as the ETL (Extract Transform & Load) process had completed was the standard in our organization. Using the average ETL completion time and adding a “window” for padding had proven to be unreliable as frequently processes ran longer than expected. While this was mostly due to the maturity of our DW (Data Warehousing) initiatives it was still a fact of life that had to be compensated for. As a result we implemented the creat ion of “Flag Files” with specific names in designated directories at the completion of each ETL process. We needed an automated solution that would check for these “Flag Files” on a periodic basis and kick off the cube builds when they were found. •SQL Criteria – An alternative to the Flag File approach was to generate the cubes based solely on a SQL query against a process database. The database would be populated with ETL success or failure indicators which would then serve as the criteria for whether or not to build and publish cubes. At a specified time or times each day the cube build process would run a query against the database and if the query returned a specific value, the cube build would start. If the condition was not met then the cube build would not commence.

•Email Notification for:•Success – Send an email notification to a group or groups showing that the cube was built successfully along with the Cube Build Macro log file showing specifics of when the build started and finished. •Called-Off – Send an email notification to a group or groups showing that the cube build was NOT attempted due to the Flag File or SQL Condition not being met.•Failure – Send an email notification to a group or groups immediately when a cube build does not complete successfully for any reason. Include in the email as much information as possible as to the cause of the failure. This email is also routed to the pagers of our 24/7 support staff.

4

SolutionSolution

Automated Cube Build / Publish

• CognosScript

ØBundled with Cognos Transformer

ØProcedural Based

ØCompletely Customizable

ØExamples available in Knowledge Base

ØCan be ran Unattended in MS Scheduler

Automated Cube Build / Publish

• CognosScript

ØBundled with Cognos Transformer

ØProcedural Based

ØCompletely Customizable

ØExamples available in Knowledge Base

ØCan be ran Unattended in MS Scheduler

Our solution was to use the CognosScript macro language provided with Cognos Transformer.

We chose this option because it was a simple procedural based language very similar to Visual Basic.

Because it is procedural based language we would have complete control to customize our scripts to exactly match our requirements.

There are lots of examples of how to implement this solution ava ilable in the documentation and the Cognos Knowledge Base.

And the script can be ran unattended using batch files kicked off from Microsoft Scheduler.

5

Files and StructureFiles and Structure

• Master Script, 12 Steps – ~ 2700 lines

• LogToDatabase – ~ 1200 lines

• SendCDOEmail – ~ 100 lines

• NewDateTimeStamp – ~ 100 lines

• DeleteFiles – ~ 100 lines

• Master Script, 12 Steps – ~ 2700 lines

• LogToDatabase – ~ 1200 lines

• SendCDOEmail – ~ 100 lines

• NewDateTimeStamp – ~ 100 lines

• DeleteFiles – ~ 100 lines

The script I came up with has grown from less that 100 lines two years ago to well 4,000 lines today.

To accomplish this I have broken it down into a master script that calls 4 external scripts. These external scripts require special declaration and a “SBH” file extension. The are in fact simple MAC files with the extension changed. The CognosScript engine compiles all of these together at run time.

All of these sub scripts are called multiple times by the master script except for the LogToDatabase script. It is only called once during the 10th step in the master macro.

6

Process FlowProcess Flow

Step 1 – Rename Old Macro Log File

Step 2 – Flag File Processing Start

Step 3 – SQL Condition Processing

Step 4 – Rename Old Transformer Cube Build Log

Step 5 – Backup the Existing Cubes

Step 6 – Delete Expired Logs and MDC Files

The cube build script is divided into eleven steps. These steps follow a logical sequence and are all controlled by variables declared and set at the beginning of the script.

The steps are:

•Step 1 – Rename Old Macro Log File – To maintain a history of previous cube builds all previous log files generated by this macro are renamed. They are renamed by simply adding the file date/time stamp to the end of the file name. After renaming the old log file, a new one is created to log this current cube build attempt.•Step 2 – Flag File Processing Start – If this cube build is contingent on the presence of a Flag file then this step will run. In this step a simple “Dir” command verifies the presence of the Flag File. If found the Flag File name is appended with “_InProcess” to indicate that the cube build process has started and to prevent additional cube build attempts. After the flag file is renamed the cube build process will continue. In the event that the Flag File is not found then the cube build process will be “Called-Off.”•Step 3 – SQL Condition Processing – If this cube build is contingent on meeting the conditions of a SQL Query then this step will run. In this step a database connection and SQL Query is ran using a System level ODBC database connection. If the query returns a “Y” or whatever other return value you are testing for the query is successful and the cube build process will continue. Otherwise the process will be “Called-Off.”

•Step 4 – Rename Old Transformer Cube Build Log File – This step simply renames the old Transformer cube build logs byappending their file date/time stamp to the end of their file name.

•Step 5 – Backup the Existing Cubes – This moves any old cubes that this Transformer model will be building and moves them to a backup file location. During this file copy/move the cubes file date/time stamp is appended to their file name to allow multiple copies of the same mdc file to be maintained as a backup. This step is important because it will allow you to roll-back to older copies of the cubes if necessary. •Step 6 – Delete Expired Logs and MDC Files – This steps deletes log files and mdc files in the backup directory after they exceed the number of days set in the macro as intDaysToKeep. This is basically a cleanup process that ensures your hard drive does not fill up with old MDC files while still allowing you to keep however many days of old files in reserve. The default setting is 10 days.

7

Process Flow (continued)Process Flow (continued)

Step 7 – Backup the Transformer Model

Step 8 – Build from Transformer Model

Step 9 – Cube Validation

Step 10 – Cube Build Move to Production

•Step 7 – Backup the Transformer Model – This backs up the Transformer model if it has a different date/time stamp from any already backed up models. These backups are never deleted and allow you to “Roll Back” to any previous model in the event you need to. This is another safety mechanism and is not intended to replace a good change control process.

•Step 8 – Build from Transformer Model – This step actually builds the cubes from the Transformer model. There are two different options for these cube builds, native Transformer object or External Process. The default is to use the native Transformer process as it allows for additional validation of cube build success or failure. •Step 9 – Cube Validation – This step is an additional set of checks to validate that the cube was built successfully without physically opening the mdc file. To do this the Transformer log file is opened and each line is evaluated against a set of criteria that include eight fatal “TR” code errors. If the log file does not contain any of those errors, passes additional validation and contains the phrase “Timing, TOTAL TIME (CREATE CUBE),” then the build is deemed successful. Otherwise the build is labeled a failure and proper notifications are generated. This step also includes a call to another process that will log the contents of the Transformer log file into a SQL Server database for later analysis and reporting. •Step 10 – Cube Build Move to Production – This step physically moves the cubes from the cube build direct ory to the folder location used by PowerPlay Enterprise Server (PPES) to respond t o PowerPlay Web requests. In this process the cubes are PPES disabled, moved, move verified and then PPES enabled.

8

Process Flow (continued)Process Flow (continued)

Step 12 – Flag File Processing Finish

On Macro Completion – Send Email Notifications

• Success – Macro & Transformer Logs

• Called-Off – Macro Log Only

• Failure – Macro & Transformer Logs

On Macro Completion – Send Email Notifications

• Success – Macro & Transformer Logs

• Called-Off – Macro Log Only

• Failure – Macro & Transformer Logs

Step 11 – SQL Update of Flag Record

•Step 11 – SQL Update of Flag Record – Performs an Update of the Record used as the Trigger in the SQL Query to build the cube. This prevents the cube from being built repeatedly.

•Step 12 – Flag File Processing Finish – If this cube build used a Flag File as a condition to build, the Flag File “_InProcess” text is removed and the file is left with a simple file date/time stamp appended to it.

At the completion of the macro email notifications are sent.

•Success – Notify by email of a successful cube build an publish

•Called-Off – Notify by email that the cube build was not attempted due to missing Flag File or SQL Query Conditions not Met

•Failed – Notify by email and pager that the cube build failed and request immediate attention from Admin staff

9

Success EmailSuccess Email

This is a sample of what our Success email notification messages look like.

Both the Macro Cube Build log file and the Transformer Cube Build log file are attached to the email.

The email shows some additional factual information that help recipients understand what was done, when it was done and if it was successful. It also contains information that helps the recipient get additional help if needed.

This is only sent out when:

•The Transformer Object built the cube and did not return any errors

•The Transformer log file does not contain one of eight different Transformer “TR” error codes

•The Transformer log file contains the phrase “Timing, TOTAL TIME (CREATE CUBE),” which indicates success•The physical MDC file is present in the Cube Build folder.

In our company this email message is set to the email only groups. It is not sent to the email distribution list that includes pagers.

10

Called-Off EmailCalled-Off Email

This is a sample of what our Called-Off email notification messages look like.

Only the Macro Cube Build log file is attached to the email. Since the cube build was never attempted there is no Transformer log to attach.

The email shows some additional factual information that help recipients understand what was done, when it was done and if it was successful. This information will include the complete Flag File path or SQL Query condition that caused the process to be called-off. It also contains information that helps the recipient get additional help if needed.

This is only sent out when:

•The Flag File used as a condition to build and publish the cube cannot be found

OR

•The SQL Query processed successfully but the condition for Success was not Satisfied

In our company this is not a critical failure so the message is set to the email only groups. It is not sent to the email distribution list that includes pagers.

11

Failed EmailFailed Email

This is a sample of what our Failure email notification messages look like.

Both the Macro Cube Build log file and the Transformer Cube Build log file are attached to the email.

The email shows some additional factual information that help recipients understand what failed, when it failed and as much specific information as possible about why the failure occurred. The log files being attached may also help the recipient get additional information about the failure.

This is sent out when:

•A Connection to the SQL Query Database is not successful

•The SQL query returns no rows or columns

•The Transformer object returns an error during its cube build.

•The Transformer log contains one of the eight Fatal Transformer “TR” errors.

•The Transformer log does not conclude with “Timing, TOTAL TIME (CREATE CUBE),” •The physical MDC file that should have been created by the cube build cannot be found

•The Cube cannot be disabled in the PowerPlay Enterprise Server

•The Cube cannot be copied into the PPES Cube folders

•The Cube file date/time stamp in the PPES Cube folder is not new er that the file date/time stamp in the cube build folder.

•The Cube cannot be Enabled in the PowerPlay Enterprise Server

•Any other unexpected failure in the cube build macro.

This is a critical failure and the email message is sent out to email distribution lists that include pagers.

12

PrerequisitesPrerequisites

For this script to operate the following must be installed:

• Windows NT 4.0 SP5 or 2000 Server SP1• PowerPlay Transformation Server (trnsfrmr.exe)

• CognosScript Editor (RunMac32.exe)

• PowerPlay Enterprise Server Administration Tools (PPAdminTool.exe)

• SMTP (Simple Mail Transport Protocol) Enabled on Server (KB: 117908)

For this script to operate the following must be installed:

• Windows NT 4.0 SP5 or 2000 Server SP1• PowerPlay Transformation Server (trnsfrmr.exe)

• CognosScript Editor (RunMac32.exe)

• PowerPlay Enterprise Server Administration Tools (PPAdminTool.exe)

• SMTP (Simple Mail Transport Protocol) Enabled on Server (KB: 117908)

This script is designed to operate on a Microsoft Windows platform. It has been run successfully on both a Windows NT and Windows 2000 Server operating system.

It is important that Transformer be loaded locally on the box. This script has not been tested for remote calling of Transformer objects.

The CognosScript editor, RunMac32.exe, must also be loaded locally on the box. It is usually loaded along with Transformer but I would physically verify its existence and location prior to running the script.

The PowerPlay Enterprise Server Administration tool must also be loaded locally on the machine as this is the tool that allows PPES cubes to be disabled and enabled during the cube move to production process.

If you will be sending email from this machine you must have SMTP enabled and the CDONTS.dll present. For more information on this refer to the Cognos KnowledgBase article 117908.

13

Macro HeaderMacro Header

This is the header section of the main macro script file. In the description you will find some very helpful information about the overall capabilities of this macro and a high level overview of how these are accomplished.

•This macro will build a single transformer model. This Transformer model may create up to eight Cubes (MDC files) which will be automatically published. To enable building of multiple cubes simply add the appropriate entries to the cube list variable. Each cube (MDC file) entry must be separated by the "|" character. Then set the variable "intCubeCount" to the correct number of cubes. The process will then loop "intCubeCount" number of times.

•In addition to simply building the cubes and publishing them, so me detailed logging capabilities have been added. These logs are helpful in troubleshooting failed cube builds and are also automatically emailed out for success and failure.

•The capability to build cubes conditionally based on SQL Queries or on the presence of a specific file being present in a specific location has been put into place in this macro. Notethat the SQL may bring back more than one record, however this macro will only check and log the first two for conditional cube building.

•The ability to Disable and Enable cubes from within this macro has also been added. This will eliminate the need to call an external batch file to execute the PowerPlay Enterprise Server Connect and Kill/Enable commands from a command line within the batch file. Now all functions needed to fully automate cube building and publishing are located within this one file.

•Lastly, the macro will check each of up to three Production PowerCube directories after the newly built cube (MDC files) has been moved to them. A check will be made that compares the Production Cube's Date/Time stamp to the Cube Build folder Cube's Date/Time stamp. The Production Cube's Date/Time stamp must be >= the Cube Build folder Cube's Date/Time stamp or the process will fail.

14

Step HeaderStep Header

Each step also has its own header section that list individual tasks performed.

Within the step these tasks are numbered which greatly reduces support and troubleshooting.

15

Setting of VariablesSetting of Variables

• All Variables for this script are set in one location (Except Email Body)

• Variables are grouped by Category• File / Folder Location• Cube Build Conditional• Email Notification Recipients• PowerPlay Enterprise Server

• All Variables for this script are set in one location (Except Email Body)

• Variables are grouped by Category• File / Folder Location• Cube Build Conditional• Email Notification Recipients• PowerPlay Enterprise Server

A common problem in scripts is that the setting of variables is spread out throughout the script. I have done what I can to eliminate this by setting the values of all variables in a single location at the top of the main subroutine.

The only exception to this is the email body text that is set in different steps throughout the script. This is done because of the early binding of variables and the need to include their contents in the emails.

Within the Variable Declarations section each set of variables is grouped by functional category as shown in the slide.

16

Variable DeclarationVariable Declaration

Just to get a visual image of how the top of this section looks like in the script. Most of your settings will be made above the line highlighted in yellow.

In this particular section you can see the settings for the Location Variable declarations. As shown above, I would strongly suggest that you use strictly UNC based paths for portability.

Here is a high level overview of some of these variables:•Model Path - This is the physical location where the MDL or PYI files are located.•Log Path – This is the physical location where the Cube Build Macro log and the Transformer log will be created.•Cubes Path – This is the physical location where the cubes built by the Transformer process will be created•Backup Path – This is the physical location where the models and old cubes will be placed.•Transformer Path – This is the physical location where the trnsfrmr.exe is located.•Publish Paths – This is the physical location where the cubes will be copied to so PPES can use the to answer requests from users.•Cube List – This is the a | delimited list of physical MDC files created by the model.

17

Flag Files & SQL QueryFlag Files & SQL Query

Conditional Cube Build VariablesConditional Cube Build Variables

The Cube Build Conditional Variables are located directly below the Location Variables sections we just discussed.

In this section we set flags to indicate whether the cube build is contingent on the presence of a Flag File or the meeting of a SQL Query.

This section falls below the highlighted line as shown on the screen.

In this section we set a Flag to “1” and the required file / path or SQL Connection and Query information.

These flags cause steps within the macro to be executed that perform the required tests.

18

Email NotificationEmail Notification

Email Notification VariablesEmail Notification Variables

This is the section where you can customize the list of recipients that you want to receive emails from this macro.

Email notifications are divided into three distinct levels:

•Success – Everything worked as expected and the cubes have been built andpublished into production

•Called-Off – The Flag File or SQL Query conditions were not found/met and the cube build was not attempted.

•Failure – Something unexpected has happened during the cube build and immediate assistance is required. Nothing has been updated in production

You can choose as I have to set up email distribution list and for some global variables or you can add delimited physical email addresses to each set of email notifications.

The File names are set automatically using the default values you provided in the Location Variables section earlier.

19

PPES VariablesPPES Variables

PowerPlay Enterprise ServerPowerPlay Enterprise Server

This section basically sets the names and port numbers of your PPES servers as well as the UserID and Password of a user with rights to disable and enable cubes. It also contains the PPES locations and names of each of up to eight cubes you will be moving into production using the script.

The important things that I have learned about this portion of the script is to be very careful in setting the strCubeName1….strCubeName8 values.

In this example I have two cubes that are created and moved into production by this macro.

The first cube is two folders deep in the PPES Hierarchy and has a space in the file name. I use the forward slash “/” to denote folders and the underscore or underline “_” to replace spaces in the PPES file name.

20

Additional ResourcesAdditional Resources

• CD Provided at Conference

• http://www.bmorrishome.org

• Cognos Support Website

• Other Forums (Yahoo Groups, etc)

• CD Provided at Conference

• http://www.bmorrishome.org

• Cognos Support Website

• Other Forums (Yahoo Groups, etc)

I will be providing at least 100 CD’s at the conference. They will have this presentation as well as a full working copy of this script.

I am still not sure about my website. I need to have the time to update it. If I include it, I will have both the presentation and the script available there as well

The Cognos Support Website goes without mentioning…..

The other forums I will mention are some of the user groups and forums that I know of on the net…..

21

Question & AnswerQuestion & Answer

??“It’s not Difficult, just Complex”

(Morris, 2002)

Remember to plan for success and that this is not Difficult, jus t Complex….

Thank you for attending this session and please feel free to contact me if you need any further assistance.