empower datamart snapshot table viewswinshuttle-help.s3.amazonaws.com/enterworks/downloads/10... ·...

Enable Technical Note

Empower DataMart Snapshot Table Views

of 17 Revised 1/16/2020

Copyright © 2007-2020 EnterWorks Acquisition, Inc., a Winshuttle company. All rights reserved.


Description: This technical note discusses Empower DataMart Snapshot Table Views and how to access them directly using SQL database management applications.

EnterWorks Product Applicability: Enable, all versions (10.0, 9.0, 8.2)

Customer Scope: All EnterWorks Enable customers using the relevant products listed above.

Contents Empower Snapshot Table Views ..................................................................................................... 1

Main Snapshot Table .................................................................................................................. 2

Category Attribute Snapshot Table ............................................................................................ 5

Multi-language Snapshot Table .................................................................................................. 8

Rebuilding Snapshot Tables .......................................................................................................... 12

Limitations of Snapshot Table Views ............................................................................................ 13

Attributes that are Good Candidates for Inclusion in the Snapshot Tables ................................. 14

Reasons to Use SQL to Access the Snapshot Tables ..................................................................... 15

Drawbacks to Using SQL to Access the Snapshot Tables .............................................................. 15

Permitting Access to Snapshot Tables .......................................................................................... 16

Additional Resources .................................................................................................................... 17

Empower Snapshot Table Views

In order for Enable to accept record attribute values that may be invalid or exceed database limits, Enable stores them in XML. Accessing or manipulating these attribute values requires parsing the XML, which can be time-consuming. For performance reasons, values that need to be accessed quickly or frequently can be included in the Empower DataMart Snapshot Table Views, (commonly called “snapshot tables”). Enable copies the indicated XML values, parses



of 17 Revised 1/16/2020


them, and stores them in the snapshot tables, in appropriately data-typed columns, where they can be searched and accessed quickly and efficiently. Only valid attribute values will be copied to the snapshot tables; invalid values will be stored as NULL.

A repository can have the following types of Enable snapshot table views:

• Main snapshot table.

• Category attribute snapshot table.

• Multi-language snapshot table.

Typically, when the main snapshot table view is referenced, it is called simply “the snapshot table”. The category attribute and multi-language snapshot tables are ancillary tables. If a main snapshot table exists, the category attributes and multi-language snapshot tables can be created to extend the abilities of the main snapshot table.

Main Snapshot Table

Each repository can have only one main snapshot table. It has the same name as the repository. It consists of all the attributes in the repository that are designated as “relational”. To designate an attribute as relational, using the Enable Classic UI, edit the repository’s attribute properties and set the Relational checkbox for the desired attribute. To create an index for the snapshot table for an attribute, set its Create Index checkbox. (For more information on making an attribute relational or building an index for it, see the Enable 10 Administration Guide.)



of 17 Revised 1/16/2020


Open the CZ Vehicle Staging Repository's Properties for Editing

Designate Attributes as Relational and Create an Index for an Attribute



of 17 Revised 1/16/2020


Each of the snapshot table’s columns are labeled with the restricted name of their corresponding attribute. The restricted name of an attribute is a human-friendly name that makes the attribute easier for a user to identify. (For more information on restricted attribute names, see the Enable 10 Administration Guide.)

The first few columns in a snapshot table are system bookkeeping columns. They are:

• InternalRecordId: This is the ID of the attribute’s repository record.

• Created: When the repository record was created.

• Last_Updated: The last time the repository record was updated for any reason, including validation.

• Data_Last_Updated: The last date any attribute in the repository record was last

updated.

• Snapshot_Last_Updated: The last time the snapshot record was updated. This may not match the Data_Last_Updated value if a change is made to one of the repository record’s attributes that is NOT in the snapshot table or if the snapshot table has been regenerated.

The attribute columns follow the system columns and are arranged alphabetically.



of 17 Revised 1/16/2020


Category Attribute Snapshot Table

The category attribute snapshot table view is sometimes called the “vertical snapshot table”, because instead of the attributes making up the columns of the table, they make up the rows. They are arranged in this manner to allow the category attribute snapshot table view to

The 1WS_FlexValues Repository's Snapshot Table View's Attribute Columns



of 17 Revised 1/16/2020


overcome the database limitation of allowing only 1022 columns, thereby freeing Enable repositories to have an unlimited number of category attributes in the snapshot tables.

To trigger the creation of a repository’s category attribute snapshot table, using the Enable Classic UI, edit the repository’s properties and check the repository’s Create Category Attribute Snapshot box. The category attribute snapshot table will be created the next time the snapshot tables are rebuilt. (For instructions on editing a repository’s properties and rebuilding snapshot tables, see the Enable 10 Administration Guide).




of 17 Revised 1/16/2020


A category attribute snapshot table view has the same name as the main snapshot table, except that it has a suffix of “_KV”. KV stands for “key value pairs”. So if a repository is named:

Product_Staging

its category attribute snapshot table view would be named:

Product_Staging_KV

A repository’s category attribute snapshot table has a row for each record’s non-null category attributes, whether the category attribute has been marked as relational or not. The schema of each row is:

• InternalRecordId: This maps to the InternalRecordId of the record in the

repository. It can be used to join a category attribute snapshot table row to a row in the main snapshot table.

• KeyName: The name of the category attribute. This will exactly match the attribute name in the repository, which can be seen by viewing the repository’s profile or opening the repository in Enable’s Detail Editor.

Set the Repository's Create Category Attribute Snapshot Flag



of 17 Revised 1/16/2020


• KeyRestrictedName: The restricted name of the attribute, as defined in the repository’s attribute definition.

• KeyValue: The attribute’s value.

• KeySequence: If a display sequence is set for the attributes in the repository, an attribute’s sequence value will be reflected in the category attribute snapshot table’s attribute’s KeySequence field.

If a category attribute is designated as relational and the repository has a category attribute snapshot table, the category attribute will appear in both the main snapshot table and in the category attribute snapshot table.

Multi-language Snapshot Table

Enable supports multi-language attributes, which may have a value for each language Enable has been configured to support. If a repository has multi-language attributes, and those attributes are defined as relational, and the repository has been configured to create multi-language snapshot tables, a multi-language snapshot table will be created for each language other than the default language.

To trigger a repository to create multi-language snapshot tables, using the Enable Classic UI, edit the repository’s properties and check the repository’s Snapshot Language Support box. The multi-language snapshot tables will be created the next time the snapshot tables are rebuilt. (For instructions on editing a repository’s properties and rebuilding snapshot tables, see the Enable 10 Administration Guide).

The Item_Staging Repository’s Category Attribute Snapshot Table Columns



of 17 Revised 1/16/2020





of 17 Revised 1/16/2020


The names of the multi-language snapshot tables will be suffixed with their language code. So if a repository is named:

Product_Staging

and Enable has been configured with English as its default language and configured to support the French, German, and Spanish languages, the following multi-language snapshot tables will be created:

Product_Staging_fr

Product_Staging_de

Product_Staging_es

Set the Repository’s Snapshot Language Support Flag



of 17 Revised 1/16/2020


The attribute’s value for the default language will be stored in the main snapshot table, while any values for the attribute’s non-default languages will be stored in their respective language-specific snapshot tables.

Unlike the main snapshot table view, which will have entries for each attribute that is defined as relational, the multi-language snapshot tables will only have entries for attributes that have a non-null value for that language.

The first few columns in the multi-language snapshot table are system bookkeeping columns. They are:

• InternalRecordId: This is the ID of the repository’s record.

• Created: When the repository record was created.

• Last_Updated: The last time the repository record was updated for any reason, including validation.

• Data_Last_Updated: The last date any attribute in the repository record was last updated.

• Snapshot_Last_Updated: The last time the snapshot record was updated. This

may not match the Data_Last_Updated value if a change is made to one of the repository record’s attributes that is NOT in the snapshot table or if the snapshot table has been regenerated.

The attribute columns follow the system columns. They are:

• InternalRecordId: This maps to the InternalRecordId of the record in the repository. It can be used to join a multi-language snapshot table row to a row in the main snapshot table.

• KeyName: The name of the attribute. This will exactly match the attribute name in the repository, which can be seen by viewing the repository’s profile or opening the repository in Enable’s Detail Editor.

The Item_Staging Repository’s Multi-language Snapshot Table Views



of 17 Revised 1/16/2020


• KeyRestrictedName: The restricted name of the attribute, as defined in the repository’s attribute definition.

• KeyValue: The attribute’s value.

• KeySequence: If a display sequence is set for the attributes in the repository, an attribute’s sequence value will be reflected in the multi-language snapshot table’s attribute’s KeySequence field.

When Enable’s multi-language feature is being used, typically only a subset of attributes are multi-language. For instance, an item’s description and feature bullets might be multi-language, but the measurements are not.

To use SQL to access an attribute in a multi-language snapshot table, query the main snapshot table to get all the global attributes, including the default language attribute values, then query or join it with the desired multi-language snapshot table, using the internal record ID to get the attribute values for that same record.

For example, if English is the default language, to get an attribute’s French value, you would query the main snapshot table to get the global attributes, then query or join it with the French snapshot table, using the internal record ID to get the French attribute value.

Rebuilding Snapshot Tables

Changes to the data model can affect a repository’s snapshot tables, including causing them to be rebuilt. An event that triggers the main snapshot table to be rebuilt will also trigger the category attribute snapshot table and the multi-language support snapshot tables to be rebuilt, if they exist.

Examples of changes that may trigger the snapshot tables to be rebuilt include:

• Changing an attribute’s data type.

• Making an attribute repeatable.

• Adding or removing an attribute from the main snapshot table.

• Changing a code in a Code Set.

Rebuilding the snapshot tables is processed as a background job. Once the job has been started, its progress can be monitored in the Job Monitor.



of 17 Revised 1/16/2020


For a detailed list of data model changes that affect the snapshot tables, see the Enable 10 Administration Guide.

When a snapshot table is rebuilt, the view is deleted and a new one is generated. Any users who had access that was explicitly granted to the old view will no longer have access to it, nor will they have access to the new view until that explicit access has been re-granted. This can be avoided by granting general access to the views so if a view is re-created, it can be accessed without being explicitly granted.

Also, rebuilding a snapshot table can take a while. While a snapshot table is being rebuilt or repopulated, only the repository records that have already been added to the snapshot table are available for use. Records that have not yet been added/re-added will be unavailable.

For these and other reasons, data model changes should not be made during the business day. The best practice is to make changes after business hours, when the impact to users is minimal. After changes to the data model are complete and the affected snapshot tables are rebuilt, if you are explicitly granting access to the views, you will need to do so.

Limitations of Snapshot Table Views

Snapshot table views have the following limitations:

• If an attribute’s value is invalid, it will appear as NULL in a snapshot table. Snapshot table columns are typed for the datatype defined for the attribute in the repository’s profile. Therefore, a snapshot table column can only hold data appropriate to its type. If an attempt is made to store an attribute value in a snapshot table that does not match its defined datatype or if the value is otherwise invalid, the value NULL will be stored.

The Snapshot Rebuilding Job in the Enable 2020 UI’s Job Monitor



of 17 Revised 1/16/2020


• Snapshot tables are limited by the database’s constraints of 1022 columns and 8060 bytes per row.

• Snapshot tables are read-only.

• Snapshot tables are volatile. A number of data model changes can cause a snapshot table to be rebuilt, which can affect permissions granted to users to access the table.

• If you define your own views on top of a snapshot table view, if the snapshot table gets rebuilt, you will lose your views. You need to have a process in place to rebuild your views.

• The snapshot table indexes you define through the Enable UI are on individual attributes. If you need composite indexes, you have to define them separately from Enable, though there are tools in Services Framework for helping maintain composite indexes.

• If the snapshot table gets rebuilt, your composite indexes will not be rebuilt automatically.

• Multi-language category attributes must be included in the main snapshot table in order for the multi-language snapshot table values to be kept. They will not be built from the category attribute snapshot table; you have to explicitly add them to the main snapshot table.

• If there are many attributes in the snapshot table, it can take longer for Enable to update the repository. When an attribute in a snapshot table is updated, it is updated in two places: the repository record and the snapshot table. If a lot of attributes in the snapshot table need to be updated at once, it may affect Enable’s performance.

Attributes that are Good Candidates for Inclusion in the Snapshot Tables

Consider adding the following types of attributes to the snapshot tables:

• Primary keys: Primary keys are accessed often.

• Join attributes in linked repositories.

• Attributes in Preference Views that are used frequently: In a Repository View, if you have a Preference View that doesn't have many columns, that you will use frequently,

Invalid Snapshot Table Values



of 17 Revised 1/16/2020


and that you want to display as quickly as possible, consider putting all its attributes in the snapshot table.

• Attributes that you want to be able to sort: When you view a repository in a Repository View using the Grid View, only attributes that appear in the snapshot table can be sorted. Attribute columns can be filtered whether or not the attributes are in the snapshot table.

• Attributes you want to range search: To be able to search for a range of values on attributes of datatypes integer or date, you must add them to the snapshot table.

• Attributes you plan on accessing through SQL: Attribute values can be read from the snapshot table using SQL.

• Attributes to be used in a bulk callout: If you want to access an attribute in a bulk callout, it must be in the snapshot table.

• Attributes included in a Scheduled Export of type View: If an attribute is included in a Scheduled Export of type View, unless the attribute is in the snapshot table, it must be extracted and parsed from the XML.

• Attributes accessed from workflow: If a workflow accesses an attribute value, unless the attribute is in the snapshot table, it must be extracted and parsed from the XML.

Reasons to Use SQL to Access the Snapshot Tables

There are several reasons you might want to use SQL to access the Empower Snapshot Table Views directly, instead of using Enable’s UI, file generation capabilities, or the REST API:

• Performance: Accessing a database through its management tool is typically faster than getting the data through Enable. It also doesn’t impact Enable’s processing rate for other jobs. However, this may not be true, depending on the type of queries being made, the number of rows being processed, and if indexes are being used efficiently.

• Simplicity: Accessing the snapshot table directly may be easier than having Enable generate files that you later need to process to get to the data.

• Ease of use: SQL queries allow you to pull data from multiple tables at once.

• Familiar Tools: You may already have a 3rd-party application that you use to query your databases.

Drawbacks to Using SQL to Access the Snapshot Tables

Using SQL to access the snapshot tables directly can have its drawbacks as well:



of 17 Revised 1/16/2020


• Data duplication: Snapshot tables hold duplicate copies of the attributes stored in XML. When an attribute value changes, Enable updates it in both the repository and in the repository’s snapshot table(s). This increases the amount of time needed to change the value.

• Using SQL to update a snapshot table does not update Enable’s database: Changes made to a snapshot table are not written back to Enable’s database and they will be overwritten when the snapshot table is next updated. You should always use an Enable UI, Enable’s REST API, or import data to make changes to the database.

• You must be careful when granting access to the snapshot tables: Depending on the access granted to users, SQL can also be used to read and write directly to Enable’s system tables and database. This is extremely dangerous. It puts your data and system’s stability and validity at risk. Do not allow users access to system tables and the Enable database.

• No audit trail: If you pull data from Enable’s snapshot tables directly, you have to keep track of when you got the data and what data you retrieved. Unlike when you have Enable generate export files, there are no artifacts or audit trails, other than those you create and manage yourself.

• You manage your data’s freshness: Enable allows you to generate files according to a schedule. You can also construct a “delta” export, so each export file only includes the records that were added or changed since the last file was generated. If you use SQL to pull values from the snapshot tables directly, you have to keep track of when you last retrieved data and if desired, determine how to retrieve only records that have been added or changed.

• Decreased performance: If you are properly indexing the snapshot tables and performing simple SELECT queries, your queries should have little impact on Enable's performance. But if your users are making complex queries on large datasets or if their queries result in table scans that require the database server to access each record, the database server’s CPU can become overloaded, which in turn slows Enable’s response time. If you find yourself in this situation, consider mirroring the snapshot tables in your database to another server that can be queried without affecting Enable’s performance. Mirroring your database, however, means you need to manage keeping the duplicate database current.

Permitting Access to Snapshot Tables

Do not give your users the same credentials Enable uses to connect to the database. The Enable user account has full access to the database. Allowing your users full access to the database not only poses a security risk, it also risks system and data stability, as it will allow



of 17 Revised 1/16/2020


your users to read and write to system and product tables. Do not give users full access to the Enable database.

Also, follow your enterprise policy and procedures for granting access to users to ensure sensitive data is not exposed, including passwords. You may want to limit a user group’s access to only the objects they need to access.

When granting access to users, grant access to roles (user groups) rather than individual users. This follows Enable’s user management best practices as it makes controlling user access easier and more efficient.

While granting access at the object level, (meaning a user can query only particular views), gives you the tightest control, it also requires the greatest amount of maintenance. Each time a snapshot table is rebuilt, the user must be re-granted access. You may want to consider writing a script to grant access and either allowing the user to run it, to have a process run it periodically, or to run it after any data model changes.

Another way to ensure the user has access to the latest view is to allow them the ability to access to all views in the database. This has the benefit of avoiding the need to re-grant view access each time a snapshot table is rebuilt.

Additional Resources

For detailed information regarding managing and accessing snapshot tables, see Enable 10 Basics, the Enable 10 Administration Guide, and your Enable solution specifications.

empower datamart snapshot table viewswinshuttle-help.s3.amazonaws.com/enterworks/downloads/10... ·...

Documents