Handling Large Datasets by Using Cross Tables“When Turning 11 million rows into 1 billion can be a good thing”

Requirements...• Query, filter rows and display them in the table view (straight table) from 550 columns and 11 million rows

• Customise the table views by selecting specific columns

• Search specific unique id(s) by pasting the id(s) within the application

What’s the challenge?

• Users wanted to query very wide data sets and there’re no aggregation expressions – we just need to expose the individual cell values in a table form (All Transactional Level).

• Having hundreds of conditional expressions on the straight table will degrade the performance and user experience, even though expressions are not aggregations.

• Each chart expression has three possible expressions – conditional expression, label and actual expression. This adds to the calculation overhead, plus managing potentially hundreds of expressions is painful or boring!

•QlikView shows better performance when calculating across rows rather than across columns

• This app needs to be intuitive even though it contains only table view(s)

With dummy row...Without dummy row...

Name Value Pair or CrossTable Approach

• Extract data from the database and store the QVD(s)

• Add dummy row to tackle the NULL issue (Using Autogenerate 1)

• Transpose the data by keeping the Unique ID & Key Fields (To join the dimension tables)

Unique ID

Type Owner Colour Home

34785 Cat NULL Black London

34786 Cat NULL White NULL

Unique ID

Field Name

Field Value

34785 Type Cat

34785 Colour Black

34785 Home London

34786 Type Cat

34786 Colour White

Unique ID

Field Name

Field Value

34785 Type Cat

34785 Colour Black

34785 Home London

34786 Type Cat

34786 Colour White

99999 Type -

99999 Owner -

99999 Colour -

99999 Home -

Unique ID

Type Owner Colour Home

34785 Cat NULL Black London

34786 Cat NULL White NULL

99999 - - - -

Dummy row

Name Value Pair or CrossTable Approach (Contd...)

• Once the data transformation is completed (i.e. after transposing the data) – we would see the following number of rows:

((Actual Rows x Transposed Columns) – (Total Number of Null Values))

(3 x 4) – (3) = 9 Rows

• Create the pivot table with two dimensions – [Unique ID] & [Field Name] and use the expression as [Field Value]

• Pivot/Drag the [Field Name] column and you will see the table

• While users think they are making column selections they are actually making row selections

Demo