json essentials for the db2 z/os dba -...

JSON Essentials for the DB2 z/OS DBA

Thanikachalam “Billy” Sundarrajan

1

Agenda

• What is JSON?

• JSON Support in DB2 11 for z/OS

• How are JSON objects/documents stored?

• JSON SQL Interface

• Retrieving JSON documents

• Useful JSON UDFs

• Updating JSON documents

• Indexing JSON documents

• nosql traces

• Questions

2

What is JSON

• JSON – Abbreviation/Acronym for Java Script Object Notation

• Subset of Java Script

• Specified in IETF RFC 7159, ECMA 404 • https://tools.ietf.org/html/rfc7159 • http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf

• Contains 6 types of values • object, array, number, string, boolean, null

• Typically not deeply nested

• Four atomic concepts • String, number, boolean, null

• Used by most modern programming languages for exchanging data

• BSON – Abbreviation for Binary JSON

3

https://tools.ietf.org/html/rfc7159

http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf








What does a JSON object/document look like?

• Collection of name/value pairs

• Object – pair of curly brackets surrounding zero or more name/value pairs

• Name is a string enclosed in quotation marks

• Value can be - object, array, number, string, boolean, null

• A single colon separates the name from the value

{"Name" : "Billy", "Hobby" : "DB2", "likesDB2": true, "rating": 99, "JSONExpert ": null }

JSON Object Value

(string) Name

Value (Boolean)

Value (number)

Value (null)

4

What does a JSON object/document look like?

• Array – Ordered sequence of JSON Values

• An array structure is a pair of square bracket tokens surrounding zero or more values

• The values are separated by commas.

• The order of the values is significant

• "Certifications.n" refers to the n+1th element of the array

• .0 refers to the first element, .1 refers to the second element

{"Certifications" : ["DB2 V10", "DB2 V11"] }

Object Value Array Value

5

What does a JSON document look like?

• Collection of name/value pairs

• Example shows an array of objects

• Array name identified by the name preceding "["

{"Employee":

{"EID": 12345,

"StartDate" : "2001-01-01",

"empexams" : [ {"exam" : "DB2 101",

"Passdate" : "2002-01-01",

"Price": 525.50},

{"exam" : "DB2 201",

"Passdate" : "2004-01-01",

"Price" : 850.50}

]

}

}

6

JSON Support in DB2 11 for z/OS

• Requires APAR PI05250 and APAR PI10521

• Server side UDFs – SYSTOOLS.JSON2BSON, SYSTOOLS.BSON2JSON

• DB2 nosql CLP – (DB2 10.5 FP4 used in the examples)

• PI39003 – Additional support to use JSON functions on a BLOB that is not part of a base table

• Quick check for enablement: • Table - SYSTOOLS.SYSJSON_INDEX

• Functions – SYSTOOLS.BSON2JSON, SYSTOOLS.JSON2BSON • db2level • Look for db2nosql.bat

7

JSON & DB2

DRFA

NoSQL/JSON CLP

JSON Java API

JSON/NoSQL Wire Listener

DB2 Engine JSON_VAL, JSON UDFs

Python Node.js PHP

JDBC Driver

DRDA

BSON Wire Protocol

Java

JSON Wire Listener -Leverages community drivers NoSQL/JSON CLP -Administration commands -Adhoc queries DB2 Data Server -JSON documents stored as BSON in UTF-8 -UDFs to process JSON documents -Indexing on JSON documents using Function based indexes -JSON_VAL

*Credits - Jane Man, IBM for schematic 8

How does DB2 store JSON objects?

• JSON objects can be stored as VARCHARs, BLOBs or CLOBS

• JSON SQL functions require that the data be in the binary JSON format (in UTF-8)

• JSON objects stored as VARCHARs or CLOBs do not need the use of UDFs

9

Creating A Table To Store JSON

• Creating a table using nosql CLP (Command line processor)

• Creating a table using user defined DDL

• JSON terminology

10

JSON DB2

database, db Schema

Collection Table

find WHERE

use SET CURRENT SCHEMA

Object/Document Row/Record

Creating A Table To Store JSON – using nosql CLI

• nosql command line interface can be invoked in the following ways: • From the DB2 client command prompt

• From a windows command prompt using java

• From DB2 client command prompt

db2nosql -user userid -hostName hostDNS -port db2port -db DB2LOCN -password passwd

• From DB2 command window (Change to sqllib/json/bin directory)

java.exe -cp "..\..\tools\jline-

0.9.93.jar;..\lib\nosqljson.jar;..\lib\js.jar;..\lib\mongo-2.8.0.jar;%CLASSPATH%"

com.ibm.nosql.json.cmd.NoSqlCmdLine --url jdbc:db2://hostName:db2port/DB2LOCN --user

userid --password passwd

• Get Version details/configuration

java.exe -cp "..\..\tools\jline-

0.9.93.jar;..\lib\nosqljson.jar;..\lib\js.jar;..\lib\mongo-2.8.0.jar;%CLASSPATH%"

com.ibm.nosql.json.tools.NoSQLVersion -version -configuration

11

-DIS DDF DETAIL Provides

connection details

Creating A Table To Store JSON – using nosql

• From nosql interface

nosql>Type your JSON query and hit <ENTER>

nosql>db

Database: jdbc:db2:// hostDNS:db2port/DB2LOCN Schema: TEST

nosql>use BILLY

CDJSN1214I Switched to schema: "BILLY"

nosql>db.createCollection("JSONEMP")

CDJSN1006I Collection: "db.BILLY."JSONEMP"" created. Use "JSONEMP".

12

Default schema “TEST”

Creating table to store JSON

• What does the table definition created by nosql look like?

CREATE TABLE BILLY.JSONEMP

(ID CHAR(12) FOR BIT DATA NOT NULL,

"DATA" BLOB(16 M) WITH DEFAULT NULL

INLINE LENGTH 25000,

ROWCREATED

TIMESTAMP (12) WITHOUT TIME ZONE NOT NULL

WITH DEFAULT,

CONSTRAINT ID

PRIMARY KEY (ID))

PARTITION BY SIZE EVERY 4 G

AUDIT NONE DATA CAPTURE NONE CCSID UNICODE

NOT VOLATILE APPEND NO ;

-Unique identifier for each JSON doc -Data type can be

changed -Generated by API

if not supplied

Binary representation of

JSON object

13

CCSID Unicode

Creating table to store JSON – changing ID datatype

nosql>db.createCollection("JSONEMP2",{ _id:"$int"})

CREATE TABLE BILLY.JSONEMP2

(ID INTEGER NOT NULL,

"DATA"

BLOB(16 M) WITH DEFAULT NULL INLINE LENGTH 25000,

ROWCREATED TIMESTAMP (12)

WITHOUT TIME ZONE NOT NULL WITH DEFAULT,

CONSTRAINT ID PRIMARY KEY (ID))

PARTITION BY SIZE EVERY 4 G

AUDIT NONE DATA CAPTURE NONE

CCSID UNICODE NOT VOLATILE APPEND NO ;

Unique identifier for each JSON doc.

Binary representation of

JSON object

_id – Reserved name that identifies

the identifier

14

Creating table to store JSON

• An entry is created in SYSTOOLS.SYSJSON_INDEX • IDXNAME is set to null

• COLLECTION is set to the table name

• DATATYPE is set to the data type of the Unique identifier

• COLLECTION_SCHEMA set to the JSON database name

• If an index is created, an additional row is created in SYSTOOLS.SYSJSON_INDEX with IDXNAME set to the name of the index

• Implicit tables created by nosql do not have the flexibility as user created objects to store JSON documents

• JSON is stored in UTF-8. If using user-created tables, be aware of CCSID of other tables it may be joined to (RI etc)

15

Storing JSON data in DB2

• Using nosql JSON objects can be inserted as-is.

• Data is converted to BSON format transparent to the end-user

nosql>db.JSONEMP.insert({"Employee": {"@ID": 12345, "StartDate" : "2001-01-01", "empexams" : [ {"exam" : "DB2 101", "Passdate" : "2002-01-01", "Price": 525.50}, {"exam" : "DB2 201", "Passdate" : "2004-01-01", "Price" : 850.50} ] } }) CDJSN1000I Command successfully completed.

Schema name set by using

the “use” command

No references to ID or DATA column. They

are implicit

16

Storing JSON Objects in DB2

• Storing JSON using DML (in BSON format) requires the use of JSON2BSON function – Pre-Insert trigger?

INSERT INTO BILLY.JSONEMP2 (ID, DATA) VALUES ( 12345, SYSTOOLS.JSON2BSON('{"Employee": {"@ID": 12345, "StartDate" : "2001-01-01", "empexams" : [ {"exam" : "DB2 101", "Passdate" : "2002-01-01", "Price": 525.50}, {"exam" : "DB2 201", "Passdate" : "2004-01-01", "Price" : 850.50} ] } } ');

Unlike nosql interface, ID value

explicitly set in DDL command

JSON value is converted to

BSON representation

by using JSON2BSON UDF

17

JSON - SQL Interface

• JSON_VAL – Built In Function • Similar to XMLQUERY – Extract segments of a JSON document

• SYSTOOLS.JSON2BSON, SYSTOOLS.BSON2JSON • UDFs to convert JSON to BSON and vice versa

• SYSTOOLS.JSON_TABLE • UDF to convert JSON document to relational – similar to XMLTABLE

• SYSTOOLS.JSON_TYPE • Parses a JSON segment to return a data type

• SYSTOOLS.JSON_LEN – UDF returns array length

18

Retrieving JSON data from DB2

• JSON data is stored in BSON format – reading the data “as-is” will provide data that is binary format

• JSON data can be retrieved using • BSON2JSON UDF to convert BSON data to text format

• nosql – commands to retrieve the data

• JSON_VAL Built In function to parse the JSON document and return selected parts of the JSON document

• JSON data is stored in BLOB (partially in-line). Distribution can be checked by LENGTH(DATA)

19

Retrieving JSON objects from DB2 - nosql

nosql>db.JSONEMP.find() nosql>Row 1: nosql> {

"_id":{"$oid":"58cd631c259abcdab1613599"}, "Employee": { "@ID":12345, "StartDate":"2001-01-01", "empexams" :[ { "exam":"DB2 101", "Passdate":"2002-01-01", "Price":525.5 }, { "exam":"DB2 201", "Passdate":"2004-01-01", "Price":850.5 } ] } }

find() command issued on JSONEMP collection

Displays all rows starting with 1.

_id is the internal identifier

20

Retrieving JSON objects from DB2 – BSON2JSON

SET PATH SYSTOOLS;

SELECT ID,BSON2JSON(DATA) FROM BILLY.JSONEMP;

ID: 0x58cd631c259abcdab1613599 DATA: {"Employee": {"@ID":12345,"StartDate":"2001-01-01", "empexams" : [{"exam":"DB2 101", "Passdate":"2002-01-01", "Price":525.5}, {"exam":"DB2 201", "Passdate":"2004-01-01", "Price":850.5}]}}

BSON2JSON to convert binary to

text

Auto generated identifier

21

Setting PATH simplifies user

interface; (optionally

consider views)

Retrieving JSON objects from DB2 – JSONVAL

• JSONVAL is a built-in UDF

• Similar to XMLQUERY

• Takes three parameters – JSON Value, Search pattern, Result type

• Search pattern contains the path qualified JSON field name

• "Employee.EID"

• "Employee.empexams.1.exam"

{"Employee":

{"EID": 12345,

"StartDate" : "2001-01-01",

"empexams" :

[ {"exam" : "DB2 101",

"Passdate" : "2002-01-01",

"Price": 525.50},

{"exam" : "DB2 201",

"Passdate" : "2004-01-01",

"Price" : 850.50}

]

}

}

22


• Result type • specifies the characteristics of the

result

• Not case sensitive

• :na can be appended to the result type to indicate JSON value cannot be an array

• :na not specified – then the first value of the array is returned

Result type

SQL data type

n DECFLOAT(34)

i INTEGER

l BIGINT

f DOUBLE

d DATE

ts TIMESTAMP(6)

t TIME

s:n VARCHAR(n)

b:n VARCHAR(n) for BIT DATA

u INTEGER with VALUES 0 or 1

23


JSONVAL (DATA,'Employee.StartDate','s:2048' )

-returns a string ' 2001-01-01'

JSONVAL (DATA,

'Employee.empexams.0.Price',‘n' )

-returns a numeric 525.50

.0 – indicates the first value in the array

JSONVAL (DATA,

'Employee.empexams.1.Price',‘n' )

-returns a numeric 850.50

.1 – indicates the second value in the array

{"Employee":

{ "EID": 12345,

"StartDate" : "2001-01-01",

"empexams" :

[

{"exam" : "DB2 101",

"Passdate" : "2002-01-01",

"Price": 525.50},

{"exam" : "DB2 201",

"Passdate" : "2004-01-01",

"Price" : 850.50}

]

}

}

24


JSON_VAL(DATA,'Certifications.1','s:40')

-returns a string ‘DB2 V11’

.1 – indicates the second value in the array

JSON_VAL(DATA,'Certifications','s:40:na')

-returns SQLCODE -20556

-:na indicates the result cannot be an array

- THE OPERATION FAILED BECAUSE MULTIPLE RESULT VALUES CANNOT BE RETURNED FROM A SCALAR FUNCTION, SYSIBM.JSON_VAL. SQLCODE=-20556, SQLSTATE=22547, DRIVER=3.68.61

JSON_VAL(DATA,'Certifications','s:40')

-returns a string ‘DB2 V10’

-without :na, if the result is an array, the first value is returned

{"Certifications" :

["DB2 V10", "DB2 V11"]

}

25


JSON_VAL(DATA, 'Passdate', 'u')

-returns 0

0 – value exists and is a null

JSON_VAL(DATA, 'Certifications', 'u')

-returns 1

1 – value exists and is not null

JSON_VAL(DATA, 'Certifications.3', 'u')

-returns NULL

NULL – value does not exist

{"Certifications" :

["DB2 V10", "DB2 V11"] ,

"Passdate":null}

26

JSON UDFs

• JSON_TYPE

• JSON_TABLE

• JSON_LEN

• JSON_UPDATE

• JSON_GET_POS_ARR_INDEX

27

JSON_TYPE

• UDF in SYSTOOLS schema that provides information on the segment of the JSON object

• JSON_TYPE takes three arguments

• JSON Object

• Path qualified JSON element

• Length of the field to be considered (default 2048)

• Returns a integer – which corresponds to the BSON type

• Most common values listed

BSON type

Data type

1 64-bit floating point

2 UTF-8 string

3 Embedded document

4 Array

8 Boolean

9 UTC datetime

10 Null value

16 32-bit integer

17 Timestamp

18 64-bit integer

28

JSON_TYPE

• SYSTOOLS.JSON_TYPE

(DATA,'Certifications',2048),

Returns 4 – which indicates an array


(DATA,'Passdate',2048)

Returns 10 – which indicates a null value


(DATA,'Goodcertification',2048)

Return 8 – which indicates a boolean value

{ "Certifications" : ["DB2 V10", "DB2 V11"] , "Passdate":null, "Goodcertification":true }

29

JSON_TYPE


(DATA,‘Employee',2048),

Returns 3 – which indicates a embedded document


(DATA,‘Employee.empexams.0.Price',2048)

Returns 1 – which indicates a floating point value

{"Employee":

{"EID": 12345,

"StartDate" : "2001-01-01",

"empexams" :

[

{"exam" : "DB2 101",

"Passdate" : "2002-01-01",

"Price": 525.50},

{"exam" : "DB2 201",

"Passdate" : "2004-01-01",

"Price" : 850.50}

]

}

}

30

JSON_TABLE

• UDF in SYSTOOLS schema - parses a JSON document and returns a table with two columns

• BSON data type

• String value – contains segment(s) of the JSON object

• JSON_TABLE takes three arguments

• JSON Object


• Return type – Values that can be passed are the same as that of JSON_VAL

• Result set columns are

• TYPE – Integer

• VALUE – VARCHAR(2048)

BSON type

Data type

1 64-bit floating point

2 UTF-8 string

3 Embedded document

4 Array

8 Boolean

9 UTC datetime

10 Null value

16 32-bit integer

17 Timestamp

18 64-bit integer

31

JSON_TABLE

SELECT X.TYPE, X.VALUE

FROM BILLY.JSONEMP,

TABLE(SYSTOOLS.JSON_TABLE

(data,'Employee.empexams.exam','s:2048')) X

-returns

• An implicit inner join is assumed

{"Employee":

{ "EID": 12345,

"StartDate" : "2001-01-01",

"empexams" :

[

{"exam" : "DB2 101",

"Passdate" : "2002-01-01",

"Price": 525.50},

{"exam" : "DB2 201",

"Passdate" : "2004-01-01",

"Price" : 850.50}

]

}

}

TYPE VALUE

2 DB2 101

2 DB2 201

32

JSON_TABLE

SELECT X.TYPE, X.VALUE

FROM BILLY.JSONEMP,


(data,'Employee.empexams','s:2048')) X

-returns

• An implicit inner join is assumed

{"Employee":

{ "EID": 12345,

"StartDate" : "2001-01-01",

"empexams" :

[ {"exam" : "DB2 101",

"Passdate" : "2002-01-01",

"Price": 525.50},

{"exam" : "DB2 201",

"Passdate" : "2004-01-01",

"Price" : 850.50}

] }}

TYPE VALUE

3 {exam:"DB2 101",Passdate:"2002-01-01",Price:525.500000}

3 {exam:"DB2 201",Passdate:"2004-01-01",Price:850.500000}

33

JSON_TABLE

Case of OUTER JOINs

SELECT billy.jsonemp2.id,

X.TYPE, X.VALUE

FROM BILLY.JSONEMP2 LEFT JOIN


(data,'Goodcertification','s:2048')) X ON 1=1

-returns

• Outer joins are required when the JSON_TABLE function does not find a qualified expressions and a NULL value is returned

-Document 1 , ID = 12348, Table – BILLY.JSONEMP2 { "Certifications" : ["DB2 V10", "DB2 V11"] , "Passdate":null, "Goodcertification":true } -Document 2 , ID = 12345, Table – BILLY.JSONEMP2 { "Certifications" : ["DB2 V10", "DB2 V11"] , "Passdate":null }

ID TYPE VALUE

12345 NULL NULL

12348 8 true

34

JSON_LEN

• UDF in SYSTOOLS schema that parses a JSON array and returns the number of elements in the array

• JSON_LEN takes two • JSON Object


• Result • Number of elements in the array

JSON_LEN(data, 'Certifications')

-returns 2

JSON_LEN(data, 'Certifications')

-returns 3

-Document 1 , ID = 12348, Table – BILLY.JSONEMP2 { "Certifications" : ["DB2 V10", "DB2 V11"] , "Passdate":null, "Goodcertification":true } -Document 2 , ID = 12345, Table – BILLY.JSONEMP2 { "Certifications" : ["DB2 V10", "DB2 V11“, “DB2 V12”] , "Passdate":null }

35

Updating JSON documents • Documents can be updated using nosql interface

• Issues a call to JSON_UPDATE - Similar to XMLMODIFY

• Function marked as internal use for nosql interface

• db.collection.update

• db.collection.update(query,updateSpec, upsert, multi, writeconcern)

• nosql>db.JSONEMP2.find({_id:12346})

nosql>Row 1:

nosql> {"_id":12346,

"Certifications":["DB2 V10","DB2 V11"]}

{"Certifications" :

["DB2 V10", "DB2 V11"]

}

36

Updating JSON documents • db.JSONEMP2.update({"_id":12346} {"$set":{"Certifications.2":"DB2 V12"}})

• Triggers a call to UDF

• UPDATE BILLY."JSONEMP2" SET DATA=SYSTOOLS.JSON_UPDATE(DATA,'$set','Certifications.2',?,SYSTOOLS.JSON_GET_POS_ARR_INDEX(DATA,?)) WHERE BILLY."JSONEMP2".ID = (SELECT ID FROM BILLY."JSONEMP2" WHERE (ID=?) FETCH FIRST ROW ONLY)

• nosql>db.JSONEMP2.find({_id:12346})

nosql> {

"_id":12346,

"Certifications":[

"DB2 V10",

"DB2 V11",

"DB2 V12" ]}

{"Certifications" :

["DB2 V10", "DB2 V11", "DB2 V12"]

}

After Update

$set – indicates all other values to be retained

37

Indexing JSON documents • Indexes can be created using nosql CLP

• Indexes can be Unique

• nosql>use BILLY

CDJSN1214I Switched to schema: "BILLY"

nosql>db.JSONEMP.ensureIndex({"Employee.EID":[1, "$int"]},"MYEMPIDX",true)

• An entry is created in SYSTOOLS.SYSJSON_INDEX

• IDXNAME set to MYEMPIDXCREATE UNIQUE INDEX BILLY.MYEMPIDX

• true – Indicates it is a Unique index.

• Similar to an XML Index – JSON Index filters to match documents that match the index

1- Ascending -1 - Descending

38

Indexing JSON documents • Indexes can be created using DDL

• Function based index – on JSON_VAL function

• CREATE UNIQUE INDEX BILLY.MYEMPIDX

ON BILLY.JSONEMP

(JSON_VAL(DATA, 'Employee.EID', 'f:na')) ;

• Descending indexes are not supported in DB2 z/OS

39

Indexing JSON documents • Indexes can be created on composite columns

– Certain constructs not supported in DB2 z/OS

- such as Descending Indexes

• nosql>db.JSONEMP.ensureIndex({"Employee.@ID":[1, "$int"],

"Employee.startdate":[1,"$string",10]},"MYEMPIDX2")

DDL executed in z/OS

CREATE INDEX BILLY."MYEMPIDX2" ON BILLY."JSONEMP"

( JSON_VAL(DATA, 'Employee.@ID' , 'f:na' ),

JSON_VAL(DATA, 'Employee.startdate' , 's:1024:na' ))

40

Indexing JSON documents • Checking to see if the index is used

• SELECT * FROM BILLY.JSONEMP WHERE (JSON_VAL(DATA, 'Employee.@ID', 'f:na')) = 12345

41

-example -SELECT * - not recommended -Stats were not run

Tracing JSON calls • nosql tracing calls controlled by

nosql.properties file

• Text file – Manually created and stored in directory from where db2nosql is invoked

• Sample contents

nosql.traceLevel=ALL

nosql.traceFile=C:/NOSQLTRC.txt

• JCC trace can be enable by using traceLevel and traceDirectory

db2nosql.bat -user userid -password passwd -url jdbc:db2://db2url:port/db2location:traceDirectory=c:\\jsontrc;traceLevel=-1;

Contents of Trace File [nosql][2017-03-20-13:27:05.181] IBM DB2 NoSQL JSON API 1.1.0.0 build 1.4.173 Detected nosql jar: file:/C:/Program%20Files/IBM/SQLLIB/json/lib/nosqljson.jar Global nosql properties in effect Note: These may be overridden by connection-specific settings: nosql.traceFile=C:/NOSQLTRC.txt nosql.traceLevel=ALL nosql.useBuiltInJsonVal=true

42

Note the use of \\ - Using a \ character to escape

Questions?

43