carlton doe and his session – aka the warm-up band for mark scranton
DESCRIPTION
Carlton Doe and his session – aka the warm-up band for Mark Scranton. Moving from IBM IDS 7.x to IBM IDS 9.x. Carlton Doe Technical Sales Manager, Dallas, TX IBM. ?. ISBN 0-13-605296-7. ISBN 0-13-080533-5. Who I Am. - PowerPoint PPT PresentationTRANSCRIPT
Carlton Doe and his session –
akathe warm-up band for
Mark Scranton
Moving from IBM IDS 7.x to IBM IDS 9.x
Carlton Doe
Technical Sales Manager,
Dallas, TX
IBM
3
ISBN 0-13-605296-7 ISBN 0-13-080533-5
?
Who I Am 12 + years of experience as DBA and engine admin mainly in retail
environments Co-founded, presided over, and former member of the Board of
Directors of the International Informix Users Group (IIUG) Written two Informix Press books:
4
What I’m Going to CoverThe Basics
IDS Futures What is IBM IDS and Foundation? Things to Know Before Moving
Reserved Words ONCONFIG Parameters How Do I Move?
Dynamic Logical Log Files Configurable Lock Modes SQL Statement Cache Raw / Standard Tables Changes to “Explain Mode” Fuzzy Checkpoints IBM Informix MaxConnect
5
Smart Large Objects and Dbspaces Simple/Complex Datatypes and Casting User Defined Routines and Functions Collections DataBlades and the Blade Manager HDR / Enterprise Replication – A Snapshot Java in the Engine
What I’m Going to CoverThe Good Stuff
6
IDS FuturesIDS Futures
7
Projected IDS Roadmap
IBM will deliver new releases of IDS about every 18 months
9.6
2003 2004 2005 2006
9.x
In PlanningAccepting Feature Requests
Projected 4Q 2004 Delivery
In Development Features Frozen
Planned 2Q 2003 Delivery
9.4 9.5
In Concept
Projected 2Q 2006 Delivery
8
Development is focused on providing a release of IDS 9 of interest to all IDS customers.
OLTP Features are the #1 Priority
Scalability Security SQL Compatibility Performance Ease of Upgrade And More!
IDS 9.40 Features
9
What can you expect down the road? Further Integration with IBM & DM Products Focus on Security Focus on SQL Compatibility Focus on Performance Leverage the SMART Initiative Support for the Latest Hardware
Platforms Operating Systems Structured Storage – NAS & SAN
IDS 9.50 & 9.60
10
Platform Directions Industry Consolidation, majority of IDS sales are on 5
platforms: Tier 1 Operating Systems: AIX, Solaris, HPUX, Windows, Linux Tier 1 Platforms: Sparc, Power, PA-RISC, Intel, IA64 Planned Continuing support (Planned for IDS 9.40) for Tru 64, Irix, Caldera
OpenUnix + Linux Kernel Personality (LKP)
Future is 64 bit. Many vendors are slowly moving towards IA64
Affected operating systems include: Windows, Linux, HPUX Whenever possible move to 64 bit servers, rather than 32 bit Most applications will run unchanged against a 64 bit server
Some will not be 64 bit clean -> thorough testing prior to deployment is recommended
Full exploitation of 64 bit server may require recompiling and/or modifying application
11
Downlevel Operating System VersionsYou need to move to newer OS levels and/or 64 bit servers
(recommended if available) Solaris 2.5.1 Solaris 2.7 (64 bit) AIX 4.3.2 and below HPUX 10.x Tru 64 4.x Dynix 4.4.x and below Windows NT Linux 2.2 Kernel HPUX 11 32 bit Caldera SCO OpenServer & UnixWare
12
What’s the DifferenceBetween IDS and
Foundation?
What’s the DifferenceBetween IDS and
Foundation?
13
IBM Informix Dynamic Server7.x
IDS - UDO9.1x
What Is IBM IDS and Foundation?
Parallelism built-in Parallel Data Query SQL92 Entry Level Enterprise Replication Many, many more...
SQL 3 Support DataBlade Support DataBlade Developer Kit User Defined Routines User Defined Datatypes User Defined Indexing R-Tree Indexing Extended B-Tree Support Row Types Collections
(sets, multiset, lists) Inheritance Polymorphism Partitioning with new data types
14
IBM Informix Dynamic Server7.x
IDS - UDO9.1x
+ =
IBM IDS 9.2 +IBM IDS 9.2 +
What Is IDS and Foundation?
15
Then What Is Foundation?
WebDataBlade
ExcaliburText
DataBlade
ObjectTranslator
OfficeConnect
J/Foundation
IBM Informix Dynamic ServerIBM Informix Dynamic Server
16
The New Face of the Foundation Family:
IBM IDS with J/Foundation New product -- IDS with the Hot-Spot JVM
Internet Foundation – has been discontinued
Financial Foundation for Capital Markets TimeSeries, NAG DataBlade Modules, TimeSeries Real Time Loader,
Office Connect, Object Translator
Law Enforcement Foundation Visionics, fingerprint, and other biometric-oriented DataBlade
Modules
IBM Informix Spatial DataBlade Module isIBM Informix Spatial DataBlade Module isfreely available to IDS 9.3 customers!!freely available to IDS 9.3 customers!!
17
Which Version Is Loaded??
(IBM IDS)Odra: onstat -
Informix Dynamic Server Version 9.30.UC1G1
(Foundation)Ebro: onstat -
Informix Dynamic Server Version 9.30.UC1G1
(MSGPATH)Wed Oct 4 07:49:52 200007:49:52 Booting Language <builtin> from module <>07:49:52 Loading Module <BUILTINNULL>07:49:57 Informix Dynamic Server Version 9.30.UC1G1 Software Serial Number
AAB#J50070507:50:12 Informix Dynamic Server Initialized -- Shared Memory Initialized07:50:15 Physical Recovery Started07:50:25 Physical Recovery Completed: 0 pages restored
18
Which Version Is Loaded??
Ebro: cd /usr/informix/9_3/extend
Ebro: ls -l
drwxr-xr-x 4 informix informix 1024 Jul 19 08:08 ETX.1.30.UC5
drwxr-xr-x 4 informix informix 1024 Jul 19 08:07 TXT.1.10.UC5
drwxr-xr-x 4 informix informix 1024 Jul 19 09:02 VTS.1.20.UC1
drwxr-xr-x 2 informix informix 1024 Jul 19 07:42 ifxbuiltins.1.1
drwxr-xr-x 2 informix informix 1024 Jul 19 07:42 ifxmngr
drwxr-xr-x 2 informix informix 1024 Jul 19 07:42 ifxrltree.2.00
drwxr-xr-x 4 informix informix 1024 Jul 19 07:49 krakatoa
drwxr-xr-x 4 informix informix 1024 Jul 19 07:42 LLD.1.20.UC2
drwxr-xr-x 4 informix informix 1024 Aug 22 13:36 WEB.4.10.UC1
19
Things to knowbefore you moveThings to know
before you move
20
Things to Know
You can not directly move from < 7.x versions IBM STRONGLY suggests that you first move to the latest
available 7.x version then move to 9.x While you may want to export/import your data, in-place
upgrades have been quite successful -- just remember to wear your belt and suspenders
HP-UX 11.0 - RUN_AS_ROOT.server script is wrongLinks created for one library goes into /usr/lib instead of
/usr/lib/pa20_64; if left alone, engine will not come up (“library not found” error). A “defect” has been entered
** Check 9.3 version, may be corrected.**
21
With IBM IDS, CPU and user-defined VPs are run as user “informix” while other VPs are still run as root. This prevents UDRs from having root access
If you have auto-start scripts, you should modify them so they “su” to “informix” before starting the engine
Install engine after client products for “finderr” to work. The order is:
tools network engine
Things to Know
22
Precision change in float / smallfloat to decimal conversion smallfloat 8 -> 9float 16 -> 17
Default behavior of “create index” is now to create detached indexes. 7.x attached indexes are supported by the upgrade process
Use the “DEFAULT_ATTACH” variable to temporarily mimic 7.x behavior, it will be discontinued however in a future release
9.4 Feature: LVARCHAR increases 2K to 31KOptional parameter added to lvarchar data-type declarations:
LVARCHAR (n) where 0 < n < 32k-2. Current syntax still supported, but old 2k limit holds for that syntax.
Things to Know
23
If using IBM Informix Enterprise Gateway with DRDA version 7.31.UC1 with IBM IDS 7.x, you must get the DRDA patch to avoid defect #132619
HADR has changed. DRAUTO is no longer supported so after a HADR failure condition is declared, the secondary server goes to read-only mode
You must manually convert the secondary server to “standard” mode “onmode -d standard”
No changes to DRINTERVAL, DRTIMEOUT, DRLOSTFOUND
Things to Know
24
9.2 / 9.3 ER and HPL – does not support named and other complex datatypes, only distinct and opaque types
9.4 feature: new functionality! see ER/HDR portion of the presentation.
Test dbexport / dbimport to see if more complex named types work
Shut down HADR / ER before upgrading
Things to Know
25
System catalog changes - columns added, moved, data types changed. New tables added and some dropped. Sysindexes and systables now a view rather than tables
Long identifiers!!The 8/18 rule is dead (the crowd cheers!!)user ids - 32 characters identifiers - 128 characters
table name, server name, database name, column name, index name, synonym name, constraint name, procedure name, dbspace name, blobspace name, optical cluster name, trigger name, type name, routine language name, access method name, operator class name, trace message class name, trace message name, procedure variable names
Things to Know
26
9.4 Feature: File size limits for utilities have been increased from 2GB to 17 billion GB
Utilities Affected:Oncheck, Onload, Onlog, Onmode, Onmonitor, Onparams, Onspaces,
Onstat, Onunload, DB-Access, Dbexport, Dbimport, Dbload, Dbschema, Onpload, Ontape
Also updating Streams API for > 2GB support
9.4 Feature: Chunk size limit changes from 2 GB to 4TB 9.4 Feature: Maximum instance chunks changes to
32,767
Things to Know
Individual instance size now 128 petabytes
27
128 petabytes!Just how BIG is 128 petabytes? At a load rate of 10 gigabytes an hour it would take well
over a 1,000 years to fill a 128 petabyte database.
At the current price for high-end structured storage it would cost $3-4 billion to buy 128 petabytes of storage.
The US Library of Congress contains an estimated 10 terabytes of printed material. 128 petabytes is equal to 12,800 Libraries of Congress.
28
9.4 Feature: # of columns in a Functional Index Old Limit 16 columns New Limit 341 Columns (Java, SPL) New Limit 102 (C)
9.4 Feature: DBServerAliases Old Limit 10 New Limit 32
9.4 Feature: Size of a Shared Memory Dump Old Limit 2 GB limit New Limit 17 Billion GB
Things to Know
29
9.4 Feature: Removed IDS libraries from /usr/lib
Each user connection uses about 5% more shared memory. Watch your shared memory allocations - both instance and operating system This can be offset by using IBM Informix MaxConnect
Make sure you drop and rebuild your distributions after upgrading!
Things to Know
30
Things to Know: New Reserved Words
CACHE COSTFUNC ITEM SELCONST INNER JOIN LEFT
LOCKS RETAIN RAW STANDARD AVOID_EXECUTE USE_SUBQF AVOID_SUBQF
31
Things to Know: One Example of New SQL Functionality9.4 Feature: ORDER BY values not in select list
Most databases implement the ability to order a list based on values not returned by the select. Solution: Implement support for this SQL syntax:
SELECT d.dept_num FROM dept d, employees e WHERE d.dept_num = e.dept_num GROUP BY d.dept_num ORDER BY avg(e.salary);
32
Things to Know:New ONCONFIG Parameters
ALLOW_NEWLINE BLOCKTIMEOUT DD_HASHMAX DD_HASHSIZE DS_HASHSIZE DS_POOLSIZE PC_HASHSIZE PC_POOLSIZE SBSPACENAME SBSPACETEMP SYSSBSPACENAME STMT_CACHE STMT_CACHE_SIZE
STMT_CACHE_HITS STMT_CACHE_NOLIMIT STMT_CACHE_NUMPOOL VPCLASS JDKVERSION JVPHOME JVPLOGFILE JVPPROPFILE JVPJAVAVM JVPJAVAHOME JVPJAVALIB JVPCLASSPATH JVMTHREAD
33
Things to Know:New ONCONFIG ParametersVPCLASS
Enables you to designate and create uniquely named, user-defined, or system classes of VPs. User-defined VPs (UDVPs) have the same functional power as CPU VPs
User-defined VPs should be created to execute user- defined routines and/or DataBlades
Some DataBlades (Verity, Excalibur Text) require their own VP as does the JVM
User-defined VPs can be added/dropped on the fly with the “onmode -p class_name” command
34
Things to Know:New ONCONFIG Parameters
VPCLASSSyntax tree:
VPCLASS name,num=X,[max=X,aff=(X-Y),noyield,noage]
VP name is not case sensitive, options are order independent, no whitespace, 128 char max/entry
Must declare multiple JVPs to execute Java UDRs in parallel (Unix)
If using “noyield”, only declare “1” vp since the UDR will execute serially, causing others to queuefor their turn
35
Things to Know:New ONCONFIG ParametersUse the VPCLASS parameter to replace the CPU and AIO ONCONFIG parameters
VPCLASS cpu,num=3,max=10,noage
Comment out AFF_SPROC, AFF_NPROCS, NOAGE, NUMCPUVPS, SINGLECPUVP parameters
VPCLASS aio,num=5,max=10
Comment out NUMAIOVPS parameter
36
ON-Archive. Use either the ontape or ON-Bar backup utility instead.
DB/Cockpit. Use ISA instead. Informix-DBA. Use Server Studio Java Edition by
AGS instead. ISM graphical user interface. The command line
provides the same functionality.
Things to Know: Discontinued Support
37
Things to Know:How do I Move?
1. Stop database processing and force a change to a new logical log
onmode -sy; onmode -l; onmode -c; onmode -ky;
2. Back up the instance(s) and critical configuration files in $INFORMIXDIR
etc/ /aaodir/adtcfg$ONCONFIG (s)** /dbssodir/adtmasks ONCONFIG.stdsm_versions sqlhosts**tctermcaptermcap
38
3. Restart the instance(s) and put them into single-user mode to ensure all open transactions are properly rolled back
oninit -sv
4. Verify data and index integrity with the oncheck utility (-cr, -ce -cc, -c [ I / D ] db_name)
5. Shut the instance(s) down again and create level 0 backup(s) and/or dbexports – make sure logical logs are backed up and clear!!
6. Install the new IBM IDS / Foundation software and change relevant scripts, global parameters, config files
Things to Know:How do I Move?
39
/usr/informix
7_3
9_3
scripts
disks (also tapes)
isa
logs
config_files
max_conn
koetari -- symbolic links
odra -- symbolic links
(current logs) /old_logs
Environment Variables:
INFORMIXDIRONCONFIGINFORMIXSQLHOSTS
My Recommended Directory Structure
40
7. Change ALARMPROGRAM to [ null | no_log.sh ] if using On-Bar or LTAPEDEV to /dev/null
8. Install any DataBlades
9. Restart each instance into quiescent mode and monitor the MSG_PATH file. Sysmaster and reserved pages conversion is automatic (see $INFORMIXDIR/etc/dummyupds7x.sql). When completed, a notice is written to MSG_PATH. If upgrading from 9.2 to 9.3, watch for sysutils and sysmaster db build successful messages in MSG_PATH before doing anything else!
10. Verify data integrity with oncheck commands
Things to Know:How do I Move?
41
11. Execute an “update statistics low drop distributions” command on each database in the instance
12. Execute an “update statistics high” command on the sysmaster database
13. Execute an “update statistics medium distributions only” command on each database in the instance.
14. Change LTAPEDEV / ALARMPROGRAM back to its original value
Things to Know:How do I Move?
42
15. Create a level 0 backup.
16. Perform system tests to check performance. Execute queries that will stress indexes to see if the optimizer properly uses them as well as their performance.
17. Let the users back in
If there are any problems -- RESTORE from tape
Things to Know:How do I Move?
43
Dynamic Logical LogsDynamic Logical Logs
44
Dynamic Logical Logs:Introduction
New feature in IBM IDS 9.3 – automatically create, insert, and activate logical logs when needed!!!
Does not eliminate long transactions, just the server hangs that can occur especially on startup.
Required: Ability to add log on the fly Insert log immediately after current logical log Bring the log into active mode without a backup of rootdbs
/ critical Dbspace(s)
45
Dynamic Logical Logs:Introduction
In the pre-9.3 engine: Could only be added when in quiescent mode Always took the first available slot in the list Required a backup of the rootdbs (level 0) to become active
46
Dynamic Logical Logs:Introduction
In IBM IDS 9.3: LOGSMAX disappears from $ONCONFIG Logs can be added while instance is processing
transactions if so configured Logs can be added after current log to avoid running
into log with "begin work" statement Newly added logs are immediately available Need to use DYNAMIC_LOGS $ONCONFIG
parameter
47
Dynamic Logical Logs:How do you add a log?
onparams –a [ -d dbspace ] [ -s size ] [ -i ]
48
Dynamic Logical Logs:How do you add a log?
49
Dynamic Logical Logs:Server actions
The server itself will attempt to add a log file if two conditions are true:
The next active log file contains an open transaction DYNAMIC_LOGS is set to 2 (the default)
The server checks for those conditions at two important points in the code:
Immediately after a log switch Beginning of the last phase of logical recovery (Transaction Cleanup)
50
Dynamic Logical Logs:Server Actions
What Dbspaces are used for automatic log allocation?
1 The dbspace that contains the newest log files. (If this dbspace is full, the engine searches other dbspaces.)
2 Mirrored dbspace that contains log files (but excluding the root dbspace)
3 All dbspaces that already contain log files (excluding the root dbspace)
4 The dbspace that contains the physical log
5 The root dbspace
6 Any mirrored dbspace
7 Any dbspace
51
Dynamic Logical Logs:DYNAMIC_LOGS parameter
Values for the DYNAMIC_LOGS parameter:
2 (default) the server is authorized to add log files automatically to avoid long transaction hangs
1 if a log file is needed, the server waits for the admin to add one manually
0 feature is turned off, mimics < 9.3 behavior
52
Dynamic Logical Logs:DYNAMIC_LOGS parameterWhy set DYNAMIC_LOGS=1?
You want to make use of the feature in order to avoid long transaction hangs, but you don’t want to give up any control over log file location or size.
Remember that once a log file is added to a non-critical Dbspace, that space becomes critical.
53
Dynamic Logical Logs:DYNAMIC_LOGS parameter
Why set DYNAMIC_LOGS=0?
Hmm, tough question. We don’t know.
Adding a log file to a Dbspace forces your next archive of that Dbspace (and the ROOT Dbspace) to be a level 0. If you’d rather risk a server hang than risk having to occasionally tweak your archive schedule, turn the feature off.
If the server does hang due to a long transaction rollback, simply set DYNAMIC_LOGS to 1 or 2 and bounce the server. Now you may have to improvise with your archive schedule, but it’s better than prolonging a down system.
54
Dynamic Logical Logs:DYNAMIC_LOGS parameterDropping logical logs in 9.3 is similar to <9.3
Never-used log files (A) drop immediately Used log files are marked “Deleted”. Once all
spaces have been archived they will automatically drop
If system has never been archived, the new drop restriction isn’t necessary, and is not applied
NEW -- Can be done while instance is on-line!!!
55
Configurable Lock ModeConfigurable Lock Mode
56
Configurable Lock Mode:Introduction
An extension of the "lock mode [ row | page ]" syntax in "create table" command
Allows users / admins to set the lock mode on newly created tables without having to explicitly set it in the table creation statement.
Uses either an environment variable or an $ONCONFIG parameter setting
57
Configurable Lock Mode:How's it set?
IFX_DEF_TABLE_LOCKMODE environment variable
setenv IFX_DEF_TABLE_LOCKMODE row
This variable can also be set in $HOME/.informix or $INFORMIXDIR/etc/informix.rc files
DEF_TABLE_LOCKMODE $ONCONFIG parameter
DEF_TABLE_LOCKMODE row
This sets LOCK MODE to ROW for all tables for all sessions in the server
58
Configurable Lock Mode:Examples
No additional syntax:
{TABLE "carlton".test_table row size = 155 number of columns = 3 index size = 0 }
create table "carlton".test_table ( col1 integer, col2 char(50), col3 varchar(100) ) extent size 32 next size 32 lock mode page;
TBLspace Report for my_nt_test:carlton.test_table Physical Address 200033 Creation date 09/09/2001 15:07:35 TBLspace Flags 901 Page Locking TBLspace contains VARCHARS TBLspace use 4 bit bit-maps Maximum row size 155
59
Configurable Lock Mode:Examples
Explicit "lock mode page" syntax with environmental variable set to "row":
{TABLE "carlton".test_table_3 row size = 155 number of columns = 3 index size = 0 }
create table "carlton".test_table_3 ( col1 integer, col2 char(50), col3 varchar(100) ) extent size 32 next size 32 lock mode page;
TBLspace Report for my_nt_test:carlton.test_table_3 Physical Address 200245 Creation date 09/09/2001 15:18:09 TBLspace Flags 901 Page Locking TBLspace contains VARCHARS TBLspace use 4 bit bit-maps Maximum row size 155
60
Configurable Lock Mode:Examples
With environment variable set to "row" but no additional syntax in "create table" command:{TABLE "carlton".test_table_2 row size = 155 number of columns = 3 index size = 0 }
create table "carlton".test_table_2 ( col1 integer, col2 char(50), col3 varchar(100) ) extent size 32 next size 32 lock mode page, row;
TBLspace Report for my_nt_test:carlton.test_table_2 Physical Address 200034 Creation date 09/09/2001 15:18:09 TBLspace Flags 903 Page Locking Row Locking TBLspace contains VARCHARS TBLspace use 4 bit bit-maps Maximum row size 155
Bug in the
code !!
61
SQL Statement CacheSQL Statement Cache
62
SQL Statement Cache: IntroductionAs implied, prepared and optimized SQL statements are stored in a section of the virtual portion of shared memory. Users executing identical DML SQL statements can pull the pre-parsed query plan and even query data structures from the cache
More complex queries (multiple columns, many filters in the WHERE clause) benefit most from caching
Turned off by default, the STMT_CACHE $ONCONFIG parameter and the “SET STATEMENT CACHE” SQL command control how the caching process works.
Can be overridden with the “onmode -e” command
63
SQL Statement Cache: How It Works
1. Incoming SQL statements are hashed to a value which is compared to other hash values in the SSC section of the virtual portion of shared memory
Caveats: statements only optimized at the PREPARE phase host variable names/placeholders are not included
in the hash algorithm white spaces and case of the statement is significant session-specific parameters (OPTCOMPIND, OPT_GOAL) are
significant and will affect hash value
64
SQL Statement Cache: How It Works
2. If hash value matches an existing value, the cached copy is used. The existing copy is NOT re-optimized
3. If a match does not exist, the statement is evaluated for inclusion in the SSC
Qualifications for inclusion in the SSC: contains DML statements using built-in data types and operators no UDRs/UDFs local database connections only no explicit temporary tables
65
SQL Statement Cache:How It WorksQualifications for inclusion in the SSC (cont):
“select *” statements can be fully expanded to include all columns in the listed tables
is not a statement generated by a stored procedure no embedded subselects in the SELECT statement
66
SQL Statement Cache:Enabling and Sizing
onmode -e {ENABLE|ON|OFF|FLUSH} ENABLEENABLE - Allows statement caching to take place. This does not turn
caching on, but allows it to be turned on
ONON - Turns on statement caching. Caching must first be enabled through the STMT_CACHE parameter or “onmode” command
OFFOFF - Turns off the SSC and immediately disables sharing. Cached statements that are in use remain in the SSC until their use count reaches zero
FLUSHFLUSH - Active statements are marked and subsequently flushed when they are released. When the SSC is turned off, sharing is immediately disabled
67
SQL Statement Cache:Enabling and Sizing
STMT_CACHE $ONCONFIG parameter:
00 - SQL Statement Cache disabled (default)11 - SSC enabled though sessions must explicitly call
for its use22 - SSC enabled for all sessions unless explicitly denied
on a session-by-session basis
Set the STMT_CACHE environment variable to “0” (off) or “1” (on)
Execute a “set statement cache [ on | off ]”command
68
SQL Statement Cache:Enabling and Sizing
STMT_CACHE_SIZE $ONCONFIG parameter:determines the size of the SSC in KBs, default = 524 kb
If SSC is full and all statements are active, if additional statements qualify for inclusion the SSC will expand to hold them. As a statement’s user counts drop to zero, the SSC will flush them and reduce back to STMT_CACHE_SIZE
69
SQL Statement Cache:Monitoring and Tuning
“onstat -g cac stmt” displays size of the SSC the cached statements, who’s currently executing a statement, and the number of times it’s been used
The “hash” column is not the statement’s hash value, rather the hash bucket in which the statement resides
To tune, monitor the size of the SSC over time, then resize as appropriate
70
SQL Statement Cache:Monitoring and Tuning
Use the STMT_CACHE_DEBUG environment variable to trace what happens with any given SQL statement
export STMT_CACHE_DEBUG=debug_value:path_to_file
debug_value can be “1” (basic) or “2” (extended)
71
SSC identity (OK:status=1): select * from yoyoSSC link (semantic error): select * from yoyoSSC identity (stmt not found): create table yoyo (col1 int) in mustang_2SSC qualify (invalid stmt type): create table yoyo (col1 int) in mustang_2SSC identity (stmt not found): insert into yoyo values (1)SSC qualify (OK): insert into yoyo values (1)SSC key insert (OK): insert into yoyo values (1)SSC full insert (OK): insert into yoyo values (1)SSC identity (stmt not found): insert into yoyo values (2)SSC qualify (OK): insert into yoyo values (2)SSC key insert (OK): insert into yoyo values (2)SSC full insert (OK): insert into yoyo values (2)SSC identity (OK:status=1): insert into yoyo values (2)SSC link (OK): insert into yoyo values (2)SSC identity (stmt not found): drop table yoyoSSC qualify (invalid stmt type): drop table yoyo
SQL Statement Cache:Monitoring and Tuning
72
New $ONCONFIG parameters: STMT_CACHE_HITS
Number of statement hits before entered into the cache
Values: 0 – statement immediately inserted N – first time statement used, key-only entry made in cache,
from 2 to N counter incremented. After N uses, full statement entered into cache
Want to set to "1" (one) to prevent ad-hoc statements from clogging cache
SQL Statement Cache:Monitoring and Tuning
73
STMT_CACHE_NOLIMITControls insertion of qualified statements into cache after its size
is greater than the STMT_CACHE_SIZE value. While statement cache can grow, this acts as a governor to stop unrestrained growth
Values: 0 – no new statements are inserted after STMT_CACHE_SIZE
reached 1 – unrestricted growth of cache is allowed
SQL Statement Cache:Monitoring and Tuning
74
STMT_CACHE_NUMPOOLSpecifies number of pools to be allocated to the SQL Statement
Cache
Values: 1 (default) to 256
As statement cache increases, number of pools supporting the cache may become a bottleneck. Use the
"onstat –g ssc pool" and "onstat –g spi" commands to monitor usage and spin waits. See pages 4-49 – 4-52 in the Performance Guide for more information.
SQL Statement Cache:Monitoring and Tuning
75
Raw / Standard TablesRaw / Standard Tables
76
Raw / Standard TablesRaw tables have been in XPS for quite some time
and are used in conjunction with its SQL-based HPL or massive static data manipulation
Intended for initial loading and data validation
Similar behavioral attributes regardless of database logging mode, i.e., don't try to use them in transactions.
77
Raw / Standard TablesRaw table attributes
nonlogged permanent table, behaves like table in a nonlogging database
uses light appends, adds rows quickly to end of each table fragments
updates, inserts, and deletes are supported but not logged Can not support indexes, referential constraints, or rollback Can only restore from last physical backup if not updated
since that backup, no MiT restore Fast recovery rolls back incomplete transactions on
STANDARD tables but not on RAW tables Wicked fast to use if done properly!!!
78
Raw / Standard TablesSyntaxcreate raw table xyz
( yada, yada
) in etc., etc.
fragment by whatever;
alter table xyz type [ raw | standard ];
When altered for massive manipulation, make sure you backup the table after converting it back from raw mode!!
79
Changes to "explain mode"Changes to "explain mode"
80
Explain ModeCan now get sqexplain.out information without having to
execute the DML statement!!!!
Two ways to activate this feature: per statement with a directive
select --+explain avoid_execute * from xyz
AVOID_EXECUTE directive works in conjunction with EXPLAIN directive. If EXPLAIN directives is omitted, i.e. if only AVOID_EXECUTE directive is used,the query will not be executed and no explain output will be generated.
81
Explain Mode Set as part of a statement block
set explain on avoid_execute;
After executing this statement, the server writes to the sqexplain.out file without executing any statements until SET EXPLAIN ON/OFF is executed.
82
What Are FuzzyCheckpoints???What Are FuzzyCheckpoints???
83
Checkpoints
IDS 9.2x introduced “fuzzy” operations and checkpoints to increase transactional throughput
Two types of checkpoints:
Full or “sync” Fuzzy
84
Steps of a SyncCheckpoint
1. Engine blocks threads from entering “critical sections” of code2. The page cleaner thread flushes the physical log buffer to log on
disk3. The page cleaner threads flush to disk all modified pages in the
buffer pool (chunk write)4. The checkpoint thread writes a checkpoint record to the logical log
buffer5. The logical log buffer is flushed to the current logical log file on
disk6. The physical log on disk is logically emptied (current entries can be
overwritten)7. The checkpoint thread updates the reserved pages with the
checkpoint record information
85
Good and bad:
logically and physically data, interrupts user activity, expensive (I/O hit), must tune to reduce duration
What Causes Sync Checkpoints to Occur? Physical log becomes 75% full “onmode -c” or “-ky” Administrative actions (adding dbspaces, altering tables) A backup or restore operation using ontape or ON-Bar End of fast recovery or full recovery Reuse of a logical log containing the oldest fuzzy operation not
yet synced to disk LTXHWM reached with fuzzy transactions Through the onmonitor menu
86
Fuzzy CheckpointsRecord “results” of fuzzy operations in the logical log
Goal: reduce or eliminate checkpoint interruptions, eliminate the physical log (over 3 - 4 phases)
Cost: increased logical recovery time
Note: Fuzzy checkpoints are not used in HADR environments
87
Fuzzy Checkpoints
Things to know: Inserts, updates, delete DML statements Currently supports “built-in” data types only (character,
numeric values) Logged databases Not “old” pages. Determined by page timestamp (4
byte, cycles 2 gb -> -2 gb -> 0) No BEFORE images are generated in the physical log During “fuzzy” checkpoints, buffered information on
fuzzy operations is **not** flushed to disk After a fuzzy checkpoint, disks are not physically
consistent
88
Steps of a Fuzzy Checkpoint
1. Engine blocks threads from entering “critical sections” of code
2. The page cleaner threads flush to disk all **non-fuzzy** modified pages in the buffer pool (chunk write) Note: MLRU queues still full
3. A Dirty Page Table (DPT) is constructed containing entries (buffer addresses) for fuzzy operations
4. The DPT is flushed to the current logical log on disk4A. The page cleaner thread flushes the physical and logical log buffers to disk
5. The checkpoint thread updates the reserved pages with the checkpoint information including the Log Sequence Number (LSN).
6. The physical log on disk is logically emptied (current entries can be overwritten)
89
Log Sequence Number (LSN)
A value representing a position in the logical log (log # and log position)
Stored in each dirty buffer heading containing fuzzy operation results along with a newly-stored timestamp
The oldest LSN is tracked in shared memory because it contains the oldest fuzzy operation. Oldest LSN can not be further back than 1 logical log and 2 checkpoints
Logs after the LSN can not be freed for re-use until a checkpoint has occurred
Net Net -- large logical logs are GOOD!!!
90
What Causes Fuzzy Checkpoints to Occur? Checkpoint interval has elapsed Physical log becomes 75% full onmode -c fuzzy Reuse of a logical log containing the last
checkpoint (either fuzzy or sync)
91
Recovery - Sync Checkpoint
1. Physical log data used to return all disk pages to original “synced” state (physical restore)
2. The most recent checkpoint (sync) record is located in the logical log files
3. All subsequent logical log records are rolled forward
4. Uncommitted transactions are rolled back
92
Recovery - Fuzzy Checkpoint
1. Physical restore proceeds as normal however disk pages are not physically consistent since there are modified pages that were not logged in the physical log
2. Logical recovery begins in two phases: Phase A -- reapply DPT records Phase B -- post checkpoint records
93
Recovery - Fuzzy Checkpoint
Phase A - Logical Recovery:1. Locate the oldest LSN representing unwritten data2. Selectively apply fuzzy operations from the
DPT/logical log from the oldest remaining logical log up to the most recent checkpoint
Conditional update based on page timestamp - if DPT record is “newer” than page timestamp, then the change is applied
Result: similar to end of physical restore when using sync checkpoints
94
Recovery - Fuzzy Checkpoint
Phase B - Logical Recovery:1. Beginning with last checkpoint, roll forward
subsequent logical log records.
2. Uncommitted transactions are rolled back
Result: similar to the logical restore when using sync checkpoints. Disks are physically and logically consistent to last recorded transaction
95
Fuzzy Checkpoints
Buffer structures, logical log records, and reserved pages are automatically upgraded when migrating to 9.x
Fuzzy checkpoints enabled by default. Set the NOFUZZYCKPT environment variable to true/on/1 to disable them prior to initializing shared memory
Set the TRACEFUZZYCKPT environment variable to trace fuzzy checkpoints. It will affect instance performance but will log in the MSGLOG the number of buffers that remain dirty after a checkpoint is completed. It also lists the location of the old LSN
96
IBM Informix MaxConnectIBM Informix MaxConnect
97
IBM Informix MaxConnectIBM Informix MaxConnect is a middle-ware communications component that removes the overhead of managing user connections from the instance
Helps you get scale your user connections past the effective 2,000 user connection limit without any change or resource impact to the database server
It supports 7.3 and 9.2x+ Unix ports of the engine but accepts user connections from any platform
NO CHANGE to existing applications!!
98
IBM Informix MaxConnect Performance
BaaN IVc World record: 11,445 BRUs on HP N4000 (8CPU, 16GB) and IDS 21% better than Oracle on identical HP system Beat Oracle’s 64-CPU Sun E10000 result 18,650 database connections using 98 transports !!
99
MaxConnect
MaxConnectIDS/Fnd
SQLClients
For large, hardware-partitionable SMP systems
MaxConnect
MaxConnect
SQLClients
Database server system
IDS/Fnd
IBM Informix MaxConnect Architecture
100
IBM Informix MaxConnect Configuration An IBM MaxConnect instance connects to only one DB
server instance Connections are managed through the SQLHOSTS file
with a new connection protocol -- imc Clients change hostname and port address (maybe) to
point to the IBM MaxConnect instance, instance name remains the same
IBM MaxConnect instance has an ADMIN entry, a client-facing entry, and a DB server-facing entry
DB instance has a DBSERVERALIAS and a DBSERVERNAME entry
101
Client SQLHOSTS file:ebro_tcp ontlitcp maxcon_1 1526
MaxConnect SQLHOSTS file:ebro_tcp ontlitcp maxcon_1 1526ebro_db ontliimc cronus 1575ebro_adm onsoctcpmaxconn_1 1600
Server SQLHOSTS file:ebro onipcshm cronus ebro_ph
(DBSERVERNAME)ebro_tcp ontliimc cronus 1575
(DBSERVERALIAS)
IBM Informix MaxConnect Configuration
102
NETTYPE entry changes:
ipcshm,1,20,CPU
# tlitcp,5,200,CPU
tliimc,1,2000,CPU or NET
MaxConnect instance environment variables:IMCONFIG - location of MaxConnect config file -
default is $INFORMIXDIR/etc/IMCconfigIMCSERVER - Client-side MaxConnect instance
nameIMCADMIN - Admin-side MaxConnect instance
name
IBM Informix MaxConnect Configuration
103
IBM Informix MaxConnect config file contains:
IMCLOG - location of MaxConnect message log
IMCWORKERTHREADS - # of worker threads, range = 1 -> 64
IMCWORKERDELAY - amt of time MaxConnect will wait and pool messages to transmit to DB server default = 0, range = 0 -> 100,000 ms
IMCTRANSPORTS - # of transport connections to DB
server. Default = 2, range = 1 -> 64, ratio = 1/100 users
IBM Informix MaxConnect Configuration
104
IBM Informix MaxConnect Tuning
Use “onstat -g imc”, the imcadmin utility, or the ISA to monitor and tune IBM Informix MaxConnect
Tune the NETTYPE sessions parameter by monitoring the q-exceed and alloc/max values. May also need to tweak the IFX_NETBUF_PVPOOL_SIZE variable
Tune IMCTRANSPORTS by monitoring blocked or partial writes
Tune IMCWORKERTHREADS by monitoring AvgQlen if > 5, increase number of threads
Tune IMCWORKERDELAY by summing and comparing histograms against AvgQlen
105
Large Objects, SBSpacesand
Dynamic Lock Allocation
Large Objects, SBSpacesand
Dynamic Lock Allocation
106
I/O size configurable by page size
configurable by page size, the data is still stored in database server sized pages, but the I/O is configurable
Buffer pool usage
partition blobs use the buffer pool, blobspace blobs use private buffers
you can choose between light I/O or normal buffering when the smart-large-object is opened
Updating inserts and deletes only ; there are no direct updates
updates are done in place or moved as needed
Data row/blob mapping
one large-object can be referenced by one data row
one large-object can be referenced by one or more sources
Simple Smart
Large Objects and SBSpaces
IBM IDS has 4 types of LOs: Simple - BYTE & TEXT Smart - CLOB & BLOB
107
Large Objects and SBSpaces
Access to C/BLOBs is faster due to “portion-level” addressing Default characteristics are easy to override With sufficient resources, C/BLOBs are transactionally recoverable
Simple SmartLoggingMethod
logging dependson the table wherethe large-object isstored
logging can be turned on or off at thelarge-object level
Size 2 GB maximum 4 TB maximum
LockingGranularity
all or nothing can lock portions or all of the object
Reads andwrites
only the wholelarge-object canbe read
all or a portion of the large-object canbe read
Fragmentation all large-objectsfor a table arestored in the sameblobspace
ability to store the large-objects forone table in multiple sbspaces
108
header (reserved) pages
metadata areaUser Data
Size and location ofthe metadata area isconfigurable atsbspace and/or chunk creation
Metadata is alwayslogged regardless ofdatabase or sbspacelogging mode
Large Objects and SBSpaces
Smart BLOBs are stored in “smart” BLOBSpaces (sbspace) which contain meta-data as well as user data
109
Large Objects and SBSpaces
onspaces -S name -g pageunit -p pathname -o offset-s size [ -m pathname offset ] -Ms mdsize -Mo mdoffset -Df default list -t
-Mo mdoffset offset, in kbs, into the disk partition where metadata will be stored
-Ms mdsize the size, in kbs, of the metadata area for the initial chunk, remainder is user data space -- not recommended though possible. Each SLO needs ~500 bytes of metadata space
-Df default list default specifications for objects stored in the sbspace. Comma separated, in double quotes (“ “)
-t new option in 9.3 for temporary sbspaces to store temporary smart LOBs
110
Tag Values Default DescriptionACCESSTIME ON or
OFFOFF ON – instance tracks access time
of SLOs stored in the sbspace.AVG_LO_SIZE For
WindowsNT:4 to 2^31
For UNIX:2 to 2^31
8 Average size, in kbs, of SLOsstored in the sbspace. Used tocalculate metadata area size if -Msoption is not specified.Error 131 is returned if you run outof metadata area in the sbspace.To allocate additional chunks tothe sbspace that consist ofmetadata area only, use the -Msoption.
BUFFERING ON orOFF
ON ON - instance uses residentportion buffer pool for SLO I/Ooperations.OFF - instance uses light I/Obuffers in the virtual portion(lightweight I/O operations) –PREFERRED !!!!
LOCK_MODE RANGE orBLOB
BLOB RANGE - only a range of bytes inthe SLO is locked.BLOB - the entire SLO is locked.
Large Objects and SBSpaces
111
Tag Values Default Description LOGGING ON or
OFF OFF ON - instance logs changes to the
SLOs in the user data. Create a level-0 backup of the sbspace after creating/changing
EXTENT_SIZE 4 to 2^31
16 Size, in kbs, of the first disk allocation for SLO storage when you create the table. Let the system select the EXTENT_SIZE value.
MIN_EXT_SIZE 2 to 2^31
4 Minimum amount of space, in kbs, to allocate for each SLO.
NEXT_SIZE 4 to 2^31
16 Size, in kbs, of the next disk allocation for smart large objects when the initial extent becomes full. Let the system select the NXT_SIZE value.
Create a level 0 backup after creating a sbspace
Large Objects and SBSpaces
112
Large Objects and SBSpaces:New stuff
Temporary SBSpaces: Like temporary DBSpaces -- no logging whatsoever occurs Set with the SBSPACETEMP $ONCONFIG parameter
Temporary smart LOBs: To create a temp smartblob, set the LO_CREATE_TEMP flag in
the ifx_lo_specset_flags or mi_lo_specset_flags function. Use mi_lo_copy or ifx_lo_copy to create a permanent smart large object from a temporary smart large object.
If you put a temporary smartblob in a permanent sbspace it gets deleted at the end of session
113
Metadata stealing:
40% of user data area reserved for either meta or user data, is allocated as needed to either portion (10% increments)
If metadata area is filling, messages in MSG_PATH
If metadata fills, and reserve is empty, no more S-BLOBs can be inserted even if user data area has free space
Must add more metadata space
Large Objects and SBSpaces:New stuff
header (reserved) pages
metadata areaUser Data
114
Large Objects and SBSpaces
To add a chunk, you can specify whether the chunk will contain only user data, metadata, or both:
onspaces -a sbspace -p pathname -o offset -s size
[-Ms mdsize -Mo mdoffset | -U ][ -m pathname offset ]
To drop the sbspace:onspaces -d sbspacename -f (force)
115
create table products (prod_sku int, prod_desc clob, prod_photo blob) put prod_desc in (sbspace_1, sbspace_2), prod_photo in (sbspace_9, sbspace_10), fragment by expression (prod_sku < 1000 in db_1, prod_sku >= 1000 in db_2);
Large Objects and SBSpaces
SLOs can be distributed in round-robin form among sbspaces like “regular” data
Creates a 72 bytedescriptor “handle”as opposed to 56bytes with “simple”LOs
116
Large Objects and SBSpaces
IBM recommends that you convert “simple” LOs to their “smart” counterparts as soon as feasible
You can, in many cases, use the "alter table" command to make the change(s):
alter table t_name modify col_name [ clob | blob ]put col_name in (sbspace(s));
117
Large Objects and SBSpaces
Monitoring / tuningoncheck -cs -cS / -ps -pS to print/view metadata
-ce / -pe has additional sbspace information
onstat -g smb s | c | h | e | tsmb s sbspace summary informationsmb c sbspace chunk informationsmb fdd LO file descriptor tablesmb lod LO header tablesmb t timing statistics for enabled server,
not well documentedsmb e undocumented
onstat -k has additional columns to document SLO byte range locks
onstat -d has additional chunk-related columns for sbspaces
118
Chunk adjunct partition
Level 1 backup partition
Level 2 backup partition
LO header partition
Sbspace descriptor partition
user-data free-list partition
header (reserved) pages
metadata area
Large Objects and SBSpaces
Backing up and restoring SLOs and sbspaces now operates like regular dbspaces -- at the page level. As a result, you can now have granular and truly incremental SLO/sbspace backups.
119
OtherOtherspacesspaces
Sbspace Sbspace headerheader
Metadata Metadata pagespages
Archive Archive descriptor pagesdescriptor pages
sbspace sbspace pagespages
Large Objects and SBSpaces
Backup: Metadata LO headers are read and a list made of pages to be backed up All individual page timestamps are compared too since updates can occur
without affecting the LO header timestamp Qualifying data is sent to the backup program through large buffer accesses
Restore: Sbspace header and metadata information is restored Archive desriptor pages contain page addresses and extent sizes for the
sbspace pages that follow. This is held in shared memory during the operation
Sbspace pages are restored to their correct places
120
Dynamic Lock Allocation In IBM IDS, the locking mechanism changes. LOCKS is now a starting allocation (resident portion) If the lock table fills, additional locks are allocated (x 16) minimum allocation: smaller of current lock table or 100,00 (virtual portion) To prevent lock table flooding, range locks will be upgraded to the whole SLO if a single transaction uses > 10% of lock table “Byte range” locks are what lock portions of a SLO. They are sorted lists of rowids, partition #s, and byte address ranges
121
Dynamic Lock Allocation Range locks can be joined or split as needed if ranges to
be locked adjoin, overlap, or become separate
Ranges locks are not stored in the lock hash table, rather an “intent” lock is placed for the LO header. Byte range locks are linked to associated hash intent lock then sorted by byte number
Byte ranges locks allocated if: < 50% of lock table is being used 50% <= lock table <= 70%, whole SLO lock will be attempted, if
unsuccessful, byte lock allocated > 70% whole SLO locks only, existing locks will be upgraded
automatically
122
Complex and User-Defined Data Types
Complex and User-Defined Data Types
123
OpaqueOpaqueOpaqueOpaque DistinctDistinctDistinctDistinct
Row Data TypeRow Data TypeRow Data TypeRow Data Type
NamedNamedNamedNamed UnnamedUnnamedUnnamedUnnamed
CollectionCollectionCollectionCollection
MultisetMultisetMultisetMultiset ListListListList
SetSetSetSet
User-DefinedUser-DefinedUser-DefinedUser-DefinedComplexComplexComplexComplex
Extended DataExtended DataTypesTypes
Extended DataExtended DataTypesTypes
BooleanBooleanInt8Int8
Serial8Serial8LvarcharLvarchar
BooleanBooleanInt8Int8
Serial8Serial8LvarcharLvarchar
New Built-inNew Built-inTypesTypes
New Built-inNew Built-inTypesTypes
Existing Built-inExisting Built-inTypesTypes
Data TypesData Types
Complex and User-Defined Data Types
124
Built-in Data Types
New Built-In: int8 (8 bytes) serial8 (8 bytes)
Range is -9,223,372,036,854,775,807 to 9,223,372,036,854,775,807 must add UC to serial8 to ensure uniqueness one serial8 and serial per table
boolean valid values: “t”, “f”, null. case sensitive
lvarchar (variable length character data type; 32k maximum – check your system for on-disk maximum)
125
OpaqueOpaqueOpaqueOpaque DistinctDistinctDistinctDistinct
Row Data TypeRow Data TypeRow Data TypeRow Data Type
NamedNamedNamedNamed UnnamedUnnamedUnnamedUnnamed
CollectionCollectionCollectionCollection
MultisetMultisetMultisetMultiset ListListListList
SetSetSetSet
User-DefinedUser-DefinedUser-DefinedUser-DefinedComplexComplexComplexComplex
Extended DataExtended DataTypesTypes
Extended DataExtended DataTypesTypes
BooleanBooleanInt8Int8
Serial8Serial8LvarcharLvarchar
BooleanBooleanInt8Int8
Serial8Serial8LvarcharLvarchar
New Built-inNew Built-inTypesTypes
New Built-inNew Built-inTypesTypes
Existing Built-inExisting Built-inTypesTypes
Data TypesData Types
Complex and User-Defined Data Types
126
No “alter type” statement, must drop and recreate
Sys Adm is more complexIntuitive
Not simple SQLRefers to a group of elements by a single name
More complexLess coding
DISADVANTAGESADVANTAGES
Complex Data Types:Row
analogous to C structure, come in two “kinds”NAMED - strongly typed, ID’ed by name, has inheritance, used
to build columns and tables
UNNAMED - weakly typed, ID’ed by structure, no inheritance, used to build columns
Can contain built-in, collection, opaque, distinct, another row type data types
Caveat: no serial or serial8
127
Named: create row type name_t (fname char(20), lname char(20));
create row type address_t (street_1 char(20), street_2 char(20), city char(20), state char(2), zip char(9));
create table student(student_id serial, name name_t, address address_t, company char(30));
Unnamed:ROW (a int, b char (10))
Note: is also equal to ROW(x int, y char(10))
create table part (part_id serial,cost decimal,part_dimensions row
(length decimal,width decimal, height decimal, weight decimal));
Complex Data Types:Row
128
Using Named Row Types in SQL statements:
Insert statement:insert into student values (1234, row (“John”,”Doe”)::name_t, row ("1234 Main Street","", "Anytown","TX","75022")::address_t, "Informix Software")
Select statement:select * from student where name.lname matches "Doe”;
Result set: student_id 1234 name ROW('John ','Doe ') address ROW('1234 Main Street ',' ,'Anytown ','TX','75022 ') company Informix Software
Use of datatypeUse of datatypekeywordkeywordUse of datatypeUse of datatypekeywordkeyword
Cast the row type!Cast the row type!Cast the row type!Cast the row type!
Access data withAccess data with‘‘dot’ notationdot’ notationAccess data withAccess data with‘‘dot’ notationdot’ notation
Complex Data Types:Row
129
Complex Data Types:Row
To drop a named row type:
drop row type address_t restrict;
130
OpaqueOpaqueOpaqueOpaque DistinctDistinctDistinctDistinct
Row Data TypeRow Data TypeRow Data TypeRow Data Type
NamedNamedNamedNamed UnnamedUnnamedUnnamedUnnamed
CollectionCollectionCollectionCollection
MultisetMultisetMultisetMultiset ListListListList
SetSetSetSet
User-DefinedUser-DefinedUser-DefinedUser-DefinedComplexComplexComplexComplex
Extended DataExtended DataTypesTypes
Extended DataExtended DataTypesTypes
BooleanBooleanInt8Int8
Serial8Serial8LvarcharLvarchar
BooleanBooleanInt8Int8
Serial8Serial8LvarcharLvarchar
New Built-inNew Built-inTypesTypes
New Built-inNew Built-inTypesTypes
Existing Built-inExisting Built-inTypesTypes
Data TypesData Types
Complex and User-Defined Data Types
131
Complex Data Types:CollectionsGrouping of elements of the same datatype
(char, int), max size = 32 KB
Used when The data is meaningless without the context of the
other members in the collection (e.g., golf scores, to-do list, set of names)
Individual data elements are not likely to be directly queried by position
The maximum number of data elements is less than 32
Can be null
132
Complex Data Types:Collections
Three kinds of collections:
Set - unordered, no duplicates allowed set {“apple”, ”orange”, ”grapefruit”, “plum”}
Multiset - unordered, duplicates allowedmultiset {“apple”, “orange”, “grapefruit”, “apple”, “plum”, “grapefruit”}
List - ordered, duplicates allowedlist {“apple”, “orange”, “grapefruit”, “apple”, “plum”, “grapefruit”}
133
Complex Data Types:Collections
create table class(class_id serial, class_name varchar(60), description lvarchar,
prereqs set(char(20) not null));
Insert syntax is similar to named row types:
insert into class values (300, “Performance and Tuning”, “Covers advanced information on tuning the Informix Dynamic Server”, (SET{“RDD”,”BSQL”}));
Use the “in” keyword to query values in a collectionselect * from class where (“ASQL”) in prereqs;
define xyz char(20)define set_var set(char(20))select prereqs into set_var from class where class_id = 300
foreach del_set_cursor forselect * into xyz from table(set_var)if xyz matches “RDD” then
delete from table(set_var) where current of del_set_cursorend if
end foreach
134
Complex Data Types:Collections
You can not update one element in a collection, you must replace the whole collection:
update class set prereqs = (set{“RDD”,”ASQL”,”BSQL”}) where class_id = 300;
update class set prereqs = set_char where class_id = 300;
135
OpaqueOpaqueOpaqueOpaque DistinctDistinctDistinctDistinct
Row Data TypeRow Data TypeRow Data TypeRow Data Type
NamedNamedNamedNamed UnnamedUnnamedUnnamedUnnamed
CollectionCollectionCollectionCollection
MultisetMultisetMultisetMultiset ListListListList
SetSetSetSet
User-DefinedUser-DefinedUser-DefinedUser-DefinedComplexComplexComplexComplex
Extended DataExtended DataTypesTypes
Extended DataExtended DataTypesTypes
BooleanBooleanInt8Int8
Serial8Serial8LvarcharLvarchar
BooleanBooleanInt8Int8
Serial8Serial8LvarcharLvarchar
New Built-inNew Built-inTypesTypes
New Built-inNew Built-inTypesTypes
Existing Built-inExisting Built-inTypesTypes
Data TypesData Types
Complex and User-Defined Data Types
136
User-Defined Data Types:Distinct
Two user-defined data types (UDTs):Distinct
data type modeled on an existing data type has a unique name to distinguish it from other similar “types” inherits the internal structure from the source type might have operations and casts defined over its
source type
Opaque data type that is unknown to the database server you must define the internal structure, functions,
and operations
137
create distinct type dollar as decimal;create distinct type aus_dollar as decimal;
create table sales ( sku int, sales_date date, us_sales dollar, aus_sales aus_dollar);
insert into sales values (1234, today, 15.0::dollar,0::aus_dollar);insert into sales values (5678, today, 0::dollar, 75.0::aus_dollar);
select sku, (sum(us_sales) + sum(aus_sales)) from sales where sales_date = today group by 1;
error: 674 - routine (plus) can not be resolved
User-Defined Data Types:Distinct
138
Need to create some UDFs that handle the type and value conversion for you:
create function usdlr_to_ausdlr(parm1 dollar) returning aus_dollar specific usd_to_ausd; return (parm1::decimal * 1.8)::aus_dollar;end function;
create function ausdlr_to_usdlr(parm1 aus_dollar) returning dollar specific ausd_to_usd; return (parm1::decimal / 1.8)::dollar;end function;
select sku, (sum(us_sales) + sum(ausdlr_to_usdlr(aus_sales))::dollar) from sales where sales_date = today group by 1;
User-Defined Data Types:Distinct
139
User-Defined Data Types:Opaque
An opaque data type stores a single “value” that cannot be divided into components by the engine.
Implemented as C or Java structures and manipulated by a set of routines written in C or Java
An opaque “value” is stored in its entirety by the engine without any interpretation of the contents or its structure
All access to an opaque type is through functions written by the user. You define the storage size of the data type and input and output routines
140
User-Defined Data Types:Opaque
1. Create the C / Java data structure to represent the internal data structure
2. Write the support functions in C / Java3. Register the opaque data type with the “create opaque type”
statementcreate opaque type type_name ( internallength = length,
alignment = num_bytes);length is in bytes, alignment = 1,2,4,8 bytes (default = 4)
create opaque type type_name ( internallength = variable,maxlen = length);
length - default = 2 KB, max value = 32 kb
4. Register the support functions with the “create function” statement. If in Java, will be stored in sbspace(s)
141
create opaque type my_type(internallength=8, alignment=4);
create function support_in(lvarchar)
returning my_type with (not variant);external name “/funcs/my_type.so”language C
end function;
create implict cast (lvarchar as my_type with support_in);
5. Grant access to the opaque data type and support functions6. Write any user-defined functions needed to support the opaque data
type - input, output, destroy, compare, aggregates, send, receive, etc.
7. Provide any customized secondary-access methods for creating indexes
User-Defined Data Types:Opaque
142
Casts and CastingCasts and Casting
143
Casts allow you to make comparisons between values of different data types or substitute a value of one data type for a value of another data type.
create function ausdlr_to_usdlr(parm1 aus_dollar) returning dollar specific ausd_to_usd; return (parm1::decimal / 1.8)::dollar;end function;
Engine provides a number of “built-in” casts (int to decimal, numeric to char, etc.) for most built-in datatypes
Must create user-defined casts for user-defined types. Must be unique with respect to source and target data types
Can not create casts for collections, LOBs, or unnamed row types
Casts and Casting
144
Casts and Casting
Two kinds of user-defined casts:explicit
. . . where price >= (cast aus_dollar as dollar) . . . .
. . . return (parm1::decimal / 1.8)::dollar;create explicit cast (aus_dollar as dollar with us_dlr_to_us_dlr);
implicitcreate implicit cast (aus_dollar as dollar);create implicit cast (aus_dollar as dollar with aus_dlr_to_us_dlr);
Automatically invoked when:one data type is passed to a user-defined routine whose parameters are of
another data typeexpressions are evaluated that need to operate on two similar data types
145
Casts and Casting – Implicit Casts
select sum(us_sales) + sum(aus_sales) from sales; # 674: Routine (plus) can not be resolved.
create implicit cast (aus_dollar as dollar);select sum(us_sales) + sum(aus_sales) from sales;
(expression) 120.00
drop cast (aus_dollar as dollar);create implicit cast (aus_dollar as dollar with ausdlr_to_usdlr);select sum(us_sales) + sum(aus_sales) from sales;
(expression) 80.00
146
select sum(us_sales) + sum(aus_sales) from sales; # 674: Routine (plus) can not be resolved
create explicit cast (aus_dollar as dollar);select sum(us_sales) + sum(aus_sales) from sales; # 674: Routine (plus) can not be resolved
select sum(us_sales) + sum(aus_sales)::dollar from sales;(expression) 120.00
drop cast (aus_dollar as dollar);create explicit cast (aus_dollar as dollar with ausdlr_to_usdlr);select sum(us_sales) + sum(aus_sales)::dollar from sales;
(expression) 80.00
Casts and Casting – Explicit Casts
147
Previous examples were “straight” casts. You can also create cast “functions” to cast types with dissimilar data structures
1. write the cast function in C / Java / SPL
create function opaque_to_opaque (input_arg my_type_1)returns my_type_2return cast(cast(input_arg as lvarchar) as my_type_2);
end function;
2. register the cast function with the “create function” command
3. register the cast with the “create cast” commandcreate explicit cast (my_type_1 as my_type_2 with opaque_to_opaque);
Casts and Casting
148
User-defined Routines(UDRs)
User-defined Routines(UDRs)
149
UDRs
There are User Defined Functions and Procedures (UDFs and UDPs)
Can be internal or external in nature. Internal are written in SPL, external can be written in C or Java.
If written in Java, compile into a class then .jar file. When registered in the engine, .jar file will be brought into a sbspace then executed when needed by the JVM in the engine.
If written in C, create a shared object library in a directory owned by “informix” with 755 permissions.
150
UDRs UDRs should be concise and precise. Don’t want engine wasting resources plowing though verbose code. The values returned should be relevant from a business perspective (e.g. proper domain)
Make sure you include error handling in the code
**ALWAYS** use a “specific” name, or alias, for every UDR to properly identify each one - used to drop, grant, or revoke permissions as well as to permit function overloading
151
UDRs
Function overloading occurs when two or more functions have the same name but different signatures
signature = UDR name and parameter list
create function plus (in_1 dollar, in_2 aus_dollar) . .create function plus (in_1 aus_dollar, in_2 lira) . . .create function plus (in_1 lira, in_2 aus_dollar) . . .
{Note: these probably should be formalized into user-defined aggregates}
When properly registered, instance will use the data types passed to determine which UDR should be used
152
UDRs create function routine_name (param_list)
{ returns | returning } typename[ specific specific_name ][ with (internal | handlesnulls | [ [not] variant]
| class=“vp_class”) ]external name ‘full_path_name_of_file‘language language [ [not] variant ]
end function;
in the $ONCONFIG file:VPCLASS fince_vp ,num=2 # vps for finance UDRs
oronmode -p +2 fince_vp
USE UDVPs FOR UDRs!!!!
153
UDRs To drop UDRs, must either use full signature
drop function plus(lire, aus_dollar);
or the “specific” name
drop specific function lir_to_ausdlr;
154
UDRs Default behavior of UDRs is single-threaded, can be “parallelized” if the following conditions are met:
only in DSS environments (PDQPRIORITY > 1) "parallelizable" keyword is added to UDR creation
statement external UDRs only (C / Java) that use PDQ thread-
safe SAPI calls (see DataBlade manual for list of SAPI calls)
no complex types in input parameters or return values
multiple fragments must be scanned
155
UDRs Not directly called (singleton execution) not an iterator - called in loops to return multiple
values similar to with resume in SPL
On some UDRs, you can add “selectivity” and “cost” modifiers that will affect optimizer activities and query plans used. Not really familiar with this
156
UDRs
create function bigger(int,int) returns intspecific big_parallelwith(parallelizable,class=“test_vps”) external name“/path/plludr.bld(bigger)” language C;
alter function bigger (int,int) with (add parallelizable, add class=“test_vps”);
Can monitor the parallel execution of UDRs through “set explain out” command the ISA onstat -g mgm onstat -g ses sess_id onstat -g ath
157
DataBlade Modules and the Blade Manager
DataBlade Modules and the Blade Manager
158
DataBlade Modules DataBlade Modules extend the functionality of the engine by
adding new UDTs, casts, interfaces, UDRs, error messages, client code (where necessary) to manipulate the UDTs
Functionality available through SQL, SPL, API calls to external functions
Can be mixed and matched to support any application
While the code is loaded at an instance level, blade registration occurs at a database level. With registration information contained in the database system tables, if you drop the database, blades become unregistered
159
DataBlade Modules
Install the blade in $INFORMIXDIR/extend (tar, cpio, ?)
May have its own “install” script to brand binaries, use serial # / license key from engine install unless third-party requires unique key setup
Use the Blade Manager to manage blade registration graphical version available with NT and ISA command line only on Unix / Linux
Is case sensitive
160
DataBlade Modules BladeManager supports the following commands:
list [database] - print list of registered DataBlade modulesset user [user]set server [server]show modules - print list of DataBlade modules on servershow databases - print list of databases on servershow servers - print list of serversshow client - print DataBlade modules with files on clientregister [module] [database]unregister [module] [database]add client [module] - install DataBlade module's client filesdel client [module] - remove DataBlade module's client filesinfo [module] - print information about a DataBlade moduledel logs - delete all the log filesshow log - prompt to display a logshow last log - display log from last actionhelp, ?bye, quit, exitset confirm on|off
161
DataBlade Modules
mustang>show databaseDatabases on server: agg1 dcs_demo dps_demo ireach kiosk media360 my_test_db visclass20
mustang>list media360DataBlade modules registered in database media360: lld.1.20.tc2 TXT.1.10.TC5 ifxrltree.2.00 ifxbuiltins.1.1 web.4.10.TC1 VTS.1.20.TC1 video.2.00.TC2
162
HDR / Enterprise Replication –A Snapshot of New Features
HDR / Enterprise Replication –A Snapshot of New Features
163
HDR and 9.4 HDR now supports:
Enterprise Replication (co-existence) Logged Smart Blob Spaces Some DataBlades (when associated smart blob
spaces are logged) Supported DataBlades:
• Spatial• Geodetic• Web• Time-Series• Excalibur Text
164
ER – A SnapshotNew Stuff
DataSync performance improvements Spooling improvements Serial column primary key support Changed columns sent with update, partial row update Smart LOBs support No raw table support Distinct / opaque type support but not complex OLD!! 9.4 Feature: Transaction size limit increases from amount of
shared memory to 4 TB 9.4 Feature: support for additional extensible types:
Row Types Lists Sets Multi-Sets
165
DataSync performance improvements include: Support of intra-replicate parallelism
Parallel apply with ordered commits Operations for a given row are serialized
Necessary to prevent ‘delete before insert’
Some tables are always serialized Page level locking or small tables
Out-of-order commits Allows commit order on target to be different than it was on source
No prior cross replicate relationships No prior referential constraint relationships
ER – A SnapshotNew Stuff
166
Decreased logical log consumption DataSync now uses buffered logging Acknowledgements are ‘buddy-bunched’ Shipping of ACKs coordinated with log flush, every 5 seconds or 50 Txn
Is automatic and self-regulated CDRM_Monitor thread
Maintains weighted history of lock failures Controls the degree of parallelism allowed Monitors distance before DDRBLOCK Coordinates ACKs with log flush Recalculates Table Statistics, user may need to run update statistics on a
target only system
Does not provide for intra-transaction parallelism (is not PDQ)
ER – A SnapshotNew Stuff
167
9.30 ER interoperates with ER on 7.31 or 9.2x (but NOT 7.30)
Limitations 9.30 does not support replicate groups, all groups should be
deleted from 9.2x/7.31 servers before inter-operation begins 9.2x/7.31 do not support replicate sets; no replicate sets should
be defined until all 9.2x/7.31 servers have been converted to 9.30
PTS 128485: must be applied to 9.2x/7.31 server(s) before beginning interoperation
ER – A SnapshotNew Stuff
168
ER – A SnapshotResult!!
9.30
9.21
Apply on Source Apply on Target
169
Introduction to Javain the Server
Introduction to Javain the Server
170
EJB
EJB
Java
JVMJVM
Applet JavaBeanJavaBean JavaBeanJavaBean
RMI | Corba | IIOP RMI | Corba | IIOP
Web Server Web Server
Applet JavaBeanJavaBean JavaBeanJavaBean
Java DBDK
JavaBeanJavaBean EJBEJB
Java in the Server J/Foundation is the marriage of the Java and SQL languages
Can write internal or external Java routines that are accessed natively by the engine
171
Java in the Server:How It Works
Java UDRs execute on Java Virtual Processors Java VPs have the same capability as CPU VP
It can process any SQL query
Avoids the inter-process communication overhead for executing Java UDRs
The Java Virtual Machine (JVM) is embedded directly into the VP code
Java runs threads in parallel
Dynamically load Java VPs Increase JVPs to balance application load
172
IBM IDS 9.30 was enhanced to take advantage of JDK 1.3 There are 3 different Java VM’s
Classic VM: jdk1.2.x VM Client HotSpot VM: tuned for user-interaction apps needing short start-
up & smaller memory footprint (JIT less aggressive) HotSpot is from Sun, IBM has other VMs that are not based on HotSpot technology.
Server HotSpot VM: tuned for servers, used in our J/Foundation product
Classic VM was used for IDS 9.21, and is not supported for the 9.30 release. This is in line with the vendor roadmaps for JDK.
Java in the Server:New Stuff
173
Performance improvements in 9.3 J/Foundation include: thread pooling, code optimization of Java UDR invocation, support for HotSpot Server VM
Performance improvements will vary by the application, but internally we’ve seen an improvement of approximately 135% for our test suites.
Java in the Server:New Stuff
174
Java in the Server:Configuration
New ONCONFIG Parameters:
VPCLASS jvp,num=1
JVPJAVAHOME /usr/java
JVPHOME /opt/informix/extend/krakatoa
JVPPROPFILE /opt/informix/extend/krakatoa/.jvpprops
JVPLOGFILE /opt/informix/jvp.log
175
9.21 $ONCONFIG had the settings JDKVERSION 1.2 JVPJAVAVM hpi:jvm:java:net:math:zip:jpeg:dcpr
9.30 $ONCONFG will need the settings JDKVERSION 1.3 JVPJAVAVM hpi:server:verify:java:net:zip:jpeg
Look at examples in onconfig.std
Java in the Server:Configuration
176
Java in the Server:JDBC Driver Information We provide a “Type 4” JDBC driver
Type 1 JDBC-ODBC bridge provides JDBC access via most ODBC drivers
Type 2 A native API and partly Java driver that converts JDBC calls into calls on the client API for Oracle, Sybase, Informix, DB2 etc.
Type 3 A net protocol, all Java driver that translates JDBC calls into a DBMS independent net protocol which is then translated to a DBMS protocol by a server
Type 4 A native protocol all-Java driver converts JDBC calls into the network protocol used by DBMSs directly. This allows a direct call from the client machine to the DBMS server
177
Java in the ServerDevelopment Install JDK 1.3 on development machine(s)
Install any O/S patches required by vendor
We recommend that you recompile your Java UDR’s using JDK 1.3
For more information on development, please see the user guides and the Informix Developers Network website, “Java” and “DataBlade Developers” corners
178
Java in the ServerDebugging
Tip #1If you see this:
9431: Can't find system class or method or library (/vobs/tristarm/sqldist/extend/krakatoa/jre/lib/sparc/server/libjvm.so).
It may mean that you have an invalid entry for JVPJAVAM in your ONCONFIG file
179
Java in the ServerDebugging
Tip # 2If you see this in MSG_PATH file:10:33:12 Cannot load lib
/vobs/tristarm/sqldist/extend/krakatoa/jre/lib/sparc/server/libjvm.so: error : ld.so.1: oninit: fatal: libCrun.so.1: open failed: No such file or directory
This could also be a sign of an invalid entry for JVPJAVAM in your $ONCONFIG file
180
Java in the ServerDebuggingTip # 3If you see this:9431: Can't find system class or method or library
(/vobs/tristarm/sqldist/extend/krakatoa/jre/lib/sparc/server/libmath.so).
Or this in MSG_PATH: 10:33:12 Cannot load lib
/vobs/tristarm/sqldist/extend/krakatoa/jre/lib/sparc/libmath.so: error : ld.so.1: oninit: fatal: /vobs/tristarm/sqldist/extend/krakatoa/jre/lib/sparc/libmath.so : open failed: No such file or directory
You have the math library specified in your JVPJAVAM, remove it.
181
Questions???