karen cannell [email protected]
DESCRIPTION
How Do I Load Data … Let Me Count The Ways. APEX Data Loading Options. Karen Cannell [email protected]. http://www.thtechnology.com. How Do I Load Data? : Agenda. Oracle Data Loading Options DBAs - Developers - End Users APEX Data Loading Options File Upload - PowerPoint PPT PresentationTRANSCRIPT
TH Technology
Karen [email protected]
http://www.thtechnology.com
How Do I Load Data …Let Me Count The WaysAPEX Data Loading Options
How Do I Load Data …
TH Technology
How Do I Load Data? : Agenda
Oracle Data Loading OptionsDBAs - Developers - End Users
APEX Data Loading OptionsFile UploadData Load Wizard2
XLS Upload – APEX ListenerWebSheets – Copy/Paste, Add Row
TH Technology
How Do I Load Data …
About Me … Karen Cannell ~ Consultant, TH Technology
SW Engineer 25+ years, Oracle since 1994. Building APEX apps for government, medical, engineering industries.Leveraging the Oracle 10g,11g (now 12c) suite of tools
Beginning Application Express, APress, 2011Agile Application Express, APress, 2011
Editor, ODTUG Technical JournalVolunteer to author ODTUG Journal Articles!
Using APEX since the HTMLDB beginning
How Do I Load Data …
TH Technology
About You … (Audience Background)
New to APEX?APEX Experience?Previous Tools?
APEX Training?Version 2.0? 3.1 ? 3.2 ? 4.1?
How Do I Load Data …
TH Technology
How DO I Load Data into Oracle?
Load Data Into Oracle7,340,000 hits
SQL LoaderOracle Database UtilitiesLoad Excel Data into Oracle
How Do I Load Data …
TH Technology
How DO I Load Data into Oracle?
Load Data Into APEX317,000 hits
Import Excel Data …APEX Data LoaderLoad Text File into APEX
How Do I Load Data …
TH Technology
How DO I Load Data into Oracle?
Load XLS Data Into APEX1,830,000 hits
Import Excel File …APEX Data LoaderLoad Excel Data …
Yikes!
TH Technology
How Do I Load Data …
Load Data into Oracle - Options
What Are the Options?Commercial Solutions - Excel-DB, QuickloadWarehouse Builder, ETL ToolsOracle Data Integratorhttp://www.oracle.com/technetwork/middleware/data-integrator/overview/index.html
Oracle Database UtilitiesOracle Data PumpSQL LoaderExternal Tables
TH Technology
How Do I Load Data …
Load Data into Oracle, cont’d
ODBC ConnectionsIDEs w/built-In Data LoadersCustom Code:
JavaPL/SQL Using UTL_FILEScripting from XLS ColumnsJava and PL/SQLPerlOther
TH Technology
How Do I Load Data …
Let’s Focus …
TH Technology
How Do I Load Data …
Load Data into APEX
Import Excel Data into Oracle …
APEX Data Loader
“Load Excel Data”
95% Say XLS but Really are CSV
The Apex Data Loader is a java based application that can bulk process the insert, update and delete on all object data into and
build queries to extract data out of salesforce.com using the Apex Web Services
(SOAP) API.
TH Technology
How Do I Load Data …
Our Problem
“Load Data into Oracle”40% Fixed-Format Text Files 60% Excel Spreadsheets Several XLS Formats200+ Files MonthlyLocations All Over the WorldSome Networked, Some NotVarying Volume of Data
TH Technology
How Do I Load Data …
Which to Choose?
It Depends ApplicationSourceTarget ( Table?)Amount of DataHow Often ? 1 or ManyAudience End UsersResources $$
Time $$
TH Technology
How Do I Load Data …
Which Make Sense for APEX ?
Oracle Database UtilitiesPL/SQL IDENo Additional Licenses PL/SQL Code No Complex SetupDeveloper-FriendlyEnd User Friendly
TH Technology
How Do I Load Data …
Oracle Database Utilities
Oracle Data PumpOracle DB to Oracle DBFaster than EXP and IMP
EXPDP 2x faster than EXPIMPDP 15-45x faster than IMP
Jobs can be RestartedSupports Network Import and Export
Load one instance form anotherRemote export
TH Technology
How Do I Load Data …
Oracle Database Utilities, cont’d
SQL LoaderExternal files into TablesVariety of formatsFilteringMultiple Table Load in One Load SessionConventionalDirect PathExternal Table Load
TH Technology
How Do I Load Data …
Oracle Database Utilities, cont’d
External TablesExternal files into TablesVariety of FormatsPreprocessing Capability
External Tables to PullRPT FilesCSV Files - That We Can Get
TH Technology
How Do I Load Data …
Oracle Database – Database Link
Don’t Forget the Database Link!SELECT …
FROM table@db_link
Simple, DirectRequires CommunicationAsk!
TH Technology
How Do I Load Data …
Use your IDE
SQL DeveloperPL/SQL DeveloperToadOthers …
Best for One-Time Load
TH Technology
How Do I Load Data …
SQL Developer – Data Load Option
Import Data … Wizard
TH Technology
How Do I Load Data …
SQL Developer – Import Data …
Import File Formats.XLS / XLSX.CSV.TSV (tab).DSV ( SQL
Server)
TH Technology
How Do I Load Data …
SQL Developer – Import Data …
Import Wiz Detects• File type• Delimiter• Skip Row
TH Technology
How Do I Load Data …
SQL Developer – Import Data …
INSERT or
GenerateINSERTScript)
TH Technology
How Do I Load Data …
SQL Developer – Import Data …
SelectColumns toImport
TH Technology
How Do I Load Data …
SQL Developer – Import Data …
Map Source to TableColumns
TH Technology
How Do I Load Data …
SQL Developer – Import Data …
Verify
TH Technology
How Do I Load Data …
SQL Developer – Import Data …
Finished!
XLS and XLSX* File Load in Very Few Clicks!*XLSX in SQL Developer 3.1
TH Technology
How Do I Load Data …
SQL Developer – Import Data …
What About XLSX?• SQL Dev 3.0 – No• SQL Developer 3.1 – Yes
XLS/XLSX File Load in Very Few Clicks!
TH Technology
How Do I Load Data …
SQL Developer – Import Data …
ProsOne Shot Data LoadDelimiter (, TAB, others)Flexible ColumnsFlexible MappingXLS FileCan be Faster than APEX Data Load
Wizard
ConsTable Must Be DefinedCannot Integrate into AppTedious to RepeatIDE SpecificOne Shot Data LoadNot for End Users
TH Technology
How Do I Load Data …
APEX Data Load/Unload Utility
APEX From the BeginningEasy Load of Spreadsheet or XML DataNew or Existing TableUpload File Or Copy/PasteDelimiter OptionColumn by Column Selection
TH Technology
How Do I Load Data …
APEX Data Load/Unload Utility
TH Technology
How Do I Load Data …
APEX Data Load/Unload Utility
1st Row Column Heading
TH Technology
How Do I Load Data …
APEX Data Load/Unload Utility
TH Technology
How Do I Load Data …
APEX Data Load/Unload Utility
Specify PK and Trigger
Finish
TH Technology
How Do I Load Data …
APEX Data Load Utility
ProsSimple: Point-Click-DoneFlexible FormatSpreadsheet ConvenienceCreates Table Copy/Paste Option
ConsMany StepsSingle TableNo XLS Upload (Must Copy/Paste or Save as CSV)One-TimeNot for End Users
TH Technology
How Do I Load Data …
What About the End User?
APEX Standard File Browse + Custom CodeData Load Wizard Wizard
APEX Manual Data Load Wizard End User-izedAPEX Web Sheet Data Grid
Copy/Paste, Data EntryAPEX Listener XLS Upload
File Browse + XLS2COLLECTION
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
Most Common (Still)File Browse ItemProcess (to Staging)Process To Destination Table(s)
MANY MANY OPTIONS …
All Require Custom Code
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
File Browse Item
WWV_FLOW_FILES – Original WayBLOB column – New Option w APEX 4+
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
WWV_FLOW_FILES – Traditional WayFile uploads to APEX Table WWV_FLOW_FILESMove Records to a Custom Table (i.e. Staging Table)Clean up!
Files accumulate in APEX_FILES Tablespace
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
BLOB Column – New w/ APEX 4File Stored in BLOB Column of Table Specified in Automatic Row Processing (DML) ProcessColumn Specified in Item SourceTable Must Have
BLOBFilenameMIME TypeCHARSET
Saves an Archiving Step
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
BLOB Column – New w/ APEX 4
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
WWV_FLOW_FILES
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
LOCAL_UPLOAD_FILES
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
Parse / Load ProcessRead Blob into local BLOB variableRead Records from BlobLoad Records into Destination Objects
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
Reusable Parse/ Load ProcessFile Browse Item
Load File to WWV_FLOW_FILEGeneric Parse File Package
Generic File Upload UtilityGeneric Store to APEX Collection
Process Code per Use/Table/Format
Custom Code, But Less of It
TH Technology
How Do I Load Data …
Generic Parse File Utility Package
Parse File to CollectionLoad Collection to Specified Table
Columns Must Match, or Adjust CodeReusable Code, Pages for Any Upload File/TableSaves One Part of Custom CodeCustom Processing from Stage Table Required
Custom Code, But Less of It
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
Parse File ProcessRead Blob into local BLOB variable
BEGIN SELECT blob_content INTO l_blob FROM wwv_flow_files WHERE name=p_file_name;EXCEPTION WHEN NO_DATA_FOUND THEN raise_application_error(-20000, 'File not found, id='||p_file_name);END;
Or BLOB column
from your BLOB table
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
Parse File ProcessRead Records from Blob Variable
TYPE varchar2_t IS TABLE OF VARCHAR2(32767) INDEX BY binary_integer; l_records VARCHAR2_T; … get_records(l_blob,l_records); … -- Initialize the APEX collection apex_collection.create_or_truncate_collection( p_collection_name);
TH Technology
How Do I Load Data …
PROCEDURE get_records(p_blob IN blob,p_records OUT varchar2_t) IS l_sep VARCHAR2(2) := chr(13)||chr(10); l_last INTEGER; l_current INTEGER;BEGIN IF (NVL(DBMS_LOB.INSTR(p_blob, utl_raw.cast_to_raw(l_sep),1,1),0)=0) THEN l_sep := chr(10); END IF; l_last := 1; LOOP l_current := dbms_lob.INSTR( p_blob, utl_raw.cast_to_raw(l_record_separator), l_last, 1 ); EXIT WHEN (nvl(l_current,0) = 0); p_records(p_records.count+1) := utl_raw.cast_to_varchar2( DBMS_LOB.SUBSTR(p_blob,l_current-l_last,l_last)); l_last := l_current+length(l_record_separator); END LOOP;END get_records;
TH Technology
How Do I Load Data …
LOOP l_current := dbms_lob.INSTR( p_blob, utl_raw.cast_to_raw(l_record_separator), l_last, 1 ); EXIT WHEN (nvl(l_current,0) = 0); p_records(p_records.count+1) := utl_raw.cast_to_varchar2( DBMS_LOB.SUBSTR(p_blob,l_current-l_last,l_last)); l_last := l_current+length(l_record_separator); END LOOP;
Loop to read records from the BLOB, line by line(separator to separator)
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
Parse File Process: Read from Blob,Load Collection
FOR i IN 2..l_records.count LOOP csv_to_array(l_records(i),l_record); l_seq_id := apex_collection.add_member(p_collection_name,'dummy'); FOR i IN 1..l_record.count LOOP apex_collection.update_member_attribute( p_collection_name=> p_collection_name, p_seq => l_seq_id, p_attr_number => i, p_attr_value => l_record(i)); END LOOP; END LOOP;
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
Parse File Process: Read Blob, Load Collection
Insert from Collection to Table … l_ddl := ‘INSERT INTO '||p_table_name||' '|| ‘SELECT '||v(p_columns_item)||' '|| ‘FROM apex_collections '|| ‘WHERE seq_id >= 1 and collection_name='''|| p_collection_name||'''';
EXECUTE IMMEDIATE l_ddl; …
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
Parse File Process: Read Blob, Load Collection, Insert,
Remove File -- Cleanup file! DELETE FROM wwv_flow_files WHERE name=p_file_name;
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
Process to Destination Table(s)
INSERT /* +append */ … SELECT … FROM staging
MERGE statement
Whatever is Required for Your Business
Custom Code
TH Technology
How Do I Load Data …
File Browse & Custom Code
ProsCustom CodeCustom Solution
ConsCustom CodeResourcesMaintenanceOne-of Solution
TH Technology
How Do I Load Data …
Data Load Wizard Wizard
The SQL Workshop Data Load Utility End-User-ized
Page Type
Creates a Shared Component
TH Technology
How Do I Load Data …
Data Load Wizard Wizard
Wizard to Create A Data Load WizardCreate Page Data Loading
TH Technology
How Do I Load Data …
Data Load Wizard Wizard
TH Technology
How Do I Load Data …
Data Load Wizard Wizard
Define Table Lookups
TH Technology
How Do I Load Data …
Data Load Wizard Wizard
Add TransformsTo Upper/Lower, R/L/Trim, Single Whitespace
TH Technology
How Do I Load Data …
Data Load Wizard Wizard
Page/Region/Breadcrumb Names
TH Technology
How Do I Load Data …
Data Load Wizard Wizard
Define Tabs (if any)
TH Technology
How Do I Load Data …
Data Load Wizard Wizard
Define Branches
TH Technology
How Do I Load Data …
Data Load Wizard Wizard
Confirmation
TH Technology
How Do I Load Data …
Data Load Wizard - Result
Working Upload for our USERS table
TH Technology
How Do I Load Data …
Data Load Wizard
End User Sees:Data Load ProgressionChance to Select Row to Include/IgnoreFailed Rows at the End
Chance to Back Out Load?Chance to Reload Clean?
TH Technology
How Do I Load Data …
Data Load Wizard
Upload OR Copy/PasteData Limit ==Specify DelimiterSpecify Enclosed BySpecify Currency, Group Separator, Decimal Characters
TH Technology
How Do I Load Data …
Data Load Wizard - Options
TH Technology
How Do I Load Data …
Data Load Wizard – Pro’s/Con’s
ProsFlexible FormDelimiter Option1st Row Heading OptCurrency – Group Separator – DecimalMapping Built InRepeatable for End User
Reusable for Developer
ConsToo Much for End User??No XLSCopy/Paste Size LimitUpload Size Limit ??Single Transform per Column
Home
TH Technology
How Do I Load Data …
Data Load Wizard Wizard
Is It Foolproof?No:
Ex: RPT file …. What does this means?
Too Many Columns?Too Few?
TH Technology
How Do I Load Data …
Data Load Wizard Wizard
Can I trick it into taking an XLS?No:
TH Technology
How Do I Load Data …
Data Load Wizard
Can I Make it “Better”?
Try:Default/Eliminate All OptionsEliminate Load/Unload Column MappingCatch/Warn of File Type, # Columns Errors
Load the Collection Behind the Schemes
TH Technology
How Do I Load Data …
Data Load Wizard – Behind the Scenes
Usual Expected ItemsFILE_BROWSE
TH Technology
How Do I Load Data …
Data Load Wizard – P 2
Parse Uploaded DataSingle Process of Type Parse Uploaded DataList of ParametersFires upon Next
TH Technology
How Do I Load Data …
TH Technology
How Do I Load Data …
Data Load Wizard – P 3
Process Uploaded DataSingle Process of Type Process Uploaded DataList of ParametersFires upon Next
TH Technology
How Do I Load Data …
Data Load Wizard – P 3
Session InfoCollectionsCharacter ColumnsNumeric ColumnsDate Columns
TH Technology
How Do I Load Data …
TH Technology
How Do I Load Data …
Data Load Wizard WizardPros
In-AppStraightforwardReusableNo Custom Code for Data Load into DB
ConsToo Many End User Steps Still Need Custom Processing Code for Staging to Tables
IF get P1 Finish by Loading Collection,
have a Two-Step File Upload for End
Users!
TH Technology
How Do I Load Data …
APEX Listener XLS Upload
File Browse ItemLoad File to WWV_FLOW_FILE
Parsing FileCustom Code per Use/Table/FormatGeneric File Upload Utility
Custom Process to Destination Tables
TH Technology
How Do I Load Data …
Parse File to CollectionLoads Collection to Specified Table
Table Exist or NotOptions
Reusable for Any Upload File/TableSaves One Part of Custom CodeCustom Processing from Stage Table Required
APEX Listener XLS Upload
TH Technology
How Do I Load Data …
APEX Listener XLS Upload
Demo
TH Technology
How Do I Load Data …
XLS Upload via APEX Listener
Must Deploy via APEX ListenerCurrent Release 1.1.3…
Allows Upload of XLS File into a Collection
Best Reference: (from RMOUG 2010)http://krisrice.blogspot.com/2010/02/another-apex-listener-ea-more-knobs-to.html
TH Technology
How Do I Load Data …
XLS Upload via APEX Listener
Install APEX ListenerDeployEdit apex_config.xmlAdd Configuration Choices:
<entry key="apex.excel2collection">true</entry><entry key="apex.excel2collection.onecollection">true</entry><entry key="apex.excel2collection.name">EXCEL_COLLECTION</entry><entry key="apex.excel2collection.useSheetName">true</entry>
TH Technology
How Do I Load Data …
XLS Upload via APEX Listener
Configuration Options:
Turn Feature On: <entry key="apex.excel2collection">true</entry>
One Collection per Worksheet:
<entry key= "apex.excel2collection.onecollection">true|false</entry>
TH Technology
How Do I Load Data …
XLS Upload via APEX Listener
Configuration Options:
Specify Name for All Collections: <entry key="apex.useSheetName">true|false</entry>
false == sequential sheet nametrue == uses sheet name
Specify Name of Collection for All Collections:
<entry key="apex.excel2collection.collection.name">MY_COLLECTION</entry>
TH Technology
How Do I Load Data …
XLS Upload via APEX Listener
File Browse ItemWWV_FLOW_FILES option
Page Item Button Name or REQUEST = XLS2COLLECTION
Report Region:SELECT * FROM apex_collectionsWHERE collection_name = ‘MY_COLLECTION’
TH Technology
How Do I Load Data …
XLS Upload via APEX Listener
TH Technology
How Do I Load Data …
XLS Upload via APEX Listener
TH Technology
How Do I Load Data …
XLS Upload via APEX Listener
“Files ending with .xls or .xlsx will be processed.”
BUT.xls processMacro-Enabled .xls process.XLSX do NOT Process as of APEX Listener 1.1.3.243.11.40.XLSX will Process as of v2
Coming Soon!
TH Technology
How Do I Load Data …
XLSX Upload via APEX Listener
TH Technology
How Do I Load Data …
XLSX Upload via APEX Listener
SEVERE: service exception:org.apache.poi.poifs.filesystem.OfficeXmlFileException: The supplied data appears to be in the Office 2007+ XML. You are calling the part of POI that deals with OLE2 Office Documents. You need to call a different part of POI to process this data (eg XSSF instead of HSSF)
TH Technology
How Do I Load Data …
XLS Upload via APEX Listener
Formulas Translate Literally
Ex: IF(E2="","",VLOOKUP(E2,$AE$4:$AF$218,2,FALSE))
TH Technology
How Do I Load Data …
XLS Upload via APEX Listener
STILL Need Custom Code to Process Data from Collection to Database Table(s)
TH Technology
How Do I Load Data …
APEX Listener XLS Upload
ProsXLS File – No Need to Save As CSVStraightforwardReusableLess Custom Code
ConsAPEX Listener DeploymentEmbedded FormulasNo XLSX - May Need a Save As Step
Will Have XLSX in V2
TH Technology
How Do I Load Data …
APEX Websheets
Data Grid Allows Copy/PasteSpreadsheet-Like EditsData Grid w/in Page
Custom Code Required to Transfer Data to Main Schema Tables
TH Technology
How Do I Load Data …
Editable Data in Websheet
TH Technology
How Do I Load Data …
APEX Websheets
Demo
TH Technology
How Do I Load Data …
Pull Data From Websheet into App
Look at APEX$_WS Tables:APEX$_WS_ROW – the dataAPEX$_WS_HISTORY – iff you care about who changed what when
Need to process from SQL Dev table to “real” tables
TH Technology
How Do I Load Data …
APEX WebSheets
ProsCopy/PasteSpreadsheet-Like Entry, EditsEnd User Can Build
ConsCustom Code to Create Websheet, Data Grid?Custom Code Required to Pull Data into Schema ObjectsCustom Code to Maintain Worksheet / Data GridMore Code Than Other Solutions ?
TH Technology
How Do I Load Data …
Our Solution
ProblemRPT FilesXLS filesDifferent Formats200+, MonthlyUsers Cannot “Save as CSV”Must Support EditsVarying VolumeLimited Network
SolutionTabular Forms - 4Macro to Save as RPTExternal Tables
Generated RPTLIS Output RPT Files
Upload File Process (to be retired)
Custom ProcessingRPT – LI Systems RPT – Spreadsheet Templates
TH Technology
How Do I Load Data …
APEX Data Load Options
TH Technology
How Do I Load Data …
Nice to Have … Future?
1-Step Data Load WizardCopy/Paste ORUpload XLSX
Easiest for Non-Networked, Low(er) Volume End Users
TH Technology
How Do I Load Data …
Some Last Thoughts …
Forms Create/Edit One Row
Tabular FormsEnter Many Rows
Java Solution.JSP to load XLS directlyPresume this means XLSX too!
TH Technology
How Do I Load Data …
How Do I Load Data ?
APEX-based SolutionWeigh Requirements
Source DataEnd User CapabilitiesResourcesTime
No One-Size-Fits-All
Apply Common Sense
TH Technology
How Do I Load Data …
TH Technology
How Do I Load Data …
Common Sense Test …
Two men play 5 games of checkers. Each wins the same # of games. There are no ties.
How can this be?
They are not playing each other.
How Do I Load Data …
TH Technology
References
Upload File into APEXoraexplorer.com – Ittichaihttp://oraexplorer.com/2007/11/apex-to-upload-text-file-and-write-into/More at avdeo.com In Oracle Milieuhttp://avdeo.com/2008/05/21/uploading-excel-sheet-using-oracle-application-express-apex/
How Do I Load Data …
TH Technology
References
Oracle Data Integratorhttp://www.oracle.com/technetwork/middleware/data-integrator/overview/index.html
Oracle Database UtilitiesSQL Loader – External Tableshttp://www.oracle.com/technetwork/database/enterprise-edition/index-093639.html
SQL Developerhttp://www.oracle.com/technetwork/developer-tools/sql-developer/overview/index.html
How Do I Load Data …
TH Technology
References
APEX Listener XLS Upload – Kris Rice Bloghttp://krisrice.blogspot.com/2010/02/another-apex-listener-ea-more-knobs-to.html
A Java-and-PL/SQL based XLS to CSV Optionhttp://www.dynamicpsp.com/!go?ln=nfaq&faq=FAQ9
How Do I Load Data?
Let Me Count The Ways
Questions ?
Please fill out the
evaluations!
Thank You
Please fill out the
evaluations!
How Do I Load Data?
Let Me Count The Ways