end to-end hadoop development using obiee, odi, oracle big data sql and oracle big data discovery

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

End-to-End Hadoop Development using OBIEE, ODI and Oracle Big Data Discovery Mark Rittman, CTO, Rittman Mead March 2015





About the Speaker

•Mark Rittman, Co-Founder of Rittman Mead •Oracle ACE Director, specialising in Oracle BI&DW •14 Years Experience with Oracle Technology •Regular columnist for Oracle Magazine •Author of two Oracle Press Oracle BI books •Oracle Business Intelligence Developers Guide •Oracle Exalytics Revealed •Writer for Rittman Mead Blog :http://www.rittmanmead.com/blog

•Email : [email protected] •Twitter : @markrittman

http://www.rittmanmead.com/blog

mailto:[email protected]





About Rittman Mead

•Oracle BI and DW Gold partner •Winner of five UKOUG Partner of the Year awards in 2013 - including BI •World leading specialist partner for technical excellence, solutions delivery and innovation in Oracle BI

•Approximately 80 consultants worldwide •All expert in Oracle BI and DW •Offices in US (Atlanta), Europe, Australia and India •Skills in broad range of supporting Oracle tools: ‣OBIEE, OBIA, ODIEE ‣Big Data, Hadoop, NoSQL & Big Data Discovery ‣Essbase, Oracle OLAP ‣GoldenGate ‣Endeca



End-to-End Oracle Big Data Example

•Rittman Mead want to understand drivers and audience for their website ‣What is our most popular content? Who are the most in-demand blog authors? ‣Who are the influencers? What do they read?

•Three data sources in scope:

RM Website Logs Twitter Stream Website Posts, Comments etc



Two Analysis Scenarios : Reporting, and Data Discovery

• Initial task will be to ingest data from webserver logs, Twitter firehose, site content + ref data •Land in Hadoop cluster, basic transform, format, store; then, analyse the data:

1 Combine with Oracle Big Data SQL for structured OBIEE dashboard analysis 2 Combine with site content, semantics, text enrichment

Catalog and explore using Oracle Big Data Discovery

What pages are people visiting? Who is referring to us on Twitter? What content has the most reach?

Why is some content more popular? Does sentiment affect viewership? What content is popular, where?



Step 1 Ingesting Basic Dataset into Hadoop



Data Loading into Hadoop

•Default load type is real-time, streaming loads ‣Batch / bulk loads only typically used to seed system

•Variety of sources including web log activity, event streams •Target is typically HDFS (Hive) or HBase •Data typically lands in “raw state” ‣Lots of files and events, need to be filtered/aggregated ‣Typically semi-structured (JSON, logs etc) ‣High volume, high velocity

-Which is why we use Hadoop rather thanRBDMS (speed vs. ACID trade-off)

‣Economics of Hadoop means its often possible toarchive all incoming data at detail level

Loading Stage

Real-Time Logs / Events

File / UnstructuredImports



Data Sources used for Project : Stage 1

Spark

Hive

HDFS

Spark

Hive

HDFS

Spark

Hive

HDFS

Cloudera CDH5.3 BDA Hadoop Cluster

Big Data SQL

Exadata Exalytics

Flume Flume MongoDB

DimAttributes

SQL for BDA Exec

Filtered &Projected Rows / Columns

OBIEE

TimesTen

12c In-Mem

Ingest Process Publish



Apache Flume : Distributed Transport for Log Activity

•Apache Flume is the standard way to transport log files from source through to target • Initial use-case was webserver log files, but can transport any file from A>B •Does not do data transformation, but can send to multiple targets / target types •Mechanisms and checks to ensure successful transport of entries

•Has a concept of “agents”, “sinks” and “channels” •Agents collect and forward log data •Sinks store it in final destination •Channels store log data en-route

•Simple configuration through INI files •Handled outside of ODI12c



Flume Source / Target Configuration

•Conf file for source system agent •TCP port, channel size+type, source type

•Conf file for target system agent •TCP port, channel size+type, sink type



Also - Apache Kafka : Reliable, Message-Based

•Developed by LinkedIn, designed to address Flume issues around reliability, throughput ‣(though many of those issues have been addressed since)

•Designed for persistent messages as the common use case ‣Website messages, events etc vs. log file entries

•Consumer (pull) rather than Producer (push) model •Supports multiple consumers per message queue •More complex to set up than Flume, and can useFlume as a consumer of messages ‣But gaining popularity, especially alongside Spark Streaming



Starting Flume Agents, Check Files Landing in HDFS Directory

•Start the Flume agents on source and target (BDA) servers •Check that incoming file data starts appearing in HDFS ‣Note - files will be continuously written-to as entries added to source log files ‣Channel size for source, target agentsdetermines max no. of events buffered ‣If buffer exceeded, new events droppeduntil buffer < channel size



Adding Social Media Datasources to the Hadoop Dataset

•The log activity from the Rittman Mead website tells us what happened, but not “why” •Common customer requirement now is to get a “360 degree view” of their activity ‣Understand what’s being said about them ‣External drivers for interest, activity ‣Understand more about customer intent, opinions

•One example is to add details of social media mentions,likes, tweets and retweets etc to the transactional dataset ‣Correlate twitter activity with sales increases, drops ‣Measure impact of social media strategy ‣Gather and include textual, sentiment, contextualdata from surveys, media etc



Example : Supplement Webserver Log Activity with Twitter Data

•Datasift provide access to the Twitter “firehose” along with Facebook data, Tumblr etc •Developer-friendly APIs and ability to define search terms, keywords etc •Pull (historical data) or Push (real-time) delivery using many formats / end-points ‣Most commonly-used consumption format is JSON, loaded into Redis, MongoDB etc



What is MongoDB?

•Open-source document-store NoSQL database •Flexible data model, each document (record) can have its own JSON schema

•Highly-scalable across multiple nodes (shards) •MongoDB databases made up of collections of documents ‣Add new attributes to a document just by using it ‣Single table (collection) design, no joins etc ‣Very useful for holding JSON output from web apps

- for example, twitter data from Datasift



Where Are We Now : Data Ingestion Phase

•Log data streamed into HDFS via Flume collector / sink •Twitter data pushed into MongoDB collection •Data is in raw form •Now it needs to be refined, combined,transformed and processed intouseful data…



Step 2 Processing, Enhancing & Transforming Data



Core Apache Hadoop Tools

•Apache Hadoop, including MapReduce and HDFS ‣Scaleable, fault-tolerant file storage for Hadoop ‣Parallel programming framework for Hadoop

•Apache Hive ‣SQL abstraction layer over HDFS ‣Perform set-based ETL within Hadoop

•Apache Pig, Spark ‣Dataflow-type languages over HDFS, Hive etc ‣Extensible through UDFs, streaming etc

•Apache Flume, Apache Sqoop, Apache Kafka ‣Real-time and batch loading into HDFS ‣Modular, fault-tolerant, wide source/target coverage



Other Tools Typically Used…

•Python, Scala, Java and other programming languages ‣For more complex and procedural transformations

•Shell scripts, sed, awk, regexes etc •R and R-on-Hadoop ‣Typically at the “discovery” phase

•And down the line - ETL tools to automate the process ‣ODI, Pentaho Data Integrator etc



ODI on Hadoop - Enterprise Data Integration for Big Data

•ODI provides an excellent framework for loading and transforming Hadoop data ‣ELT approach pushes transformations down to Hadoop - leveraging power of cluster

•Hive, HBase, Sqoop and OLH/ODCH KMs provide native Hadoop loading / transformation ‣Whilst still preserving RDBMS push-down ‣Extensible to cover Pig, Spark etc

•Process orchestration •Data quality / error handling •Metadata and model-driven



Oracle Big Data Connectors

•Oracle-licensed utilities to connect Hadoop to Oracle RBDMS ‣Bulk-extract data from Hadoop to Oracle, or expose HDFS / Hive data as external tables ‣Run R analysis and processing on Hadoop ‣Leverage Hadoop compute resources to offload ETL and other work from Oracle RBDMS ‣Enable Oracle SQL to access and load Hadoop data



Hive SerDes & Storage Handlers

•Plug-in technologies that extend Hive to handle new data formats and semi-structured sources •Typically distributed as JAR files, hosted on sites such as GitHub •Can be used to parse log files, access data in NoSQL databases, Amazon S3 etc

CREATE EXTERNAL TABLE apachelog ( host STRING, identity STRING, user STRING, time STRING, request STRING, status STRING, size STRING, referer STRING, agent STRING) ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe' WITH SERDEPROPERTIES ( "input.regex" = "([^ ]*) ([^ ]*) ([^ ]*) (-|\\[[^\\]]*\\]) ([^ \"]*|\"[^\"]*\") (-|[0-9]*) (-|[0-9]*)(?: ([^ \"]*|\"[^\"]*\") ([^ \"]*|\"[^\"]*\"))?", "output.format.string" = "%1$s %2$s %3$s %4$s %5$s %6$s %7$s %8$s %9$s" ) STORED AS TEXTFILE LOCATION '/user/root/logs';

CREATE TABLE tweet_data( interactionId string, username string, content string, author_followers int) ROW FORMAT SERDE 'com.mongodb.hadoop.hive.BSONSerDe' STORED BY 'com.mongodb.hadoop.hive.MongoStorageHandler' WITH SERDEPROPERTIES ( 'mongo.columns.mapping'='{"interactionId":"interactionId", "username":"interaction.interaction.author.username", "content":\"interaction.interaction.content", "author_followers_count":"interaction.twitter.user.followers_count"}' ) TBLPROPERTIES ( 'mongo.uri'='mongodb://cdh51-node1:27017/datasiftmongodb.rm_tweets' )



Configuring ODI12c Topology and Models

•HDFS data servers (source) defined using generic File technology •Workaround to support IKM Hive Control Append •Leave JDBC driver blank, put HDFS URL in JDBC URL field



Defining Topology and Model for Hive Sources

•Hive supported “out-of-the-box” with ODI12c (but requires ODIAAH license for KMs) •Most recent Hadoop distributions use HiveServer2 rather than HiveServer

•Need to ensure JDBC drivers support Hive version •Use correct JDBC URL format (jdbc:hive2//…)



HDFS & Hive Model and Datastore Definitions

•HDFS files for incoming log data, and any other input data •Hive tables for ETL targets and downstream processing •Use RKM Hive to reverse-engineer column definition from Hive



Hive and MongoDB

•MongoDB Hadoop connector provides a storage handler for Hive tables •Rather than store its data in HDFS, the Hive table uses MongoDB for storage instead •Define in SerDe properties the Collection elements you want to access, using dot notation •https://github.com/mongodb/mongo-hadoop

CREATE TABLE tweet_data( interactionId string, username string, content string, author_followers int) ROW FORMAT SERDE 'com.mongodb.hadoop.hive.BSONSerDe' STORED BY 'com.mongodb.hadoop.hive.MongoStorageHandler' WITH SERDEPROPERTIES ( 'mongo.columns.mapping'='{"interactionId":"interactionId", "username":"interaction.interaction.author.username", "content":\"interaction.interaction.content", "author_followers_count":"interaction.twitter.user.followers_count"}' ) TBLPROPERTIES ( 'mongo.uri'='mongodb://cdh51-node1:27017/datasiftmongodb.rm_tweets' )

https://github.com/mongodb/mongo-hadoop



Adding MongoDB Datasets into the ODI Repository

•Define Hive table outside of ODI, using MongoDB storage handler •Select the document elements of interest, project into Hive columns •Add Hive source to Topology if needed, then use Hive RKM to bring in column metadata



Load Incoming Log Files into Hive Table

•First step in process is to load the incoming log files into a Hive table ‣Also need to parse the log entries to extract request, date, IP address etc columns ‣Hive table can then easily be used in downstream transformations

•Use IKM File to Hive (LOAD DATA) KM ‣Source can be local files or HDFS ‣Either load file into Hive HDFS area,or leave as external Hive table ‣Ability to use SerDe to parse file data

1



Specifying a SerDe to Parse Incoming Hive Data

•SerDe (Serializer-Deserializer) interfaces give Hive the ability to process new file formats •Distributed as JAR file, gives Hive ability to parse semi-structured formats •We can use the RegEx SerDe to parse the Apache CombinedLogFormat file into columns •Enabled through OVERRIDE_ROW_FORMAT IKM File to Hive (LOAD DATA) KM option



Join to Additional Hive Tables, Transform using HiveQL

• IKM Hive to Hive Control Append can be used to perform Hive table joins, filtering, agg. etc. • INSERT only, no DELETE, UPDATE etc •Not all ODI12c mapping operators supported, but basic functionality works OK •Use this KM to join to other Hive tables,adding more details on post, title etc

•Perform DISTINCT on join output, loadinto summary Hive table

2



Joining Hive Tables

•Only equi-joins supported •Must use ANSI syntax •More complex joins may not producevalid HiveQL (subqueries etc)



Filtering, Aggregating and Transforming Within Hive

•Aggregate (GROUP BY), DISTINCT, FILTER, EXPRESSION, JOIN, SORT etc mapping operators can be added to mapping to manipulate data

•Generates HiveQL functions, clauses etc



Loading Hive Tables - Streaming CDC using GoldenGate

•Hive tables in Hadoop cluster can be loaded from file and other sources •For streaming updates from RDBMS sources, Oracle GoldenGate for Big Data can be used ‣RDBMS to Hive ‣RDBMS to HDFS ‣RDBMS to Flume

•Can combine with Flume,Kafka etc for real-time Hive/HDFS data sync



Bring in Reference Data from Oracle Database

• In this third step, additional reference data from Oracle Database needs to be added • In theory, should be able to add Oracle-sourced datastores to mapping and join as usual •But … Oracle / JDBC-generic LKMs don’t get work with Hive

3



Options for Importing Oracle / RDBMS Data into Hadoop

•Could export RBDMS data to file, and load using IKM File to Hive •Oracle Big Data Connectors only export to Oracle, not import to Hadoop •Best option is to use Apache Sqoop, and new IKM SQL to Hive-HBase-File knowledge module •Hadoop-native, automatically runs in parallel •Uses native JDBC drivers, or OraOop (for example) •Bi-directional in-and-out of Hadoop to RDBMS •Run from OS command-line



Loading RDBMS Data into Hive using Sqoop

•First step is to stage Oracle data into equivalent Hive table •Use special LKM SQL Multi-Connect Global load knowledge module for Oracle source ‣Passes responsibility for load (extract) to following IKM

•Then use IKM SQL to Hive-HBase-File (Sqoop) to load the Hive table



Join Oracle-Sourced Hive Table to Existing Hive Table

•Oracle-sourced reference data in Hive can then be joined to existing Hive table as normal •Filters, aggregation operators etc can be added to mapping if required •Use IKM Hive Control Append as integration KM



Adding the Twitter Data from MongoDB

•Previous steps exposed the Twitter data, in MongoDB, through a Hive table •RM_RELATED_TWEETS Hive table now included in ODI Topology + Model

hive> describe rm_related_tweets; OK id string from deserializer interactionid string from deserializer username string from deserializer author_name string from deserializer created_at string from deserializer content string from deserializer twitter_link string from deserializer language string from deserializer sentiment int from deserializer author_tweet_count int from deserializer author_followers_co int from deserializer author_profile_img_u string from deserializer author_timezone string from deserializer normalized_url string from deserializer mentions string from deserializer hashtags string from deserializer Time taken: 0.109 seconds, Fetched: 16 row(s)



Filter Log Entries to Only Leave Blog Post Views

•We’re only interested in Twitter activity around blog posts •Create an additional Hive table to contain just blog post views, to then join to tweets



Filter Tweets Down to Just Those Linking to RM Website

•Filter the list of tweets down to just those that link to RM blog



Join Twitter Data to Page View Data

•Create summary table showing twitter activity per page on the blog



Create ODI Package for Processing Steps, and Execute

•Create ODI Package or Load Plan to run steps in sequence ‣With load plan, can also add exceptions and recoverability

•Execute package to load data into final Hive tables



Summary : Data Processing Phase

•We’ve now processed the incoming data, filtering it and transforming to required state •Joined (“mashed-up”) datasets from website activity, and social media mentions • Ingestion and the load/processing stages are now complete •Now we want to make the Hadoop output available to a wider, non-technical audience…



Step 3 Publishing & Analyzing Data



New in OBIEE 11.1.1.7 : Hadoop Connectivity through Hive

•MapReduce jobs are typically written in Java, but Hive can make this simpler •Hive is a query environment over Hadoop/MapReduce to support SQL-like queries •Hive server accepts HiveQL queries via HiveODBC or HiveJDBC, automaticallycreates MapReduce jobs against data previously loaded into the Hive HDFS tables

•Approach used by ODI and OBIEE to gain access to Hadoop data •Allows Hadoop data to be accessed just like any other data source



Importing Hadoop/Hive Metadata into RPD

•HiveODBC driver has to be installed into Windows environment, so that BI Administration tool can connect to Hive and return table metadata

• Import as ODBC datasource, change physical DB type to Apache Hadoop afterwards •Note that OBIEE queries cannot span >1 Hive schema (no table prefixes)

1

2

3



Set up ODBC Connection at the OBIEE Server

•OBIEE 11.1.1.7+ ships with HiveODBC drivers, need to use 7.x versions though (only Linux supported)

•Configure the ODBC connection in odbc.ini, name needs to match RPD ODBC name •BI Server should then be able to connect to the Hive server, and Hadoop/MapReduce

[ODBC Data Sources] AnalyticsWeb=Oracle BI Server Cluster=Oracle BI Server SSL_Sample=Oracle BI Server bigdatalite=Oracle 7.1 Apache Hive Wire Protocol

[bigdatalite] Driver=/u01/app/Middleware/Oracle_BI1/common/ODBC/ Merant/7.0.1/lib/ARhive27.so Description=Oracle 7.1 Apache Hive Wire ProtocolArraySize=16384 Database=default DefaultLongDataBuffLen=1024 EnableLongDataBuffLen=1024 EnableDescribeParam=0 Hostname=bigdatalite LoginTimeout=30 MaxVarcharSize=2000 PortNumber=10000 RemoveColumnQualifiers=0 StringDescribeType=12 TransactionMode=0 UseCurrentSchema=0



OBIEE 11.1.1.7 / HiveServer2 ODBC Driver Issue

•Most customers using BDAs are using CDH4 or CDH5 - which uses HiveServer2 •OBIEE 11.1.1.7 only ships/supports HiveServer1 ODBC drivers •But … OBIEE 11.1.1.7 on Windows can use the Cloudera HiveServer2 ODBC drivers ‣which isn’t supported by Oracle ‣but works!



Using Big Data SQL with OBIEE11g

•Preferred Hadoop access path for customers with Oracle Big Data Appliance is Big Data SQL •Oracle SQL Access to both relational, and Hive/NoSQL data sources •Exadata-type SmartScan against Hadoop datasets

•Response-time equivalent to Impala or Hive on Tez •No issues around HiveQL limitations • Insulates end-users around differencesbetween Oracle and Hive datasets



Using Oracle Big Data SQL to Federate Hive with Oracle

•Big Data SQL can also be used to add, on-the-fly, additional Oracle reference data to Hive ‣For example : geoip country lookuprequiring non-equi join ‣Additional dimensions not present in Hive ‣Calculated measures, derived attributesusing Oracle SQL

Hive Weblog Activity tableOracle Dimension lookup tables

Combined output in report form



Big Data SQL Example : Add Geolookup Data to Hive

•Hive ACCESS_PER_POST_CATEGORIES table contains Host IP addresses • Ideally would like to translate into country, city etc ‣See readership by country ‣Match topics of interest to country, city



How GeoIP Geocoding Works

•Uses free Geocoding API and database from Maxmind •Convert IP address to an integer •Find which integer range our IP address sits within •But Hive can’t use BETWEEN in a join…

•Solution : Expose ACCESS_PER_POST_CATEGORIES using Big Data SQL, then join



Create ORACLE_HIVE version of Hive Table, Join to GeoIP

access_per_post_categories.ip_integerBETWEEN geoip_country.start_ip_int

AND geoip_country.end_ip_int



Import Oracle Tables, Create RPD joining Tables Together

•No need to use Hive ODBC drivers - Oracle OCI connection instead •No issue around HiveServer1 vs HiveServer2; also Big Data SQL handles authenticationwith Hadoop cluster in background, Kerberos etc

•Transparent to OBIEE - all appear as Oracle tables •Join across schemas if required



Create Physical Data Model from Imported Table Metadata

•Join ORACLE_HIVE external table containing log data, to reference tables from Oracle DB



Create Business Model and Presentation Layers

•Map incoming physical tables into a star schema •Add aggregation method for fact measures •Add logical keys for logical dimension tables •Aggregate (Count) on page ID fact column •Remove columns from fact table that aren’t measures



Create Initial Analyses Against Combined Dataset

•Create analyses usingfull SQL features

•Access to Oracle RDBMSAdvanced Analytics functionsthrough EVALUATE,EVALUATE_AGGR etc

•Big Data SQL SmartScan featureprovides fast, ad-hoc accessto Hive data, avoiding MapReduce



Create Views over Fact Table To Support Time-Series Analysis

•Time field comes in from Hive as string (date and time, plus timezone) •Create view over base ORACLE_HIVE external table to break-out date, time etc • Import time dimension table from Oracle to support date/time analysis, forward forecasting



Now Enables Time-Series Reporting Incl. Country Lookups



Use Exalytics In-Memory Aggregate Cache if Required

• If further query acceleration is required, Exalytics In-Memory Cache can be used •Enabled through Summary Advisor, caches commonly-used aggregates in in-memory cache •Options for TimesTen or Oracle Database 12c In-Memory Option •Returns aggregated data “at the speed of thought”



Step 4 Enabling for Data Discovery



Enable Incoming Site Activity Data for Data Discovery

•Another use-case for Hadoop data is “data discovery” ‣Load data into the data reservoir ‣Catalog and understand separate datasets ‣Enrich data using graphical tools ‣Join separate datasets together ‣Present textual data alongside measuresand key attributes ‣Explore and analyse using faceted search

2 Combine with site content, semantics, text enrichment Catalog and explore using Oracle Big Data Discovery

Why is some content more popular? Does sentiment affect viewership? What content is popular, where?



Oracle Big Data Discovery

• “The Visual Face of Hadoop” - cataloging, analysis and discovery for the data reservoir •Runs on Cloudera CDH5.3+ (Hortonworks support coming soon) •Combines Endeca Server + Studio technology with Hadoop-native (Spark) transformations



Data Sources used for Project : Stage 2

Spark

Hive

HDFS

Spark

Hive

HDFS

Spark

Hive

HDFS

Cloudera CDH5.3 BDA Hadoop Cluster

Hive Client

HDFS Client

BDD DGraphGateway

Hive Client

BDD StudioWeb UI

BDD Node

BDD Data Processing

BDD Data Processing

BDD Data Processing

Ingest semi-process logs

(1m rows)

Ingest processedTwitter activity

Write-backTransformations

to full datasets

UploadSite page and

comment contents

Persist uploaded DGraphcontent in Hive / HDFS

Data Discovery using Studio

web-based app



Oracle Big Data Discovery Architecture

•Adds additional nodes into the CDH5.3 cluster, for running DGraph and Studio •DGraph engine based on Endeca Server technology, can also be clustered

•Hive (HCatalog) used for reading table metadata,mapping back to underlying HDFS files

•Apache Spark then used to upload (ingest)data into DGraph, typically 1m row sample

•Data then held for online analysis in DGraph •Option to write-back transformations tounderlying Hive/HDFS files using Apache Spark



Ingesting & Sampling Datasets for the DGraph Engine

•Datasets in Hive have to be ingested into DGraph engine before analysis, transformation •Can either define an automatic Hive table detector process, or manually upload •Typically ingests 1m row random sample ‣1m row sample provides > 99% confidence that answer is within 2% of value shownno matter how big the full dataset (1m, 1b, 1q+) ‣Makes interactivity cheap - representative dataset

Amount'of'data'queried

The'100%'premium

CostAccuracy



Ingesting Site Activity and Tweet Data into DGraph

•Two output datasets from ODI process have to be ingested into DGraph engine •Upload triggered by manual call to BDD Data Processing CLI ‣Runs Oozie job in the background to profile,enrich and then ingest data into DGraph

[oracle@bddnode1 ~]$ cd /home/oracle/Middleware/BDD1.0/dataprocessing/edp_cli [oracle@bddnode1 edp_cli]$ ./data_processing_CLI -t pageviews [oracle@bddnode1 edp_cli]$ ./data_processing_CLI -t rm_linked_tweets

Hive

Apache Spark

pageviews X rows

pageviews >1m rows

Profiling pageviews >1m rows

Enrichment pageviews >1m rows

BDD

pageviews >1m rows

{ "@class" : "com.oracle.endeca.pdi.client.config.workflow. ProvisionDataSetFromHiveConfig", "hiveTableName" : "rm_linked_tweets", "hiveDatabaseName" : "default", "newCollectionName" : “edp_cli_edp_a5dbdb38-b065…”, "runEnrichment" : true, "maxRecordsForNewDataSet" : 1000000, "languageOverride" : "unknown" }

1

23



View Ingested Datasets, Create New Project

• Ingested datasets are now visible in Big Data Discovery Studio •Create new project from first dataset, then add second



Automatic Enrichment of Ingested Datasets

• Ingestion process has automatically geo-coded host IP addresses •Other automatic enrichments run after initial discovery step, based on datatypes, content



Datatype Conversion Example : String to Date / Time

•Datatypes can be converted into other datatypes, with data transformed if required •Example : convert Apache Combined Format Log date/time to Java date/time



Other Transformation Types, Filtering and Grouping

•Group and bin attribute values; filter on attribute values, etc •Use Transformation Editor for custom transformations (Groovy, incl. enrichment functions)



Commit Transforms to DGraph, or Create New Hive Table

•Transformation changes have to be committed to DGraph sample of dataset ‣Project transformations kept separate from other project copies of dataset

•Transformations can also be applied to full dataset, using Apache Spark ‣Creates new Hive table of complete dataset



Upload Additional Datasets

•Users can upload their own datasets into BDD, from MS Excel or CSV file •Uploaded data is first loaded into Hive table, then sampled/ingested as normal

12

3



Join Datasets On Common Attributes

•Creates left-outer join between primary, and secondary dataset ‣Add metrics (sum of page visits, for example) to activity dataset (e.g. tweets) ‣Join more attributes to main dataset



Create Discovery Pages for Dataset Analysis

•Select from palette of visualisation components •Select measures, attributes for display



Select From Multiple Visualisation Types



•Search on attribute values, text in attributes across all datasets •Extracted keywords, free text field search

Faceted Search Across Project



Key BDD Studio Differentiator : Faceted Search Across Hadoop

•BDD Studio dashboards support faceted search across all attributes, refinements •Auto-filter dashboard contents on selected attribute values •Fast analysis and summarisation through Endeca Server technology

“Mark Rittman” selected from Post Authors Results filtered on

selected refinement

1 2



Key BDD Studio Differentiator : Faceted Search Across Hadoop

•BDD Studio dashboards support faceted search across all attributes, refinements •Auto-filter dashboard contents on selected attribute values - for data discovery •Fast analysis and summarisation through Endeca Server technology

Further refinement on“OBIEE” in post keywords

3Results now filteredon two refinements

4



Summary

•Oracle Big Data, together with OBIEE, ODI and Oracle Big Data Discovery •Complete end-to-end solution with engineered hardware, and Hadoop-native tooling

1 Combine with Oracle Big Data SQL for structured OBIEE dashboard analysis 2 Combine with site content, semantics, text enrichment

Catalog and explore using Oracle Big Data Discovery





Thank You for Attending!

•Thank you for attending this presentation, and more information can be found at http://www.rittmanmead.com

•Contact us at [email protected] or [email protected] •Look out for our book, “Oracle Business Intelligence Developers Guide” out now! •Follow-us on Twitter (@rittmanmead) or Facebook (facebook.com/rittmanmead)

http://www.rittmanmead.com





End-to-End Hadoop Development using OBIEE, ODI and Oracle Big Data Discovery Mark Rittman, CTO, Rittman Mead March 2015

end to-end hadoop development using obiee, odi, oracle big data sql and oracle big data discovery

Data & Analytics