how customers are optimizing their edw for fast, secure, and effective insights

1 ©HortonworksInc.2011–2016.AllRightsReserved1 ©HortonworksInc.2011–2017.AllRightsReserved

2 ©HortonworksInc.2011–2016.AllRightsReserved

How Customers are Optimizing their EDW for Fast, Secure, Cost Effective Actionable Insights

Wei Wang Sr. Director, Product Marketing

Paige RobertsBig Data Product Manager

Speakers:


ActionableInsights

HumanConnections

BalancedSupplyChains

NewProducts&Services

OperationalEfficiencies

DataIsTheNewCurrency


TheOldWayOperationalEfficiencies

Hierarchical

Historical

Highly Aggregated

One-size-fits-all

TheNewWayEngagementandInnovation

Multi-structured

Predictive

Agile &Real-time

Context Sensitive

TransformingTheEnterpriseDataWarehouse(EDW)


HortonworksConnectedDataPlatformsandSolutions

HortonworksConnection

HortonworksSolutions

EnterpriseDataWarehouseOptimization

CyberSecurityandThreatManagement

InternetofThingsandStreamingAnalytics

HortonworksConnectionSubscriptionSupportSmartSense

PremierSupportEducationalServicesProfessionalServices

CommunityConnection

CloudHortonworks DataCloudAWS HDInsight

DataCenterHortonworks DataSuite

HDFHDP


TheSolutionforEnterpriseDataWarehouseOptimization

DramaticCostReductionsThroughthePowerofOpenSourceandApacheHadoopReducecostofyourEDWImplementationbyaugmentationorreplacementofETLprocesses

DeployBusinessIntelligenceonHadoopEnablementBusinessusers(viaadesktopsolution)seeaggregatedata,drillup,drilldownthroughbusinessdatamodels.

EnrichwithUnstructuredDataSupportDeliversupportforphotos,video,textfilestoenrichyouranalytics


LegacyEDWvs.EDWOptimizationSolutionwithConnectedDataPlatforms

910

920

930

940

950

960

970

980

990

1,000

Difficulty transforming data

into a suitable form for analysis.

Difficulty integrating

Big Data with existing

infrastructure.

Difficulty merging multiple, disparate

data sources.

Lack of skilled Big Data

practitioners.

Difficulty maintaining application

performance for large volume of

concurent users.

ImplementingtheModernDataArchitectureIsn’tEasy

8

Source:Wikibon BigDataAnalyticsAdoption Survey,2014-2015

Syncsort Confidential and Proprietary - do not copy or distribute

Oftheselectedtechnology-relatedbarrierstorealizingthefullvalueofyourBigDataAnalytics,pleaserankthetop3.

910

920

930

940

950

960

970

980

990

1,000





infrastructure.


data sources.


practitioners.



concurent users.


9

AdditionalChallenges• Long,drawn-outdevelopmentcycles


9Syncsort Confidential and Proprietary - do not copy or distribute


910

920

930

940

950

960

970

980

990

1,000





infrastructure.


data sources.


practitioners.



concurent users.


10


• Rapidlychangingtechnologypresentsamovingtarget,forcingconstantre-design




910

920

930

940

950

960

970

980

990

1,000





infrastructure.


data sources.


practitioners.



concurent users.


11



• Difficultyaccessingandintegrating legacyandnewdatasourcesincludingthemainframe




910

920

930

940

950

960

970

980

990

1,000





infrastructure.


data sources.


practitioners.



concurent users.


12




• Functionalitygapsinsecurity,governance,andcompliance




910

920

930

940

950

960

970

980

990

1,000





infrastructure.


data sources.


practitioners.



concurent users.


13




• Functionalitygapsinsecurity,governance,andcompliance



Bigdataintegrationiscomplicated!



HortonworksEDWOptimizationSolutionComponents

HadoopScalableStorageandCompute

HiveLLAPHighPerformanceSQLDataMart

Fast,scalableSQLanalyticsIntelligentin-memorycaching



SyncsortHigh-PerformanceDataMovement



SourceDataSystems


Highperformancedataimportfrom allmajorEDWplatforms



SyncsortHigh-PerformanceDataMovement



AtScaleIntelligencePlatformOLAPCubesforHigherPerformance

SourceDataSystems


DefineOLAPcubesfor10xfasterqueriesUnifiedsemanticlayerforallBItools

Highperformancedataimportfrom allmajorEDWplatforms

Pre-aggregateddata

...Or,full-fidelitydata


EDWOptimization:FastBIonHadoop

Ã TheProblem:– ProprietaryEDWsystemswereadoptedfor

FastBIanddeepslice-and-diceanalytics,butEDWpricesareunsustainablyhigh.

Ã TheSolution:– InteractiveSQLisarealityonHadooptoday.– AtScaleIntelligencePlatformaddsOLAP

capabilitiesfordeepdrilldownatscale.

Ã TheResult:– Queryterabytesofdatainseconds.– ConnectyourfavoriteBItoolslikeTableauand

ExcelthroughSQLandMDXinterfaces.– TheEDWOptimizationSolutionistailor-made

todeliverFastBIonHadoop.

ETL/ELT

DATAMART

DATALANDING&

DEEPARCHIVE

CUBEMART

ENDUSER

APPLICATIONS

APPLICATIONS

APPLICATIONS

ENDUSERSANDAPPS

EDWOPTIMIZATIONSOLUTION


EDWOptimization:ETLOffload

Ã TheProblem:– EDWsconsumebetween50%and90%of

CPUjustonETL/ELTtasks.– Thesejobsinterferewithmorebusiness-

criticaltaskslikeBIandadvancedanalytics.

Ã TheSolution:– HiveandHDPdeliverETLthatscalesto

petabytes.– SyncsortDMX-hforsimpledrag-and-dropETL

workflows.– Economicalscale-outprocessingon

commodityservers.

Ã TheResult:– BetterSLAsformission-criticalanalytics.– LimitEDWexpansionorretireoldsystems.

ETL/ELT

DATAMART

DATALANDING&

DEEPARCHIVE

CUBEMART

ENDUSER

APPLICATIONS

APPLICATIONS

APPLICATIONS

ENDUSERSANDAPPS



EDWOptimization:ActiveArchive

Ã TheProblem:– Increasingdatavolumesandcostpressure

forcedatatobearchivedtotape.– Archiveddatanotavailableforanalytics,or

mustberetrievedatgreatexpense.

Ã TheSolution:– AdoptingHadoopdeliverscostperterabyte

onparwithtapebackupsolutions.– DatainHadoopcanbeanalyzedbyallmajor

BItools,allowinganalyticsonarchivedata.

Ã TheResult:– Dataalwaysavailableforanalytics.– Storeyearsofdataratherthanmonths.

ETL/ELT

DATAMART

DATALANDING&

DEEPARCHIVE

CUBEMART

ENDUSER

APPLICATIONS

APPLICATIONS

APPLICATIONS

ENDUSERSANDAPPS


PoweringtheConnectedDataPlatformWithEDWOptimization

Paige RobertsBig Data Product Manager@RobertsPaige

GoalsoftheModernDataArchitecture

•Centralizeallyourdata

•Turnrawdataintoinsights

•Maintaingovernance,complianceandsecuritystandards

•EliminatecomplexitieswithinIT

21

SyncsortStrategicFocusonBigData&Hadoop

Lightfootprint

Self-tuningengine

Singleinstall.No3rd partydependencies

World-classdataprocessing,mainframeexpertise

JIRA:MAPREDUCE-2454MAPREDUCE-4807MAPREDUCE-4049MAPREDUCE-5455HIVE-8347SQOOP-1272PARQUET-134Spark-packagesand more!

|


OngoingContributionstotheOpenSourceCommunity1

LeverageSyncsortTechnologyInnovations&MainframeHeritage2

StrongPartnershipswithStrategicBigData&HadoopPlayers

1

3

22

Ã Connecttovirtualanydatasource,includingmainframeandMPPdatabases.

Ã MovedataintoandoutofHadoopupto6xfasterwithouttheneedformanualscripts.

Ã DevelopETLprocesseswithoutwritingcode.

Ã AutomaticallyoptimizeHadoopperformanceandscalabilityforETLoperations.

Ã FullycertifiedandintegratedwithHortonworksDataPlatformandHDF

Ã Secure– Kerberos,Ranger,Sentry

SyncsortBenefits

Syncsort: High Performance Import from Existing Sources

BringALLEnterpriseDataSecurelytotheDataLake

24

• Collectvirtuallyanydatafrommainframetorelational,cloudandNoSQLsources

• Access,re-formatandloaddatadirectlyintoHive&Hadoopfileformats.Nostagingrequired!

• Batch&streamingsources

• Pullhundredsoftablesatonceintoyourdatahub,wholeDBschemasinoneinvocation


•LoadmoredataintoHadoopinlesstime

GetYourDatabasedataintoHadoop,AtthePressofaButton

25

• Pullmultipledatasourcesandfunnelintoyourdatalake• ExtractandmapwholeDBschemasinoneinvocation• Extractfrommultipledatasources:DB2/z,Netezza,Oracle,Teradata,…

• One-stepdatamovement,auto-generatingjobs,auto-generatingHivetargettables,andupdateHivestatistics

• Processmultiplefunnelsinparallelonyouredgenodeor fromdatanodes‒ LeveragesDMX-hhighspeeddataengineviaDTL‒ GeneratedapplicationscanbeimportedintoGUI

• In-flighttransformations‒ Filtering,funneldependencyordering,mixedsource/target,datatypefiltering,tableexclusion/inclusion


DMXDataFunnel™

IntelligentExecutionLayer

DesignOnce,DeployAnywhere

26

IntelligentExecution- InsulateyourpeoplefromunderlyingcomplexitiesofHadoop.

Oneinterfacetodesignjobstorunon:SingleNode,ClusterMapReduce,Spark,FuturePlatformsWindows,Unix,LinuxOn-Premise,CloudBatch,Streaming

• UseexistingETLskills.• Noworriesaboutmappers,reducers,bigside,smallside,andsoon.• Automaticoptimizationforbestperformance,loadbalancing,etc.• Nochangesortuningrequired,evenifyouchangeexecutionframeworks• Future-proofjobdesignsforemergingcomputeframeworks,e.g.Spark

Insurance:EasyAccesstoALLDataforBetterAnalytics

27

• Challenge: Neededhard-to-accessoperationaldataforadvancedanalytics

• Solution:• Quicklyload~1000databasetablesintoHDPwiththeclickofa

button• Access&integratecomplexMainframeVSAMfiles,datafrom

DB2/z,Oracle&SQLServer• Trackchanges&keepdatauptodate

• Benefits:• Insight: Betterandfasteranalytics• Agility: Reclaimdevelopmenttime;singletooltoingest,detectchangesandpopulatethedatalake• Compliance: Buildaudittrails,keepHDPdatalakecurrent• Productivity:NoneedfordeepunderstandingofHadoop

HotelChain:EaseofUse,Timely&Up-to-DateReporting

28

• Challenge: Moretimelycollection&reportingonroomavailability,eventbookings,inventoryandotherhoteldatafrom4,000+propertiesglobally

• Solution:• Nearreal-timereporting• DMX-hconsumespropertyupdatesfromKafkaevery10s• DMX-hprocessesdataonHDP,loadingtoTDevery30min• DeployedonGoogleCloudPlatform

• Benefits:• TimetoValue:DMX-heaseofusedrasticallycutdevelopmenttime• Agility:Reportsupdatedevery30minutesvsevery24hours• Productivity:LeveragingETLteamforHadoop(Spark),visualunderstandingofdatapipeline• Insight: Up-to-datedata=betterbusinessdecisions=happiercustomers

LeadingMediaCompany:AccelerateNewBusinessInitiatives

29

• Challenge: Buildscalableplatformtosupportnewbusinessinitiatives&scalefordouble-digitdatagrowth,whilereducingescalatingEDW&ELTCosts

• Solution:• Shiftdatastorage&processingoutoftheEDWintoHadoop• Migrate500+SQLELTworkloadstoDMX-honHDP

• Benefits:• Agility: Scalablearchitecturetodeploynewbusinessinitiatives– analyzemoresettopboxdata,blend

websiteuseractivitydata,etc.• Cost:MillionsofdollarsinsavingsfromEDW,includingSQLtuning&maintenancecosts• Productivity:ETLdeveloperscanstopcoding&tuning,andgetup&runningonHadoopquickly


Q/A

LearnMore:Ã EDWOptimizationwithHDP

– http://hortonworks.com/solutions/edw-optimization/– EDWOptimization7minvideo

Ã TrySyncsort DMX-h:syncsort.com/try– Yourexistingclusteror useourfullyfunctionalHortonworksSandbox– Getajump-startwithourlibraryofpre-builtjobs,includingUseCaseAcceleratorsforHadoop

ETLandHDFSExtract&Load

Ã Syncsort DMX-h– http://hortonworks.com/partner/syncsort/