Introduction to Sqoop

Table of Contents

Sqoop - Introduction

Integration of RDBMS and Sqoop

Sqoop use case

Sample sqoop commands

Key features of Sqoop

What is Sqoop?

Sqoop is

… a suite of tools that connect Hadoop and database systems

Major functions of Sqoop• Import tables from databases into HDFS for deep

analysis• Replicate database schemas in Hive’s metastore• Export MapReduce results back to a database for

presentation to end-users

RDBMS important but vulnerable?

Importance of RDBMS• Holds a lot of valuable data in the form of structured tables of

several hundred GB• Provides fast access for OLTP applications like Update / delete

records, Add individual records, Complex transactions

Vulnerability• Can’t store very large datasets (1 TB+)• Poor support for complex datatypes/ large objects• Schema evolution is hard• Analytic queries better suited to a batch-oriented system

RDBMS and Hadoop

RDBMS

HDFS

Historical data(before processing)

Results of data Analysis

(after processing)

Sqoop use case : Demographics-aware site analytics

Sample Sqoop commands

Import using Sqoop• sqoop import

--connect jdbc:mysql://db.foo.com/corp

--table user-profiles

Export using Sqoop• sqoop export

--connect jdbc:mysql://db.foo.com/corp

--table ads_results

--export-dir results

JDBC mysql driver

Input : mysql tableHdfs location with

analysis results

Output : mysql table

Key features of Sqoop

JDBC-based implementation - Works with many popular database vendors

Auto-generation of tedious user-side code- Writing MapReduce applications to work with data, faster

Integration with Hive - Allows to stay in a SQL-based environment

THANK YOU