python for data analysis - wordpress.com · data frame data types pandas type native python type...
TRANSCRIPT
![Page 1: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/1.jpg)
Python for Data Analysis Prof.Sushila Aghav-Palwe
Assistant Professor
MIT
![Page 2: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/2.jpg)
Four steps to apply data analytics: 1. Define your Objective
• What are you trying to achieve?
• What could the result look like?
2. Understand Your Data Source • What information do I need?
• Can I get the data myself, or do I need to ask an IT resource? Or any other sources
3. Prepare Your Data • Does the data need to be cleansed?
• Does the data need to be normalized?
4. Analyze Data
2
![Page 3: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/3.jpg)
Data Information and Knowledge
• Data–Items that are the most elementary descriptions of things, events, activities, and transactions, May be internal or external
• Information–Organized data that has meaning and value
• Knowledge–Processed data or information that conveys understanding or learning applicable to a problem or activity
Data Information Knowledge
![Page 4: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/4.jpg)
Decision Support System
• Decision-support systems are used to make business decisions, often based on data collected by on-line transaction-processing systems.
• Examples of business decisions: What items to stock?
• Examples of data used for making decisions Retail sales transaction details
Customer profiles (income, age, gender, etc.)
![Page 5: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/5.jpg)
DSS Overview
• A data warehouse archives information gathered from multiple sources, and stores it under a unified schema, at a single site.
–Important for large businesses that generate data from multiple divisions, possibly at multiple sites
–Data may also be purchased externally
• Data analysis tasks are simplified by specialized tools and SQL extensions and Statistical Analysis
For each product category and each region, what were the total sales in the last quarter and how do they compare with the same quarter last year
As above, for each product category and each customer category
• Data mining seeks to discover knowledge automatically in the form of statistical rules and patterns from large databases.
![Page 6: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/6.jpg)
Python Libraries for Data Science
Many popular Python toolboxes/libraries: • NumPy • Pandas • SciPy • SciKit-Learn
Visualization libraries • matplotlib • Seaborn
and many more …
6
![Page 7: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/7.jpg)
Python Libraries for Data Science
NumPy: introduces objects for multidimensional arrays and matrices, as well as
functions that allow to easily perform advanced mathematical and statistical operations on those objects
provides vectorization of mathematical operations on arrays and matrices which significantly improves the performance
many other python libraries are built on NumPy
7
Link: http://www.numpy.org/
![Page 8: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/8.jpg)
Python Libraries for Data Science
SciPy: collection of algorithms for linear algebra, differential equations, numerical
integration, optimization, statistics and more
part of SciPy Stack
built on NumPy
8
Link: https://www.scipy.org/scipylib/
![Page 9: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/9.jpg)
Python Libraries for Data Science
Pandas: adds data structures and tools designed to work with table-like data
provides tools for data manipulation: reshaping, merging, sorting, slicing, aggregation etc.
allows handling missing data
9
Link: http://pandas.pydata.org/
![Page 10: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/10.jpg)
Link: http://scikit-learn.org/
Python Libraries for Data Science
SciKit-Learn: provides machine learning algorithms: classification, regression, clustering,
model validation etc.
built on NumPy, SciPy and matplotlib
10
![Page 11: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/11.jpg)
matplotlib: python 2D plotting library which produces publication quality figures in a
variety of hardcopy formats
a set of functionalities similar to those of MATLAB
line plots, scatter plots, barcharts, histograms, pie charts etc.
relatively low-level; some effort needed to create advanced visualization
Link: https://matplotlib.org/
Python Libraries for Data Science
11
![Page 12: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/12.jpg)
Seaborn: based on matplotlib
provides high level interface for drawing attractive statistical graphics
Similar (in style) to the popular ggplot2 library in R
Link: https://seaborn.pydata.org/
Python Libraries for Data Science
12
![Page 13: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/13.jpg)
Start Jupyter nootebook
13
![Page 14: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/14.jpg)
In [ ]:
Loading Python Libraries
14
#Import Python Libraries
import numpy as np
import scipy as sp
import pandas as pd
import matplotlib as mpl
import seaborn as sns
Press Shift+Enter to execute the jupyter cell
![Page 15: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/15.jpg)
In [ ]:
Reading data using pandas
15
#Read csv file
df = pd.read_csv("C:\\Users\\Administrator\\Desktop\\Data Analytics with
Python\\Emp.csv")
There is a number of pandas commands to read other data formats:
pd.read_excel('myfile.xlsx',sheet_name='Sheet1’)
pd.read_stata('myfile.dta')
pd.read_sas('myfile.sas7bdat')
pd.read_hdf('myfile.h5','df')
Note: The above command has many optional arguments to fine-tune the data import process.
![Page 16: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/16.jpg)
In [3]:
Exploring data frames
16
#List first 5 records
df.head()
Out[3]:
![Page 17: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/17.jpg)
Hands-on exercises
17
Try to read the first 10, 20, 50 records;
Can you guess how to view the last few records;
![Page 18: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/18.jpg)
Data Frame data types
Pandas Type Native Python Type Description
object string The most general dtype. Will be assigned to your column if column has mixed types (numbers and strings).
int64 int Numeric characters. 64 refers to the memory allocated to hold this character.
float64 float Numeric characters with decimals. If a column contains numbers and NaNs(see below), pandas will default to float64, in case your missing value has a decimal.
datetime64, timedelta[ns] N/A (but see the datetime module in Python’s standard library)
Values meant to hold time data. Look into these for time series experiments.
18
![Page 19: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/19.jpg)
In [4]:
Data Frame data types
19
#Check a particular column type
df['salary'].dtype
Out[4]: dtype('int64')
In [5]: #Check types for all the columns df.dtypes
Out[4]: rank discipline phd service sex salary dtype: object
object object int64 int64 object int64
![Page 20: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/20.jpg)
Data Frames attributes
20
Python objects have attributes and methods.
df.attribute description
dtypes list the types of the columns
columns list the column names
axes list the row labels and column names
ndim number of dimensions
size number of elements
shape return a tuple representing the dimensionality
values numpy representation of the data
![Page 21: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/21.jpg)
Hands-on exercises
21
Read Emp.csv File
Display first 5 Rows
Display Datatype of all columns
How many elements are there?
What are the column names?
![Page 22: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/22.jpg)
Data Frames methods
22
df.method() description
head( [n] ), tail( [n] ) first/last n rows
describe() generate descriptive statistics (for numeric columns only)
max(), min() return max/min values for all numeric columns
mean(), median() return mean/median values for all numeric columns
std() standard deviation
sample([n]) returns a random sample of the data frame
dropna() drop all the records with missing values
Unlike attributes, python methods have parenthesis. All attributes and methods can be listed with a dir() function: dir(df)
![Page 23: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/23.jpg)
Hands-on exercises
23
Give the summary for the numeric columns in the dataset
Calculate standard deviation for all numeric columns;
What are the mean values of the first 5 records in the dataset?
![Page 24: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/24.jpg)
Selecting a column in a Data Frame
Method 1: Subset the data frame using column name:
df['sex']
Method 2: Use the column name as an attribute:
df.sex
Note: there is an attribute rank for pandas data frames, so to select a column with a name "rank" we should use method 1.
24
![Page 25: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/25.jpg)
Hands-on exercises
25
Calculate the basic statistics for the salary column;
Find how many values in the salary column (use count method);
Calculate the average salary;
![Page 26: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/26.jpg)
Data Frames groupby method
26
Using "group by" method we can:
• Split the data into groups based on some criteria • Calculate statistics (or apply a function) to each group • Similar to dplyr() function in R
In [ ]: #Group data using rank df_rank = df.groupby(['rank'])
In [ ]: #Calculate mean value for each numeric column per each group df_rank.mean()
![Page 27: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/27.jpg)
Data Frames groupby method
27
Once groupby object is create we can calculate various statistics for each group:
In [ ]: #Calculate mean salary for each professor rank: df.groupby('rank')[['salary']].mean()
Note: If single brackets are used to specify the column (e.g. salary), then the output is Pandas Series object. When double brackets are used the output is a Data Frame
![Page 28: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/28.jpg)
Data Frames groupby method
28
groupby performance notes:
- no grouping/splitting occurs until it's needed. Creating the groupby object only verifies that you have passed a valid mapping - by default the group keys are sorted during the groupby operation. You may want to pass sort=False for potential speedup:
In [ ]: #Calculate mean salary for each professor rank: df.groupby(['rank'], sort=False)[['salary']].mean()
![Page 29: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/29.jpg)
Data Frame: filtering
29
To subset the data we can apply Boolean indexing. This indexing is commonly known as a filter. For example if we want to subset the rows in which the salary value is greater than $120K:
In [ ]: #Calculate mean salary for each professor rank:
df_sub = df[ df['salary'] > 120000 ]
In [ ]: #Select only those rows that contain female professors: df_f = df[ df['sex'] == 'Female' ]
Any Boolean operator can be used to subset the data: > greater; >= greater or equal; < less; <= less or equal; == equal; != not equal;
![Page 30: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/30.jpg)
Data Frames: Slicing
30
There are a number of ways to subset the Data Frame: • one or more columns • one or more rows • a subset of rows and columns
Rows and columns can be selected by their position or label
![Page 31: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/31.jpg)
Data Frames: Slicing
31
When selecting one column, it is possible to use single set of brackets, but the resulting object will be a Series (not a DataFrame):
In [ ]: #Select column salary: df['salary']
When we need to select more than one column and/or make the output to be a DataFrame, we should use double brackets:
In [ ]: #Select column salary: df[['rank','salary']]
![Page 32: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/32.jpg)
Data Frames: Selecting rows
32
If we need to select a range of rows, we can specify the range using ":"
In [ ]: #Select rows by their position: df[10:20]
Notice that the first row has a position 0, and the last value in the range is omitted: So for 0:10 range the first 10 rows are returned with the positions starting with 0 and ending with 9
![Page 33: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/33.jpg)
Data Frames: method loc
33
If we need to select a range of rows, using their labels we can use method loc:
In [ ]: #Select rows by their labels: df_sub.loc[10:20,['rank','sex','salary']]
Out[ ]:
![Page 34: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/34.jpg)
Data Frames: method iloc
34
If we need to select a range of rows and/or columns, using their positions we can use method iloc:
In [ ]: #Select rows by their labels: df_sub.iloc[10:20,[0, 3, 4, 5]]
Out[ ]:
![Page 35: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/35.jpg)
Data Frames: method iloc (summary)
35
df.iloc[0] # First row of a data frame
df.iloc[i] #(i+1)th row
df.iloc[-1] # Last row
df.iloc[:, 0] # First column
df.iloc[:, -1] # Last column
df.iloc[0:7] #First 7 rows
df.iloc[:, 0:2] #First 2 columns
df.iloc[1:3, 0:2] #Second through third rows and first 2 columns
df.iloc[[0,5], [1,3]] #1st and 6th rows and 2nd and 4th columns
![Page 36: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/36.jpg)
Data Frames: Sorting
36
We can sort the data by a value in the column. By default the sorting will occur in ascending order and a new data frame is return.
In [ ]: # Create a new data frame from the original sorted by the column Salary df_sorted = df.sort_values( by ='service')
df_sorted.head()
Out[ ]:
![Page 37: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/37.jpg)
Data Frames: Sorting
37
We can sort the data using 2 or more columns:
In [ ]: df_sorted = df.sort_values( by =['service', 'salary'], ascending = [True, False]) df_sorted.head(10)
Out[ ]:
![Page 38: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/38.jpg)
Aggregation Functions in Pandas
38
Aggregation - computing a summary statistic about each group, i.e. • compute group sums or means • compute group sizes/counts
Common aggregation functions:
min, max count, sum, prod mean, median, mode, mad std, var
![Page 39: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/39.jpg)
Aggregation Functions in Pandas
39
agg() method are useful when multiple statistics are computed per column:
In [ ]: flights[['dep_delay','arr_delay']].agg(['min','mean','max'])
Out[ ]:
![Page 40: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/40.jpg)
Basic Descriptive Statistics
40
df.method() description
describe Basic statistics (count, mean, std, min, quantiles, max)
min, max Minimum and maximum values
mean, median, mode Arithmetic average, median and mode
var, std Variance and standard deviation
sem Standard error of mean
skew Sample skewness
kurt kurtosis
![Page 41: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/41.jpg)
Graphics to explore the data
41
To show graphs within Python notebook include inline directive:
In [ ]: %matplotlib inline
Seaborn package is built on matplotlib but provides high level interface for drawing attractive statistical graphics, similar to ggplot2 library in R. It specifically targets statistical data visualization
![Page 42: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/42.jpg)
Graphics
42
description
distplot histogram
barplot estimate of central tendency for a numeric variable
violinplot similar to boxplot, also shows the probability density of the data
jointplot Scatterplot
regplot Regression plot
pairplot Pairplot
boxplot boxplot
swarmplot categorical scatterplot
factorplot General categorical plot
![Page 43: Python for Data Analysis - WordPress.com · Data Frame data types Pandas Type Native Python Type Description object string The most general dtype. Will be assigned to your column](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f0360677e708231d408e93b/html5/thumbnails/43.jpg)
Basic statistical Analysis
43
statsmodel and scikit-learn - both have a number of function for statistical analysis The first one is mostly used for regular analysis using R style formulas, while scikit-learn is more tailored for Machine Learning. statsmodels:
• linear regressions • ANOVA tests • hypothesis testings • many more ...
scikit-learn:
• kmeans • support vector machines • random forests • many more ...
See examples in the Tutorial Notebook