introduction to data science – info 480 – drexel university’s ischool

Introduction to Data Science – INFO 480 – Drexel University’s

iSchoolSean P. Goggins, PhD

April 2, 2013Week Three

What is Data Science?

Storytelling Database Theory – How you organize your

data has a big influence on what you can do with it.

Agile Manifesto – Key thing is iterative development; it’s a technology value system.

Spiral Dynamics – What we view as fact and what we desire emerges from the data presented to us.

Credit: http://www.datascientists.net/what-is-data-science

Database Theory Relational Algebra & Set Theory Thinking in relations helps you to connect

disparate data; What is the connecting field? What is the cardinality?

Set Theory Helps you think about summarizing data What time period? Weeks? Months? By person? By Group? By Geography?

Agile Manifesto Individuals and interactions over

processes and tools Working software over comprehensive

documentation Customer collaboration over contract

negotiation Responding to change over following a

plan

http://www.agilemanifesto.org

Spiral DynamicsNew research unveiled at this year’s AERA conference documents a disturbing trend among the nation’s secondary schools: Between 2001 and 2012, high school graduation rates regularly spiked in late May and early June, ballooning from near zero to a staggering average of 78 percent.

What you’ll need for this course

Interest in learning data analysis tools R Python

Curiosity A laptop to bring to class (see me if this is a problem) Persistence A Github Account Willingness to do weekly homeworks and participate in

online iteration of data products you and your course mates develop

A dropbox account will be helpful

Discuss Homework Analysis Questions. Write up a short essay with

tables or graphs if needed to describe how you would: Build a network using the scripts from week1

against the mention connections? Reply-To connections? In this sample data. What transformations are required? How would you filter the data? Use the actual data to ground your thinking. Feel free to actually write or modify the R code samples from the first two weeks to experiment. Some of you will be more comfortable doing this; some will be more comfortable addressing the question conceptually. This is OK.

Using GitHub for Software Sharing

Creating a GitHub Account Creating a GitHub Project Using the GitHub Desktop client Committing & Syncing The Pull Request

Anarchist in the Library

Chapter Four – The Music Industry Take five minutes to prepare a one minute

summary of how you use file sharing and distribution to use music and videos

I will mute the class capture when you share your stories

A personal story of Music

Who is Zooey Deschanel?

To Spotif

y!

Multiple Variable Distributions

Examples in Python

Two Variables Where are the data points

located, and how far do they spread? What are typical, as

well as minimal and maximal, values?

• How are the points distributed? Are they spread out evenly or do they cluster in certain areas?

• How many points are there? Is this a large data set or a relatively small one?

Health Expenditures

Plotting 2 variables Health Expenditures vs life expectancy Health Expenditures vs Doctor Visits

Interpolation

Spline

Polynomial Intertpolation

Draft Lottery LOESS Curve

LOESS in R

LOESS & Well Water Testing

Credit to “Davids Statistics”

False Positives & False Negatives

• This is a modern statistical method that is useful when the relationship between x & y are unknown and complicated

• “Locally weighted polynomial regression• Basic regression• Localized regression

• Localized subsets of data• Q – smoothing parameter• Bigger q = more smoothing

• Assumption: Any local model can be well approximated in a small neighborhood

• Models

The Data

Simple LOESS

Detects & Non-Detects

First Curve

Second Curve

Smoothing Curve

ggplot2 library

Detects & ND’s With Sep Curves & CI’s

Activity Download the GitHub Project Get the Python code to run inside of Canopy

(Week 4 Folder) Draft Data

R-Code Twitter Data – @j_tsar

Looking at various types of interpolation? How might interpolated data help tell a story?

Now, what other data sets do you want to get?

Underpants Gnomes

With much discourtesy from the US TV Program “South Park”

Motivation

Underpants GnomesMotivation

Addressing The Underpants Gnome

Postulate

34

Discussion Post•Read•Response

Classification•Open Coding•Axial Coding

Identification of Coordination Events•Time proximity•Topical proximity

Aggregation of Posts by

Topic

Weighted Network

Analysis of Interactions

Methodological Approach

Weight Connections Based on Time Distance, GroupedBy Topic and informed by analysis of time distance between posts.

Identify Key InformationBrokers

Group Informatics Described

Week Five – Assignment Two

Software Sharing #1 (Share scripts produced in week 3 using an open source software configuration management tool). Students will refine and then share their scripts

with other students Included in the assignment is a 500 word

explanation of how their script could be improved, optimized and adapted to other data of a similar type.

The “read me” file distributed with the script will explain to another user how to apply the script to the data distributed in assignment one. This will include specific, technical specifications.

upcoming Readings Week 5: Data Presentation Tools (We’re a

week behind on assignments) Software Sharing #1 (Share scripts

produced in week 3 using an open source software configuration management tool).

Readings and Assignments Due: Data Visualization Example Presentation Part Three of “Data Analysis with Open Source

Tools”.