programmability in spss statistics 17

What's New with Programmability and Scripting

Jon Peck

Technical Advisor and Principal Software Engineer

Las Vegas, November 2008 Copyright (c) SPSS Inc, 2008Copyright (c) SPSS Inc, 2008

Do you have...

• Repetitive SPSS tasks?• A need to reduce manual work?• Problems you can't solve with traditional

syntax?• A need for statistical procedures not in SPSS

Statistics?• A need for higher productivity?• A need to reduce code maintenance time?

See if programmability can help...

Copyright (c) SPSS Inc, 2008

Agenda

• Programmability introduction• Four examples

– Automating repetitive work: applySyntaxToFiles

– Integrating programs and scripting: SPSSINC MODIFY TABLES

– Adding R statistical procedures: SPSSINC QUANTILE REGRESSION

– Making your own user interface with .NETThe Statistical Explorer


SPSS Statistics embeds three programming languages

• Traditional syntax and Basic scripting still present• Plug-ins let you extend capabilities using

– Python

– R

– .NET languages (Windows only)

• Free plug-in downloads or get from cd• SPSS Developer Central web site provides

articles, SPSS-written modules, plug-ins and user contributions


http://www.spss.com/devcentral

BEGIN PROGRAM PYTHON.import spssprint "Hello, Las Vegas"END PROGRAM

• Python or R program code goes in the normal SPSS Statistics syntax window

My first program


• A program in the input stream can communicate with SPSS Statistics and control it and use Python or R facilities and modules (internal mode)

spss.Submit("GET FILE='c:/data/cars.sav'.")

• A Python or .NET application can embed SPSS Statistics inside itself (external mode)– User interface does not appear

• There is a lower level C API available in an SDK

Programmability combines SPSS Statistics with Python or R


Python and R are open source software

• Programmability plug-ins are an optional install– They are free– They make possible tapping the work of the

Python and R communities– Python and R have license agreements– SPSS has a Freeware license agreement

• SPSS is not the owner or licensor of the Python or R software. Any user of Python or R must agree to the terms of the license agreement located on the Python or R web site. SPSS is not making any statement about the quality of the Python or R programs. SPSS fully disclaims all liability associated with your use of the Python or R programs.


Programmability increases your power, flexibility, and productivity

• Generalization

– React flexibly to metadata, results, and the environment

• Automation

– Embed program logic in jobs

• Extension

– Add or extend procedures and transformations

– Tap existing R or Python statistical modules

• Integration

– Connect SPSS inputs and outputs to other agents

• And programmability is more funCopyright (c) SPSS Inc, 2008

Version 17 has new programmability and scripting features

V 15

• Processor– SPSS syntax– Python

programs– .NET

programs

• Front End– SaxBasic

scripts– COM support

V 16 Additions

• Processor– R programs– Extension

commands– Dataset class

• Front End– Basic script

upgrade– Python

scripts

V 17 Additions

• Processor– Extension

improvements– Program and

script integration– R graphics in

Viewer

• Front End– Custom Dialog

Builder– New syntax

editorCopyright (c) SPSS Inc, 2008

• Programs run in the regular syntax stream• Users can define SPSS Statistics syntax for

program and scripts via Extension mechanism.

• Users can create dialog boxes and menus using the Custom Dialog Builder.– Not just for extensions or programs

• Python and R output appears in the Viewer– plain text– pivot tables– charts

Programmability functionality is fully integrated into SPSS Statistics


• New for version 17– SETSMACRO: syntax for using Variable Sets– SPSSINC APRIORI: association rules (R)– SPSSINC HETCOR: polychoric and serial correlation (R)– SPSSINC MERGE TABLES: merge pivot tables– SPSSINC MODIFY OUTPUT: outline titling and styling– SPSSINC MODIFY TABLES: pivot table styling– SPSSINC QUANTREG: quantile regression (R)– SPSSINC RAKE: adjust weights to control totals– SPSSINC RANFOR/RANPRED: random forests (R)– SPSSINC RASCH: Rasch models (R)– SPSSINC ROBUST REGR: robust regression (R)– SPSSINC TOBIT REGR: Tobit regression (R)

• For version 16 (and 17)– Seven others

There are new extension commands available from Developer Central


– Write Python or R functions to implement the functionality• Use input API's to get data to Python or R• Use output API's to create pivot tables

– For extensions,• Define the syntax in an xml file• Use tools in extension.py to receive parsed output and pass to

implementing function– R extensions can be wrapped in Python code

• Use organization or author name as first word of command

– Use the Custom Dialog Builder to create the interface• The CDB is not just for extensions

– Test!– Package and distribute– Contributions to Developer Central are welcome

• Documentation available at SPSS Developer Central

You can create and share your own additions to SPSS Statistics



Expand the audience by creating SPSS Statistics syntax and dialog boxes


Generalize and automate work

Example I


• You have syntax files and need to process datasets not known in advance every day

• applySyntaxToFiles function applies a syntax file to each file in input specification

• Optionally saves new sav and output files• Optionally creates a log• Can be run in internal or external mode• Can be integrated into a larger process

applySyntaxToFiles


• Apply standard processing to an unknown set of files• Produce processed data and reports

Automating a routine process


begin program.import spss, spssaux3spssaux3.applySyntaxToFiles(inputspec="c:/temp/parts/*.sav", syntax = "c:/userconf2008lasvegas/dailychecks.sps", outputdatadir = "c:/temp/processed", outputfiledir = "c:/temp/processed", logfile ="c:/temp/processed/report.txt")end program.

• dailychecks.sps could apply data cleaning rules, modify data, and create reports

• This program could be run daily through Production Mode or PES job scheduler or used interactively

Use a program to drive processing


Use integrated scripting for better table presentation

Example II


• TableLooks provide static formatting for entire areas of a table– data cells– row and column layers

• You want tables with formatting beyond tableLooks• Many users copy tables to Excel and manually format

them • Basic and Python Scripting provide programmatic way

to do formatting• SPSSINC MODIFY TABLES provides syntax for

extensive formatting– Eliminates need to know scripting– Uses Extension mechanism for programs and Python

scripting

SPSSINC MODIFY TABLES extension command


SPSSINC MODIFY TABLES SUBTYPE='Crosstabulation' DIMENSION=ROWS SELECT='Std. Residual'/STYLES TEXTSTYLE=BOLD BACKGROUNDCOLOR=255 0 0APPLYTO='abs(x) >2'.

Use dynamic highlighting to make crosstab table easier to read


• Dialog created withCustom Dialog Builder

• Generates extension command syntax

• Easy to distribute

Custom dialog boxes are easy to create


SPSSINC MODIFY TABLES subtype='variables in the equation' SELECT="B" "Sig."/STYLES TEXTCOLOR = 0 0 255 BACKGROUNDCOLOR=0 255 0.


Use static formatting to call out parts of a table

SPSSINC MODIFY TABLES SUBTYPE="Custom Table" SELECT = "Total" DIMENSION=ROWS/STYLES BACKGROUNDCOLOR=255 255 88 TEXTSTYLE = BOLD

Format CTABLES totals to call them out


SPSSINC MODIFY TABLES SUBTYPE='Report' SELECT="<<ALL>>"/STYLES APPLYTO=DATACELLS TEXTCOLOR=255 255 255 TEXTSTYLE=BOLDCUSTOMFUNCTION="customstylefunctions.washColumnsBlue".

def washColumnsBlue(obj, i, j, numrows, numcols, section, more): mincolor=150. maxcolor=255. increment = (maxcolor - mincolor)/(numcols-1) colorvalue = round(mincolor + increment * j) obj.SetBackgroundColorAt(i,j, RGB((mincolor, mincolor, colorvalue)))

Use custom functions for special effects



It is possible to get carried away with this

Add R procedures seamlessly to SPSS Statistics

Example III


• R – has many advanced statistical procedures– is a complex language for programming statistics– is difficult to learn

• The R plug-in makes it easy to use R packages– SPSS Statistics datasets and Viewer output can be

processed by R using plug-in– Graphical, text, and table output appear in the Viewer– New SPSS datasets can be created from R– R can communicate with SPSS via 30 API's– Integration requires writing some R wrapper code

• Plug-in is downloadable from Developer Central

R procedures can be accessed from SPSS Statistics using the R plug-in


• Ordinary regression models conditional mean• Median regression is 50th quantile• Estimating quantiles is useful with varying

spread, asymmetries, outliers• Areas of application include

– empirical finance• value at risk• mutual fund investment styles• credit scoring

– school quality– demand analysis– others

Quantile regression models conditional quantiles


SPSSINC QUANTILE REGRESSION extension embeds R quantreg


Plots and pivot tables appear in the Viewer


New datasets appear in Data Editor windows


The .NET plug-in allows an easy .NET way to do your own user interface for SPSS Statistics

Example IV


The Statistical Explorer uses the .NET plug-in


The Statistical Explorer


Statistics and chart shown depends on variable measurement level


• SPSS Statistics 17 completes programmability and scripting building blocks

• Unification of programs and scripts

• Custom Dialog Builder

• Extension improvements and new extensions

• R integration and graphics in Viewer

• SPSS Developer Central is your friend

Where we have been today



? ?

Questions


• Generalization and automation

– applySyntaxToFiles

– SPSS MODIFY TABLES

• Extension

– SPSSINC MODIFY TABLES using integrated scripting

– SPSS QUANTREG using R

– Many new extension commands available

• Integration

– applySyntaxToFiles as part of a process

• And it's still more fun

Programmability increases your power, flexibility, and productivity


Contact


programmability in spss statistics 17

Data & Analytics

spss inputs

program copyright c

r communities python

fun copyright c spss

user of python

spsswritten modules

license agreements spss

r software