fully automated updating of arbitrarily complex excel
TRANSCRIPT
Fully Automated Updating of Arbitrarily Complex Excel WorkbooksDavid OesperLands’ End
WIILSUJune 28, 2017
The Challenge: Automation
• A coworker left the company and I inherited a set of his programs that he ran manually at the end of every fiscal month.
• He generated a series of data sheets which he then manually cut and pasted into existing Excel workbooks.
• After all that, he manually emailed the revised Excel workbook to several recipients.
The Challenge: Excel
• The Excel workbooks my coworker was updating were not simple grids of rows and columns.
• Our clients who received these Excel workbooks designed them and preferred not to make any changes to the design.
• It was evident to me that SAS alone would not be able to reproduce these workbooks exactly (at least not without a lot of work!).
The Challenge: A SAS Solution?
• I knew how to add hidden data sheets and add references to them on the main worksheet.
• But could SAS, running on Linux, make a “strategic strike” on an existing Excel workbook, and update or replace existing data sheets?
• If I could update the workbook, then emailing the revised workbook and/or uploading it to a Windows directory would be easy.
Five Step Approach
• Create a static version of your Excel workbook to use as a “model” for SAS, and
• Add one or more hidden data sheets
• Clone a copy of your Excel workbook for editing
• Replace your data sheets “on the fly”
• Save or email to the destination(s) of your choice
What You’ll Need
• SAS 9.4 TS1M2 or later
• SAS/ACCESS to PC Files
• Microsoft Excel (to prepare your model Excel workbook with data sheets)
Step #1
Create a static version of your Excel workbook to use as a “model” for SAS
Excel Workbook to UpdateU.S. Census Divisions – Population by State
Add one or more hidden data sheets
Step #2
Excel Workbook to UpdateU.S. Census Divisions – Population by State
Excel Workbook to UpdateOur Data Sheet
proc print data=sashelp.us_data;title 'sashelp.us_data';var statename population_2010
division;run;
Excel Workbook to UpdateU.S. Census Divisions – Population by State
Data!B4
Excel Workbook to UpdateOur Data Sheet
proc print data=sashelp.us_data;title 'sashelp.us_data';var statename population_2010
division;run;
Data!B4
Excel Workbook to UpdateU.S. Census Divisions – Population by State
Clone a copy of your Excel workbook for editing
Step #3
Copy Excel Workbook for EditingPC SAS
/* Today's date in yyyymmdd format: example 20170503 */%let tdate = %sysfunc(today(),yymmddn8.);
/* Path and File Name of model Excel workbook */%let xlinpath = C:\Users\David\Documents\SASIN;%let xlinfile = U.S. Census Divisions.xlsx;
/* Path and File Name of output Excel workbook */%let xloutpath = C:\Users\David\Documents\SASOUT;%let xloutfile = U.S. Census Divisions &tdate..xlsx;
Copy Excel Workbook for EditingPC SAS
/* Copy the Excel Workbook */options noxwait noxsync;data _null_;
call system(%unquote(%bquote(')copy "&xlinpath\&xlinfile"
"&xloutpath\&xloutfile"%bquote(')));
run;options xwait xsync;
Copy Excel Workbook for EditingLinux SAS
%let userid = daoespe;
/* Today's date in yyyymmdd format: example 20170503 */%let tdate = %sysfunc(today(),yymmddn8.);
/* Path and File Name of model Excel workbook */%let xlinpath = /sasdevr/devr180/&userid;%let xlinfile = U.S. Census Divisions.xlsx;
/* Path and File Name of output Excel workbook */%let xloutpath = /sasdevr/devr7/&userid;%let xloutfile = U.S. Census Divisions &tdate..xlsx;
Copy Excel Workbook for EditingLinux SAS
/* Copy the Excel Workbook */data _null_;
/* change to input directory */call system("cd &xlinpath");/* copy model Excel workbook to output directory */call system("cp –f %unquote(%bquote(')
&xlinfile%bquote(')) &xloutpath");/* change to output directory */call system("cd &xloutpath");/* rename excel file on output directory */call system("mv %unquote(%bquote(')
&xlinfile%bquote('))%unquote(%bquote(')
&xloutfile%bquote('))");run;
Replace your data sheets “on the fly”
Step #4
Getting Our Updated Census DataScoop .csv file off a web site: filename url
2016 U.S. population estimate
Getting Our Updated Census DataScoop .csv file off a web site: filename url
filename newpop url"https://www2.census.gov/programs-surveys/popest/datasets/2010-2016/state/asrh/scprc-est2016-18+pop-res.csv";
2016 U.S. population estimate
You could also use PROC HTTP to get the data.
Getting Our Updated Census Data
Existingdata sheet
SAS data set for new data sheet
Excel row #s off by 1 -Column headers take a row
Create an Updated Excel WorkbookUpdate Working Copy of Excel File!
/* use LIBNAME XLSX access method */libname xlout xlsx "&xloutpath/&xloutfile";
/* update data sheet with a merge */data xlout.'Data'n;
merge xlout.'Data'n sasuser.newpop;by statename;
run; This is the Key!!LIBNAME XLSX
+Simple DATA Step MERGE!
SAS Analytics Pro – PC SASDoes Not Include SAS/ACCESS to PC Files
/* update data sheet with a merge */data xlout.'Data'n;
merge xlout.'Data'n sasuser.newpop;by statename;
run;
SAS LOGlibname xlout xlsx "&xloutpath/&xloutfile";
ERROR: The product with which the engine XLSX is associated is either not licensed for your system or the product license has expired. Please contact your SAS installation representative.ERROR: Error in the LIBNAME statement.
Updated Excel Workbook2016 Population Estimates
Data sheet was hidden in the original workbook – remains hidden
Save or email to the destination(s) of your choice
Step #5
Copy Updated Excel WorkbookFrom Linux to Windows
/* filename same as output libname */filename xlout "&xloutpath/&xloutfile";
/* filename for FTP to Windows drive */filename ftpout ftp
"DIRMARKET124/BA/Staff/Dave/SASOUT/&xloutfile"recfm=s host='********' user='********‘pass='********';
Copy Updated Excel WorkbookFrom Linux to Windows
/* copy the file byte-for-byte */data _null_;
length filein 8 fileid 8;filein = fopen('xlout','I',1,'B');fileid = fopen('ftpout','O',1,'B');rec = '20'x;do while(fread(filein)=0);
rc = fget(filein,rec,1);rc = fput(fileid,rec);rc = fwrite(fileid);
end;rc = fclose(filein);rc = fclose(fileid);
run;
Email Updated Excel WorkbookFrom Linux
/* output filename using the email accessmethod - notice we are emailing the Excelworkbook we created on Linux */
filename eml emailto = ('[email protected]')cc = ('[email protected]')type = 'text/plain‘emailid = '[email protected]'attach = ("&xloutpath/&xloutfile"ct="application/octet-stream" ext="xlsx")subject = 'Updated U.S. Census Data';
Email Updated Excel WorkbookFrom Linux
/* Send email, but only if no errors */%macro sendemail;%if %length(&syserrortext) ne 0 %then %return;data _null_;
file eml;put "Here are the updated population estimates
for each state from the U.S. CensusBureau." /;
put 'Best Regards,' /;put 'Dave';
run;%mend sendemail;%sendemail;
Email Updated Excel WorkbookFrom Linux
Email Updated Excel WorkbookFrom Linux
Email Updated Excel WorkbookFrom Linux
Summary
• We used the LIBNAME XLSX engine, a part of SAS/ACCESS to PC File Formats, to update data sheets in an Excel workbook using a simple DATA step MERGE.
• We used FILENAME URL to “scoop” a .csv file off of a web site to get our updated data.
• We used FILENAME FTP to copy our updated Excel workbook from a Linux server to a Windows directory.
• We used FILENAME EMAIL to email our updated Excel workbook to a recipient list.