1

Forget Crystal - Use Excel insteadSimple lists of data from M3 can be extracted directly by Excel using Data / Import External Data routine. It isn't so good for reports where you need conditional suppression, sub reports etc, but for a listing you want to end up in Excel it is brilliant. You may need to install the Query component if this wasn't done when Excel was loaded on your PC. You can join tables, have complex criteria and apply formulae. Here’s how:

Open a blank Excel worksheet then click menu [in 2010] DATA / FROM OTHER SOURCES / FROM MICROSOFT QUERY.

[in 2007] DATA / GET EXTERNAL DATA / NEW DATABASE QUERY.

You may be lucky and the data source is already in the listing as something like MVM Live or M3PP. If so, pick it.

If it isn’t in the list of available ones then choose NEW DATA SOURCE then the driver you use eg Microsoft ODBC for Oracle and CONNECT. Now enter your username, password and server (NLP11GL1 possibly – M3 Help-About will show you what the database name is on your server).

You can select a default table eg PPV_WORKSHEETS if you like. Do NOT tick SAVE MY ID and PASSWORD if there is a risk someone without authorisation could open the Excel file – you’d have breached the system security.

You will now see a list of all the tables and views. Scroll down to the views which start with PPV_ for the set we normally use. You can partially filter the list of tables and views via the OPTIONS on the CHOOSE COLUMNS menu eg just VIEWS or Owner = PPSV.

2Query wizard for one table/viewFind the table you need eg PPV_Worksheets then open it up by clicking on the + beside it. This will show all the fields within that table in roughly the order they appear on screen (or alphabetical order if you chose that option via the Choose Columns Options). You probably won’t be able to see the full name, so a good knowledge of what you need is vital. The Crystal Data Dictionaries produced by me can help here. Anyhow, you can scroll down until you find the first field you need in your query, or type its initial letter to jump to it quicker. Click the > to add any fields you want into your query. Be sure to add any fields you need to filter on even if you don’t really need to see them in the finished spreadsheet. When you’ve got them all, you can shuffle the order of them using the up and down icons to the right of the “columns in your query” then click NEXT.

To set your selection criteria, click on DEPARTMENT then set the top middle box to IS EQUAL TO and type EH in the top right box. You don’t need to put this text in quotes as MS Query will automatically add the

single quotes it needs (NB not the double quotes used by Crystal). You could add further criteria to the filter on Department in the middle and bottom boxes (and more if needed) eg DEPARTMENT = EH or ABC or XYZ, but we don’t need to. Then click on OUTCOME and set this to IS NOT NULL. Note that you only get to see the latest element of your query but the earlier DEPARTMENT field is now shown in bold indicating it has been used to build the query. NEXT

Add some sort criteria

NEXT

You now get three final options before you click FINISH.

3A) Return data to Excel will finish the process and extract the information (takes a minute or so). You

get a choice of location eg another tab in the spreadsheet you started in or a different cell as the top right corner of the data. Before committing to all this, please click PROPERTIES to make a couple of vital choices.

Keep SAVE QUERY DEFINITION

If you tick “SAVE PASSWORD” then anyone would be able to run this query in the future without needing to add an M3 password each time. And they would be able to see what your normal M3 password is. Quite a security risk! However you would need a password only when yours is reset (every 60 days for our database). Or perhaps your database manager can supply you with a non-changing password for this reporting purpose.

At the bottom Add FILL DOWN FORMULAS....

OK, OK

For a worrying few minutes you will stare at a screen like this and then eventually a wonderful batch of real data will fill your spreadsheet.

In Excel 2003 you get a new toolbar for EXTERNAL DATA. In 2010/2013 these controls are on the DATA ribbon,

CONNECTIONS.

EDIT QUERY (just what I need to edit the criteria and change OUTCOME to IS NULL)

DATA RANGE PROPERTIES (eg save password, Fill down formulas etc)

QUERY PARAMETERS,

REFRESH DATA ie update the information from the database. NB if this Exclamation mark icon isn’t red but is greyed out then just move your cursor so it is clicked on a cell containing some to the data query results

CANCEL REFRESH,

REFRESH ALL ie refresh all data queries in this worksheet (if there are others)

REFRESH STATUS - coloured blue if data retrieval is going on in the background and greys out once this is completed.

I used EDIT QUERY to edit the criteria and change OUTCOME to IS NULL. This then removed all the old data from Excel and replaced it with the 10,000 open worksheets we have.

4Adding formulaeYou can now enhance the raw data by inserting extra columns wherever you like and using Excel’s formula functions. These are very similar to Crystal ones, but different enough to trip you up. For example, I could have a column with the property INDSTATUS in it which displays as null for most properties, and X for any which have been closed. If this were in column H then I could use a blank column immediately to the right of* all the data or by inserting a new column inside the main data for this purpose. Type in a suitable column title eg OPEN? Click in the blank cell in row 2 and type in your formula eg

=if (H2=”X”,”Closed”,”Open”)

Which is the same as Crystal formula like IF PPV_PROPERTY.INDSTATUS=”X” then “Closed” Else “Open”

Now refresh the data and, provided you have that property “FILL DOWN FORMULAS” ticked, the formula should be added to all rows automatically.

(*NB don’t miss a column out though – if Excel sees a blank row or column around the main data it may not treat the orphaned extras as part of it when resorting or mail merge or formula fill down etc) .

Sometimes the formula isn't copied down the page automatically – so you may have to do this step manually: select the cell at the top which contains the formula and all the cells you want to have it and use CTRL d

Saving your fileIf you save your file with its hundreds of rows of data then that is fine. Whenever you click the ! refresh icon then all this old data is replaced by the new. The query external data range properties options allow some control of this, but I can't work out what impact these would have.

If you want to distribute the file to others without any of your data then you can simply delete all the rows of data and just keep the headings as the linkage to the embedded query. However if you have added a few formulae of your own then it will be necessary to keep those in the top data row. You can choose to clear the data from those other cells in this single remaining row (to preserve confidentiality of data perhaps) or overtype it with examples or just clear the text from them. Whichever you choose will be overwritten by new data on refresh.

MS query for a couple of tables/viewsIf your query is just going to get data from a single table or a few tables with inner joins then we can use the Query Wizard to hold our hand through the process. (If you get a refusal to cooperate when you try adding a more complicated collection of tables then you'll need to hack the SQL code directly.)

Let's get a list of all the open worksheets for the EH department and the phone number for those properties. This will involve using PPV_PROPERTY as well as the PPV_WORKSHEETS in our query. Note the OPTIONS icon opens up the screen where you can choose to just see VIEWS, which makes selecting the correct table easier.

5

After adding all the tables you need, click Close and then draw in the joins just as in Crystal. And a double click on the join line shows the options for join type.

If you didn’t want an inner join then click REMOVE, click on join type 2 or 3 (Left outer or right outer) and ADD

You can now add your selection criteria by clicking on CRITERIA / ADD CRITERIA. The VALUES button is the same as BROWSE DATA in Crystal – it will show a sample of the values being used in the database after a couple of minutes searching.

Click ADD and your criteria will be added to a grid half way down the screen (just like Access). Overtype the field, operator and value with your next .

6 Note that all the criteria typed on a single row have to be true for a record to be selected whereas criteria on other rows are alternative sets or criteria. So if tests A, B and C are on row 1, D and E on row 2 and F,G,H on row 3 then this would be shown in Crystal formulae like

A and B and C or (D and E) or (F and G and H)

You now need to add the fields you want shown in your query. Just double click on them in the tables and they will get added to the lower window (the Data Pane) with some sample data.

Finally, FILE / RETURN DATA TO EXCEL and the Query package will disappear and your data appear in Excel.

At this point your can choose a different location for the top left corner of the data. You can also choose to create a PIVOT TABLE. These are like Crystals Cross Tabs

Beware, when I picked this Pivot table option and have lost the option of returning to Microsoft Query to edit my choices. Perhaps I needed to actively save my query.

OperatorsDescription MSQ wizard MSQ sql code Crystal

Equal to ‘apple’ {field} =’apple’ {field} =”apple”Equal to date #31/03/2014# TO_DATE('31-03-

2014','DD-MM-YYYY') Or{ts '2014-03-31 00:00:00'}

Not equal to <> 42 {field} <> 42 {field} <> 42Greater than > 42 {field} > 42 {field} > 42Between BETWEEN x AND y {field} BETWEEN x AND y {field} IN x TO yNot between NOT BETWEEN x

AND y{field} NOT BETWEEN x AND y

NOT ({field} IN x TO y)

Any one of these values IN (1, 17, 42) {field} IN (1, 17, 42) {field} IN [1, 17, 42]Not one of these values NOT IN (1, 17, 42) {field} NOT IN (1, 17, 42) NOT({field} IN [1, 17, 42])Matches this pattern_ = any single character% = any character(s) or

LIKE ‘ap_l%’ {field} LIKE ‘ap_l%’ {field} LIKE “ap?l*”

7noneStarts with LIKE ‘app%’ {field} LIKE ‘app%’ {field} startswith “app”Contains LIKE ‘%apple%’ {field} LIKE ‘%app%’ {field} LIKE “*apple*”Data is missing / blank IS NULL {field} IS NULL ISNULL({field})Data is present IS NOT NULL {field} IS NOT NULL NOT ISNULL({field})

Changing the column headingsIn other versions of software using SQL it is possible to use the AS statement can be used to modify the column heading Excel displays which helps interpret the weird field names database designers use.

SELECT {field} AS ‘My heading’

BUT this doesn’t seem to work in the dialect used in Excel Microsoft Query with Oracle.

One work around is to place your resulting table onto row 2 (onwards) and use the very top row for all your aliases.

8

Adding parameters1. Create a query that includes the tables and fields for the records you want. 2. Make sure Auto Query is not pressed in. 3. Make sure Show/Hide Criteria is pressed in to display the Criteria pane. 4. Click the first cell in the Criteria Field row, and then click the arrow in the cell to select a field from the

list that you want to use as the parameter for the query. Click the first cell in the Value row. Type [ (a left square bracket) followed by the text for the prompt you want Microsoft Query to display when the query is run, and then type ] (a right square bracket). The text of the prompt must be different from the field name, although it can include the field name.To create more than one parameter for the query, repeat steps 4 and 5 for each parameter you want to add in the next available column in the Criteria pane.

5. Press ENTER. 6. To run the query, click Query Now. 7. From now on, whenever the user refreshes the data query a box will pop up asking for the parameter.8. Alternatively you can use any cell on the worksheet outside of the data query and set it so this value is

used whenever the query is refreshed. Use the Query Parameters icon in the query toolbar to specify which cell is to be used.

SQL codeYou will see an SQL icon on the Microsoft Query toolbar. Click on this and you'll be able to see the actual code that Query has generated using all its friendly tools. You can edit this directly if you understand SQL, but if you do anything to tricky then, although the query will work, you may get a message like "Unable to display graphically" and the window showing how the tables link may disappear. But this really doesn't matter to the end result.

Most valuable is the ability to copy the SQL code from a Crystal report you are happy with (MENU:Database/Show SQL query), and paste it into this SQL code window in MS Query, so it totally replaces whatever code is there from the query you started writing. That is a very quick way of replicating the basis of a complex report.

Here's an example of the SQL code from a report that lists all commercial premises and joins the Property with the Property_lpg views:

SELECT PPV_PROPERTY_LPG.TRADNAME, PPV_PROPERTY_LPG.UKEY, PPV_PROPERTY_LPG.STREETID,     PPV_PROPERTY.INDTYPE, PPV_PROPERTY.INDSTATUS, PPV_PROPERTY.DTCLOSED, PPV_PROPERTY.AREA, PPV_PROPERTY.TEL, PPV_PROPERTY.EMPTOT, PPV_PROPERTY.COMMAREA, PPV_PROPERTY.USEEH, PPV_PROPERTY.USEEHDESC, PPV_PROPERTY.MAFF, PPV_PROPERTY.MAFFDESC, PPV_PROPERTY.HASWA, PPV_PROPERTY.HASWADESC, PPV_PROPERTY_LPG.POSNREF, PPV_PROPERTY_LPG.NAMESEC, PPV_PROPERTY_LPG.NAMEPRI, PPV_PROPERTY_LPG.STNAME, PPV_PROPERTY_LPG.STLOCAL, PPV_PROPERTY_LPG.STTOWN, PPV_PROPERTY_LPG.STCOUNTY, PPV_PROPERTY_LPG.POSTCD, PPV_PROPERTY_LPG.CORPUPRN, PPV_PROPERTY_LPG.EASTING, PPV_PROPERTY_LPG.NORTHING,PPV_PROPERTY.ALERT, PPV_PROPERTY.REMARKFROMPPSV.PPV_PROPERTY PPV_PROPERTY,PPSV.PPV_PROPERTY_LPG PPV_PROPERTY_LPGWHEREPPV_PROPERTY.UKEY = PPV_PROPERTY_LPG.UKEY AND(PPV_PROPERTY.INDTYPE = 'M' OR PPV_PROPERTY.INDTYPE = 'C')ORDER BYPPV_PROPERTY_LPG.STNAME ASC,PPV_PROPERTY_LPG.STREETID ASC,PPV_PROPERTY_LPG.POSNREF ASC,PPV_PROPERTY_LPG.CORPUPRN ASC,

9 PPV_PROPERTY.INDSTATUS DESC,PPV_PROPERTY.DTCLOSED DESC

(Note – code copied from Crystal has double quotes around all the fields but it runs just as well without them – and is then easier to edit. But the single quotes around string values are essential. To make future editing easier, I paste the code from Crystal into Notepad (NOT Word) and then use the Replace command to strip out all ")

SelectIn SQL the SELECT statement just lists all the columns, in sequence, for the output. If you want a blank column then just have a pair of single quotes (not a double quote) then a comma like '',

You can use SELECT * FROM tablename to get every column without having to type them out.

Note the column name comprises table.field just as in Crystal, but without the curly brackets.

FromIn SQL the FROM statement lists the tables (or views) then gives them a shortened alias. It may also specify the joins between multiple tables, or that might be shown in the WHERE statement (as example above).

An inner join between two tables could look like:

FROM PPSV.PPV_ACTIONS PPV_ACTIONS INNER JOIN PPSV.PPV_WORKSHEETS PPV_WORKSHEETS ON PPV_ACTIONS.SRCREF=PPV_WORKSHEETS.WORKID

Alternatively, to make this easier to understand, though it wouldn't work if you tried it, I'm going to express the full table name and its shorter alias with the letters "A" an "B"

FROM A INNER JOIN B ON x=y

An inner joins with a third table requires the bit in red added in:

FROM (PPSV.PPV_ACTIONS PPV_ACTIONS INNER JOIN PPSV.PPV_WORKSHEETS PPV_WORKSHEETS ON PPV_ACTIONS.SRCREF=PPV_WORKSHEETS.WORKID) INNER JOIN PPSV.PPV_PROPERTY PPV_PROPERTY ON PPV_WORKSHEETS.UKEY=PPV_PROPERTY.UKEY

Or

FROM (A INNER JOIN B ON x=y)INNER JOIN C ON x=z

Or a left outer join with a third table would look like this:

FROM (PPSV.PPV_WORKSHEETS PPV_WORKSHEETS INNER JOIN PPSV.PPV_PROPERTY PPV_PROPERTY

A B

A B C

10ON PPV_WORKSHEETS.UKEY=PPV_PROPERTY.UKEY) LEFT OUTER JOIN PPSV.PPV_ACTIONS PPV_ACTIONS ON PPV_WORKSHEETS.WORKID=PPV_ACTIONS.SRCREF

Or

FROM (A INNER JOIN B ON x=y)LEFT OUTER JOIN C ON x=z

….Note that the left outer join is to the first mentioned table in the inner join statement. The join in this example will show ALL worksheets (even if they don't have any actions yet) and show the actions related to those worksheets.

Three tables with left outer joins would go like:

FROM (A LEFT OUTER JOIN B ON x=y)LEFT OUTER JOIN C ON x=z

You can also have OUTER joins to list all records from both tables even if there isn't a match. For an Oracle database this is written like:

FROMA, BWHERE A.field = B.field (+)

And CROSS joins, or Cartesian joins with every possible combination of values from each table (usually a huge mistake where you forgot to specify the join )

WhereIn SQL the WHERE statement specifies the filter criteria (and the joins if not already shown) eg

PPV_SUMMVSEF.DTRATED > TO_DATE ('01-01-2006', 'DD-MM-YYYY') ANDPPV_SUMMVSEF.NEXTCAT >= 'A'

Order byAn optional statement at the end pick the sorting order for your records eg

ORDER BYPPV_PROPERTY_LPG.CORPUPRN ASC,PPV_PROPERTY.INDSTATUS DESC,

A B C

A B C

11

Sharing your Excel Data Query with othersIf you send your finished spreadsheet to someone else, the query will fail to run if their connection to the database is different. That can even happen within the same authority if the original connection was machine-based rather than system-based. But if you create a brand new query on their setup to establish the

correct connection string, copy it and then paste it into the “old” report, it should then run OK.

To find the menu for this connection string, use DATA / PROPERTIES / then click on the Data Properties

icon. Select the DEFINITION tab to get the correct screen for the Connection File location or the Connection String.

Example connection strings:

DRIVER={Oracle in OraClient11g_home1};SERVER=NLP11GL1;UID=smith;;DBQ=NLP11GL1;DBA=W;APA=T;EXC=F;XSM=Default;FEN=T;QTO=T;FRC=10;FDL=10;LOB=T;RST=T;BTD=F;BNF=F;BAM=IfAllSuccessful;NUM=NLS;DPM=F;MTS=T;MDI=Me;CSR=F;FWC=F;FBS=60000;TLO=O;MLD=0;ODA=F;

DRIVER=SQL Server;SERVER=WBCENVIROSQL;UID=SMITH;APP=Microsoft Office 2010;WSID=J6409

Oracle vs SQL ServerWhen attempting to use a file created to interrogate an Oracle version of M3 at a site with a SQL Server version of the database, as well as changing the connection string I had to remove the PPSV. from the FROM statement in the SQL. Also found the function SYSDATE that gives todays date with an Oracle database failed with the SQL server – so I used TODAY() in the Excel file instead (though now read that GETDATE() should work).

Not all queries will be so easy to migrate but most of the fields and tables are common to both versions.

12

More SQL magic (Not all tested – Excel query is sometimes different to the pure SQL code, and that has different flavours for database type)

In these examples square brackets [ ] are not typed but show sections that are optional

To get this… For an Oracle database For a SQL Server database if differentUpper case UPPER(PPV_PROPERTY.TRADNAME)Lower case LOWER(PPV_PROPERTY.TRADNAME)To retrieve all the DIFFERENT values in a table SELECT UNIQUE column_name FROM

table_nameSELECT DISTINCT column_name FROM table_name

To use a complex criteria use AND / OR and brackets to define their combination

WHERE(x > 17 AND (y ='Apple' OR y="Pear)) OR z=42

Find the length of a string LENGTH(PPV_PROPERTY.TRADNAME) LEN(PPV_PROPERTY.TRADNAME)Part of a string eg first 3 characters from the postcode (field, start, length)

SUBSTR(PPV_PROPERTY.POSTCD,1,3)(with Oracle, you can omit the length if you want all the rest of the string)

SUBSTRING(PPV_PROPERTY.POSTCD,1,3)

Show the starting position of a string within a larger string eg the word ‘Royal’ within the trading name. Gives a null value if string is null, a zero if no match or a number eg 5 for the position in ‘The Royal Arms’.

INSTR (PPV_PROPERTY.TRADNAME, ‘Royal’,1,1)With Oracle you can specify to skip x characters or the first y occurrences by using larger digits at the end of the expression.

INSTR (PPV_PROPERTY.TRADNAME, ‘Royal’)

Strip characters from the start of the field, if present

LTRIM(PPV_PROPERTY.TRADNAME, ‘The ‘)

Strip characters from the end of the field, if present

RTRIM(PPV_PROPERTY.TRADNAME, ‘ End‘)

Strip white space(s) from start of string LTRIM(PPV_PROPERTY.TRADNAME)Change one substring to another REPLACE(PPV_PROPERTY.TRADNAME, ‘The ‘, ‘A

‘)Concatenate several string fields(type shifted \ twice)

Column1 || ' ' || Column2 Column1 + ' ' + Column2

Change a numerical field’s value you can use +-/*etc

GRANTS.COST *1.2

Summaries: SUM, AVG, COUNT, MAX, MIN SELECT 'My_label', SUM(column_name) FROM

13

table_nameSummaries for each thing in a group you create SELECT xyz, SUM(column_name) FROM

table_name GROUP BY xyzSummaries based on a condition SELECT xyz, SUM(column_name) FROM

table_name WHERE z IN ('Apple','Pear')Select just the records where a summary meets your condition

SELECT ["column_name1"], function("column_name2")FROM "table_name"[GROUP BY "column_name1"]HAVING (arithmetic function condition);

SELECT Store_Name, SUM(Sales)FROM Store_InformationGROUP BY Store_NameHAVING SUM(Sales) > 1500

Alias: simplify the column names in your output Column_name AS Shorter_nameRounding values ROUND (expression, [decimal place])

ROUND (expression, 1) eg 93.2ROUND (expression,0) eg 93ROUND (expression,-1) eg 90

Convert a string into a date TO_DATE (String,Format)EgTO_DATE(StartDate,'DAY DD MONTH YY')

Add an interval to a date DATEADD(m, 3, StartDate)Other interval values include s, mi, hh, d, wk, q, y

Calculate number of weeks between two dates DATEDIFF (wk, Date1, Date2)Extracts part of a date eg the month DATEPART (m, Date1)The current date and time SYSDATE GETDATE()

14

Stephen Tribbick www.crystal4m3.co.uk August 2015