corporate data warehouse file extract specification chad

NSS IT: File Extract Specification

Version 1.1 of 21 © Common Services Agency, NHS Scotland

National Services Scotland Architecture & Solutions Service

NSS Information Technology

Corporate Data Warehouse

File Extract Specification

CHAD Project

Community Mental Health Data Mart



Document Control

Version Date Author Description

0.1 15/08/2016 Angela Forbes Initial draft

0.2 17/08/2016 Angela Forbes Integrates feedback from Abhishek

0.3 30/08/2016 Angela Forbes Integrates feedback from Elaine – a new SWIFT account specified

and diagnosis added

1.0 08/09/2016 Angela Forbes For sign off.

Additional diagnosis positions added plus related validation and

the default Referral Received Date removed.

1.1 14/02/2017 Angela Forbes Duplicate diagnosis validation rule removed

Supporting Documentation

Document Owner

Business User Requirements Business

Reference Information NSS IT

Functional Requirements NSS IT

Additional Information NSS IT

Universe Structure NSS IT

Business Contacts

Name Role Email Telephone

Elaine McNish Data Manager [email protected] 0141 300 1926

Margaret Parsons Information Manager [email protected] 0131 275 6528

Rosie Lind Principal Information Development

Manager

[email protected] 0131 275 6961

NSS IT Contacts

Name Role Email Telephone

Irene Gow Project Manager [email protected] 0131 275 6209

Angela Forbes Analyst [email protected] 0131 275 6275

Neil Sinclair Technical Lead [email protected] 0131 275 6952

Jill Smith Head of Business Intelligence &

Corporate Applications

[email protected]

0131 275 6788

mailto:[email protected]






Glossary

Acronym/Word Description

BI Business Intelligence.

Business Objects Business Objects is a reporting tool which allows users to create reports which

can be made available to other users who have the appropriate access

permissions.

Business Objects Universe The Business Objects universe is the reporting environment which is built on and

allows access to the associated data mart.

CDD Common Dimensions Database – the database which holds all of the common

dimension tables used by the data marts within the Corporate Data Warehouse.

CDW The Corporate Data Warehouse is a central data store maintained by NSS IT. It

is made up of a collection of independent data marts which share a common

database for reference information.

CHI Community Health Index number – a unique patient identifier which is issued to

residents in Scotland when they register with NHS Scotland.

Data mart A subset of a data warehouse which has been designed to satisfy a particular

function or reporting area within an organisation. A data mart can be designed

and developed independently of other data marts. A data warehouse will contain

multiple data marts.

ETL Extract, Transform and Load - The process of extracting data from external

sources, transforming the data if necessary to meet operational needs and

loading it into the data warehouse.

Fact A fact is a table in a data mart which contains data, usually measures, which can

be reported on. The measures can usually (but not always) be aggregated at a

dimensional level, e.g. a count of patients can be aggregated to give the counts

per location, organisation etc.

SWIFT Submission with internet file transfer

UAS User Access System



Contents

1. Introduction ............................................................................................................................................................ 5

1.1. Purpose ........................................................................................................................................................ 5

1.2. Overview....................................................................................................................................................... 5

1.3. General Data Assumptions .......................................................................................................................... 5

2. File Extracts ........................................................................................................................................................... 5

3. File Submission ..................................................................................................................................................... 6

4. File Processing ...................................................................................................................................................... 6

4.1. File Processing Schedule ............................................................................................................................. 6

5. Validation ............................................................................................................................................................... 6

5.1. Accessing Validation Results ....................................................................................................................... 7

6. Automated Emails .................................................................................................................................................. 7

7. File Format ............................................................................................................................................................. 8

7.1. File Name ..................................................................................................................................................... 8

7.2. Header Record ............................................................................................................................................. 8

7.3. General File Rules ........................................................................................................................................ 9

7.4. Data Element Rules ..................................................................................................................................... 9

7.5. File Content ................................................................................................................................................ 10

8. Appendix A – Automated File Transfer Overview ............................................................................................... 14

8.1. Prerequisites .............................................................................................................................................. 14

8.2. Information Required .................................................................................................................................. 14

8.3. Private Keys ............................................................................................................................................... 14

9. Appendix B – Data Validation Specification ........................................................................................................ 15

10. Appendix C – Sending Locations .................................................................................................................... 21



1. Introduction

1.1. Purpose

The purpose of this document is to detail the file schema and processing rules for files submitted to a Community

Mental Health data mart. The signed off document will be used in conjunction with other supporting documentation

by the BI team to develop the data mart as specified.

This document is intended to give details of the file specification and how the files will be processed into the data

mart. It is not intended to provide recording guidance.

1.2. Overview

Community mental health data will be submitted by the NHS Boards and will be held in the Corporate Data

Warehouse in a Community Mental Health data mart. This will be delivered as part of the second phase of the

Community Health Activity Data (CHAD) project.

1.3. General Data Assumptions

The following general assumptions have been made about the data being submitted

All files will be sent from a single location within a NHS Board, i.e. multiple sites will not be responsible for

submitting information for the NHS Board.

The NHS Board code used in the header record to identify which NHS Board has submitted the data will be

included as a data item in the data mart to identify the NHS Board of Treatment.

A patient can have more than one contact.

Free text fields will not include confidential information.

Unique Record ID

It must be possible to uniquely identify a contact record to avoid duplicate records being processed and to

allow records to be updated if required.

All NHS Boards will be able to provide a system generated Unique Record ID

o The Contact ID will uniquely identify a contact within the NHS Board. The Contact ID will be fixed

for a given contact and will not differ between the original contact and any update to that contact.

This will allow Contact information to be updated.

2. File Extracts

Extract Type Description Number of

Records per file

CMH The file will contain records which have been added or updated

during the quarter.

~ 100,000 per

quarter



NOTE

1. There is no requirement for a separate historic data load. Any historic data submissions will conform to this

specification.

3. File Submission

The files will be submitted using SWIFT or automatic file transfer (see Appendix A).

1. A SWIFT account has been created for the submission of CMH test files. This is still required and a new SWIFT

account is required for submitting files to the data mart.

4. File Processing

The files will be processed using an insert and update process.

The following logic will be applied

If a record is submitted and the Unique Record ID does not exist in the data table the record will be

inserted/added to the data table.

If a record is submitted and the Unique Record ID already exists in the data table it will overwrite the record

already held in full - it is not a selective update so the update record must contain all of the relevant

information.

It is important that the latest record submitted for a Unique Record ID contains all of the relevant information.

Records can be submitted or resubmitted from any historic timeframe if required.

Records cannot be automatically deleted. If a record needs to be deleted a fast track CR will be raised to

have the record removed.

4.1. File Processing Schedule

The automated file load process will run three times daily at 11am, 3pm and 6pm.

5. Validation

The validation rules are specified in Appendix B.

If any records fail a validation rule the file will fail and the error must be corrected and the file resubmitted.

The file will go through two stages of validation as detailed in the table below. For the data validation to begin the

file must pass the initial file validation stage.

Validation

Stage

Error Type Description Outcome of Validation Rule Failure

File File Name

Header

The format of the file is validated, e.g.

file name, header record, number of

The file will fail. The error must be

corrected by the NHS Board and the file



Schema columns. resubmitted.

Data Data The content of the file is validated.

The file will fail. The error must be

corrected by the NHS Board and the file

resubmitted.

NOTE

1. Some of the reference tables contain values which have not been specified as valid in the Community Mental

Health recording guidance. If values are submitted which exist on the reference tables they will be accepted as

valid. If the validation needs to be modified in the future to flag where these values are being used a CR will be

raised.

5.1. Accessing Validation Results

Validation results will be accessed using Business Objects.

6. Automated Emails

The following emails will be generated. The emails can be sent to multiple recipients.

Email Description Recipient

Passed Contains details of the file submitted and that it passed

validation.

NHS Board (email address(s) to be

provided).

CHAD support mailbox

[email protected]

ISD Data Management mailbox

[email protected]

Failed Contains details of the file submitted.

The number of records processed.

The number of records which passed validation.

The number of records which failed validation.

If the file has failed totally it will include an indication of

why the file could not be processed.


provided).


[email protected]


[email protected]

Loaded The data will not be available immediately after

validation because of the processing required. This

email will advise when the data are available for

analysis.


provided).


[email protected]


[email protected]









7. File Format

The data extract(s) will follow the general standards.

7.1. File Name

The file will fail validation if the file name does not conform to the following format.

DATAMART_SENDING LOCATION_EXTRACT DATE. EXTENSION

e.g. CMH_N_20160114.csv

Where:

Item Description Example

Data Mart Identifies the data mart which the files are submitted to CMH

Sending Location The cipher for the sending location e.g. N = Grampian

See Appendix C for valid sending locations.

N

Extract Date The date the file was compiled. Date format: YYYYMMDD 20160114

File extension Identifies the type of file, must be a .csv .csv

NOTE

1. Ciphers have been used to identify the sending location to be consistent with what is submitted for the CHAD

District Nursing data mart. The latest 9 digit code for the NHS Board will be available for reporting.

7.2. Header Record

The first row in the file must contain a header record which contains details of the file. All data items in the header

record are mandatory and must be in the specified format. The information in the header record should be

consistent with the information supplied in the file name.

The file will fail validation if the header format does not conform to the following comma delimited format.

DATA MART,SENDING LOCATION,EXTRACT DATE,SUBMISSION REFERENCE NUMBER,RECORD COUNT

e.g. CMH,N,20160114,1,12100

Where:

Item Format Description Example

Data Mart Varchar (3) Identifies the data mart which the files are

submitted to

CMH

Sending Location Varchar (1) The cipher for the sending NHS Board e.g. N =

Grampian.

See Appendix C for valid sending locations.

N

Extract Date Varchar (8) The date the file was compiled. Date format:

YYYYMMDD

20160114

Submission

Reference Number

Integer (6) A unique number identifying the number of the

submission for the sending location.

1



Record Count Integer A count of the number of records in the file

(excluding the header record and column names).

12100

NOTE

1. The header record is not case sensitive.

2. The Submission Reference Number is required to allow multiple files for the same location to be processed in the

correct order, and to flag if there are any gaps in the received files. The Submission Reference Number should

increment per successful validation for each sending location. Files which are invalid and rejected for any reason

must be corrected and resubmitted with the same Submission Reference Number as the original.

3. There should be no characters (including commas) after the last item.

7.3. General File Rules

The following table include the general rules which apply to the file format

Category Description

Columns The columns are comma delimited to separate fields and must be in the order specified

in the file specification. If a field is empty the column must still be included in the

extract.

Column Names The file will contain column name headings. The headings are for local use only and

will be ignored when the file is loaded. The name can only contain a comma when

surrounded by double quotes.

Column Width The column width is the minimum width from a development perspective. The data

mart will be developed to process columns with a width wider than documented to

ensure that the field is not truncated to a valid value.

Blank Rows Blank data rows must not be included in the data file.

Null/Missing Values Empty fields will indicate where a value is null (missing). Zero must not be used.

Spaces (non-

printable values)

Spaces should not be present between fields. The comma alone should delimit fields.

Spaces will be trimmed where applicable.

7.4. Data Element Rules

The following table includes the data element rules which apply to the data in the file

Category Description

Number Type Numeric data items will appear in the text file as a string of continuous numeric digits.

For large numbers the ‘thousand’ display format characters such as the comma will not

be included. Where the number contains decimals, the period character ‘.’ will be used

as the decimal point.

Text/Character Type The text string will contain any alphanumeric character including the space character

other than double quotes. The format of the text field will be defined as ‘Varchar (n)’

where Varchar (n) is a variable length string of maximum length ‘n’.



Free text fields should be qualified/surrounded by double quotes.

Codes and descriptions will be converted to upper case for all data items.

Date Type A column containing date strings will be formatted as DDMMYYYY (e.g. 14012016).

If a 7 digit date is submitted it will be left zero padded to 8 digits.

Time Type A column containing date time strings will be formatted as HH:MM using the 24 hour

clock (e.g. 17:35).

Leading Zeros Leading zeros must not be stripped off codes which have been identified as having

leading zeros.

If a 9 digit CHI number is submitted it will be left zero padded to 10 digits.

Data Item Priorities The priority categories for including data items are as follows.

M = Mandatory the data item must be populated

S = Should be populated if available

O = Optional

7.5. File Content

The table below gives details of each data item in the order it will appear in the file.

Data Item Description Format Priority

Contact ID Identifier used within the NHS Board to uniquely identify a

contact.

Varchar (100) M

CHI Number The identifier used by NHS Scotland to uniquely identify a

patient.

9 digit codes will be left zero padded to 10 digits.

Varchar (10) S

Surname The surname of the patient.

Required for CHI seeding.

Must be provided if a valid CHI Number is not provided.

Varchar (35) S

Forename The first name of the patient.

Required for CHI seeding.


Varchar (35) S

Date of Birth The patient’s full date of birth.

Required for CHI seeding and will be used to derive age where a

valid CHI number is not available.

Date S




Postcode The patient’s postcode of residence.

Required for CHI seeding and will be used to derive the area of

residence where a valid CHI number or associated postcode is

not available.


Varchar (8) S

Gender The gender of the patient, e.g. 1 = Male, 2 = Female.

Required for CHI seeding and to populate gender where a valid

CHI number is not available.


Varchar (1) S

Ethnicity The patient’s ethnic group. Varchar (2) S

Referral Received

Date

The date the health and/or social care service received a

referral.

Date

S

Source of Referral The source of referral. Varchar (1) O

Referral Urgency A judgement made by the relevant receiving team regarding how

quickly a patient/client needs to be seen when referred.

Varchar (1) O

Contact Date The date the contact between a health and/or social care

professional and the patient/client took place.

Date M

Contact Start

Time

The time the contact between a health and/or social care

professional and the patient/client began.

Time S

Contact End Time The time the contact between a health and/or social care

professional and the patient/client ended.

Time S

Duration of

Contact

The length of time contact occurred between the health and or

social care professional and the patient in minutes.

Must be completed if Contact Start Time and Contact End Time

are not recorded.

Varchar (5) S

Location of

Contact

The type of location where the contact occurred, e.g. Health

Centre.

Varchar (1) S

Contact Category Indicates if the patient/cl ient contact with the health and/or

social care provider was direct or indirect.

Varchar (1) S

Contact Type Indicates if the patient/client contact took place in an individual,

group or family setting.

Varchar (1) S

Joint Contact Indicates if two or more staff members were required for the

patient contact.

The value can be 0 or 1 where 1 indicates that joint contact was

required.

Varchar (1) S

Number of Staff The number of staff members in attendance during a joint visit. Varchar (1) S



Present in Joint

Contact

Contact Status Indicates if the patient/client attended/was actually seen at the

contact.

Varchar (1) S

Duration of

Patient/Client

Related Activity

The length of time in minutes taken by staff to carry out activities

which are client related but do not involve contact, e.g.

administration.

Varchar (5) O

Travel Time The number of minutes taken to travel by the staff member to

the location of contact.

Varchar (5) S

Staff Pay Band The current staff band as featured on the staff payslip. Varchar (2) S

Service Team The name of the lead team responsible for delivering the

service.

Free text.

Varchar (100) S

Specialty The main specialty/discipline for the contact. Varchar (3) S

Main Aim of

Contact

A desired achievement of a programme of care e.g. the primary

reason for a referral to a Community Health Care Professional.

Varchar (1) S

Other Aim of

Contact (1)

Other desired achievement of a programme of care, but not

deemed to be the primary desired achievement.

Varchar (1) O

Other Aim of

Contact (2)



Varchar (1) O

Other Aim of

Contact (3)



Varchar (1) O

Other Aim of

Contact (4)



Varchar (1) O

Diagnosis (1) The ICD10 code of the diagnosis in position 1. Varchar (6) O






NOTE

1. The recording guidance states that CHI number is mandatory however if it cannot be provided the demographic

information must be submitted and this will be used for CHI seeding.

2. 7 digit dates will be left zero padded to 8 digits.

3. Contact End Date is not included and it will be assumed that the contact will start and end on the same date.

4. A combination of the Sending Location and Contact ID will uniquely identify a record in the data mart.

5. Valid codes are specified in the reference information document for the following data items - Gender, Ethnicity,

Source of Referral, Referral Urgency, Location of Contact, Contact Category, Contact Type, Contact Status, Staff

Pay Band, Specialty, Aim of Contact. A sample of the Diagnosis reference information is also included.

6. The ICD10 code submitted can contain up to 6 characters. 3, 4 and 6 character codes and descriptions will be

included in the Business Objects universe.



7. The Diagnosis fields must be included in the file format, but they are not yet part of the agreed dataset and can

therefore be null. Users will be advised when these fields should start to be populated.



8. Appendix A – Automated File Transfer Overview

The National Services Scotland Information Technology (NSS IT) strategic business unit has a facility in place

which allows external data suppliers to submit data to be loaded into the Corporate Data Warehouse automatically.

The transfer mechanism uses Secure Copy (SCP) protocol. A username and private key is passed to the data

supplier and these details can then be incorporated into a batch file or UNIX script which can be scheduled to run at

required intervals.

8.1. Prerequisites

If the data supplier is using a UNIX server to submit the data, SCP must be installed on the server. The data

supplier server administrator should be able to implement this if it is not already installed. If Windows is used then

pscp must be installed on the server. This can be downloaded for free from

http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html.

8.2. Information Required

The following information is required by the Business Intelligence team to request that an account is created.

The name, telephone number and email address of the server administrator who will be implementing the

automation script for the data transfer.

The data supplier who the request is for.

The data mart the data will be submitted to.

The operating system the data will be submitted from (i.e. UNIX or Windows).

8.3. Private Keys

Once the account has been set up by the UNIX team the data provider’s server administrator will be sent an email

which will contain the following information.

Production environment private key and username

Test environment private key and username

Server name

A one line example of how a script can be used to transfer the data which can be implemented into an

automated cron job or similar

http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html



9. Appendix B – Data Validation Specification

The table below includes details of the File Validation and Data Validation Errors.

If a validation rules needs to be turned off a Change Request will be raised.

Error

Code

Error Type Error

Severity

Error Message Additional Information

FE1 File Name Error Invalid File Name: Data mart is missing

or not recognised

Must be CMH

FE2 File Name Error Invalid File Name: Sending location is

missing or not recognised

Must be one of A, B, F, G, H, L,

N, R, S, T, V, W, Y and Z.

FE3 File Name Error Invalid File Name: Extract Date is

missing or invalid. Should be a date in

the format YYYYMMDD

HE1 Header Error Invalid Header Record: Could not be

read – check format

HE2 Header Error Invalid Header Record: Data mart is

missing, not recognised or does not

match the file name

HE3 Header Error Invalid Header Record: Sending

Location is missing, not recognised or

does not match the file name

HE4 Header Error Invalid Header Record: Extract Date is

missing, invalid or does not match the

file name. Should be a date in the

format YYYYMMDD.

HE5 Header Error Invalid Header Record: The Submission

Reference supplied is out of sequence

for the supplied Sending Location

HE6 Header Error Invalid Header Record: The number of

records is inconsistent with the number

of records submitted

SE1 Schema Error Data record could not be read – check

file format

SE2 Schema Error Contact ID must be provided and must

be unique within the file

VE1 Data Error CHI number must be valid Conditional check only applied if

CHI number has been provided.

9-digit codes will be left zero

padded to 10 digits.

Must exist on the relevant

common dimension.



VE2 Data Error Surname must be provided and in the

correct format

Conditional check only applied if

CHI number has not been

provided.

The following criteria must be met

The first character must be

alphabetic.

Can contain alphabetic

characters, a hyphen, a

space or an apostrophe.

Must be a minimum of 2

characters.

Maximum field length 35

VE3 Data Error Forename must be provided and in the

correct format



provided.

The following criteria must be met

The first character must be

alphabetic.

Can contain alphabetic

characters, a hyphen, a

space or an apostrophe.

Must be a minimum of 2

characters.

Maximum field length 35

VE4 Data Error Date of Birth must be provided and

must be a valid date in the format

DDMMCCYY



provided.


common dimension.

VE5 Data Error Postcode must be provided and must

be valid



provided.


common dimension.

VE6 Data Error Gender must be provided and must be

valid



provided.


common dimension.



VE7 Data Error Ethnicity must be valid Conditional check only applied if

Ethnicity has been provided.


common dimension.

VE8 Data Error Date Referral Received must be a valid

date in the format DDMMCCYY


Date Referral Received has been

provided.


common dimension.

VE9 Data Error Source of Referral must be valid Conditional check only applied if

Source of Referral has been

provided.


common dimension.

VE10 Data Error Referral Urgency must be valid Conditional check only applied if

Referral Urgency has been

provided.


common dimension.

VE11 Data Error Contact Date must be provided and

must be a valid date in the format

DDMMCCYY

Contact Date is mandatory.


common dimension.

VE12 Data Error Contact Start Time must be a valid time

in the format HH:MM


Contact Start Time has been

provided.


common dimension.

VE13 Data Error Contact End Time must be a valid time

in the format HH:MM


Contact End Time has been

provided.


common dimension.

VE14 Data Error Duration of Contact must be numeric Conditional check only applied if

Duration of Contact has been

provided.

VE15 Data Error Location of Contact must be valid Conditional check only applied if

Location of Contact has been



provided.


common dimension.

VE16 Data Error Contact Category must be valid Conditional check only applied if

Contact Category has been

provided.


common dimension.

VE17 Data Error Contact Type must be valid Conditional check only applied if

Contact Type has been provided.


common dimension.

VE18 Data Error Joint Contact must be 0 or 1 Conditional check only applied if

Joint Contact has been provided.

VE19 Data Error Number of Staff Present in Joint

Contact must be numeric


Number of Staff Present in Joint

Contact has been provided.

Must be a whole number.

VE20 Data Error Contact Status must be valid Conditional check only applied if

Contact Status has been

provided.


common dimension.

VE21 Data Error Duration of Patient/Client Related

Activity must be numeric


Duration of Patient/Client Related

Activity has been provided.

VE22 Data Error Travel Time must be numeric Conditional check only applied if

Travel Time has been provided.

VE23 Data Error Staff Pay Band must be valid Conditional check only applied if

Staff Pay Band has been

provided.

VE24 Data Error Specialty must be valid Conditional check only applied if

Specialty has been provided.

VE25 Data Error Main Aim of Contact must be valid Conditional check only applied if

Main Aim of Contact has been

provided.


common dimension.



VE26 Data Error Other Aim of Contact (1) must be valid Conditional check only applied if

Other Aim of Contact (1) has

been provided.


common dimension.



been provided.


common dimension.



been provided.


common dimension.



been provided.


common dimension.

VE30 Data Error Diagnosis (1) must be valid Conditional check only applied if

Diagnosis (1) has been provided.


common dimension.




common dimension.




common dimension.




common dimension.






common dimension.




common dimension.



10. Appendix C – Sending Locations

NHS Board Sending Location Code

Ayrshire & Arran A

Borders B

Dumfries & Galloway Y

Fife F

Forth Valley V

Grampian N

Greater Glasgow and Clyde G

Highland H

Lanarkshire L

Lothian S

Orkney R

Shetland Z

Tayside T

Western Isles W

corporate data warehouse file extract specification chad

Documents