Testing ofXBRL files for the supervisory reporting

Eurofiling Workshop in Rome

Katrin Heinze

5 May2014

Rubric

www.ecb.europa.eu ©

The objective of this presentation is

to provide an overview on the testing strategy of the SUBA project

to explain the test plan and sample test cases

to inform about the next steps

Page 2Eurofiling Workshop in Rome, 5th of May 2014

Objectives

Rubric

www.ecb.europa.eu ©

1

2

3

Sample XBRL test cases

Next steps

SUBA* test strategy

Overview

Page 3Eurofiling Workshop in Rome, 5th of May 2014

* SUpervisory BAnking Data System

Agenda

Rubric

www.ecb.europa.eu ©

1

2

3

Sample XBRL test cases

Next steps

SUBA Test Strategy

Overview

Page 4

Agenda

Eurofiling Workshop in Rome, 5th of May 2014

Rubric

www.ecb.europa.eu ©

175

28

33

114

Total No run Failed Passed

Functional Tests Non-Functional Tests Regression Tests

Based on User requirements Use cases Experience of users Technical requirements Source code (unit tests) …

Categorised following Load tests (number of users) Performance tests (processing time) Volume test (quantity of data) Stress test Test of the (data) security Usability tests …

Done after changes on Components

o XBRL taxonomyo Validation rules

Interfaceso EXDIo RIAD

Infrastructure …

Technical Test Execution Status17.04.2014

SUBA Test Strategy

Eurofiling Workshop in Rome, 5th of May 2014 Page 5

Rubric

www.ecb.europa.eu ©

HP ALM usage within the SUBA project

Scope

Planning of releases and test cycles Enable traceability between requirements, tests, defects and

according code changes Increase visibility with dashboards on analysis and KPIs

Achievements

Definition of test cases linked to the defined requirements Execution of test cases based on defined test data Defect tracking and documentation of bug fixes

Way forward

Increase the level of automation on test execution Customise reports and analysis based on the project needs Preparation of test cases for the next release

SUBA Test Strategy

Eurofiling Workshop in Rome, 5th of May 2014 Page 6

Rubric

www.ecb.europa.eu ©

1. level: Categories

2. level: Use cases

3. level: Test cases

Conventio

ns on numberin

g

Structure in the Test Lab

Test plan structure

Eurofiling Workshop in Rome, 5th of May 2014 Page 7

Rubric

www.ecb.europa.eu ©

Tracking of the test progress in HP ALM

Eurofiling Workshop in Rome, 5th of May 2014 Page 8

Rubric

www.ecb.europa.eu ©

1

2

3

Sample XBRL test cases

Next steps

SUBA Test Strategy

Overview

Page 9

Agenda

Eurofiling Workshop in Rome, 5th of May 2014

Rubric

www.ecb.europa.eu ©

Positive functional test cases

Creation of test data for each module of the COREP taxonomy which is a valid XBRL instance document.

 Valid according to

XML specification XBRL 2.1 specification XBRL Dimension 1.0 specification XBRL Formula 1.0 specification EBA disabled validation rules EBA Filing Rules for COREP and FINREP

 By using different file sizes

small medium large

Sample XBRL test cases

Eurofiling Workshop in Rome, 5th of May 2014 Page 10

Rubric

www.ecb.europa.eu ©

Negative functional test cases

Tests that ensure that the application shows a proper behaviour when invalid files are received.

Creation of test data where the system is expected to provide correct error messages.

Possible test cases : A reporting entity sends a file which is not a well-formed XML document. has not the correct format, i.e. jpg, txt, xls etc. has not the expected encoding. includes neither facts nor contexts. misses a namespace in the xbrli root element includes letters where numbers are expected as fact value misses an explicit member in the scenario element Includes a fact with no value but also no nil attribute is provided Refers to a non-existing module Refers to two different modules.

Sample XBRL test cases

Eurofiling Workshop in Rome, 5th of May 2014 Page 11

Rubric

www.ecb.europa.eu ©

1

2

3

Sample XBRL test cases

Next steps

SUBA Test Strategy

Overview

Page 12

Agenda

Eurofiling Workshop in Rome, 5th of May 2014

Rubric

www.ecb.europa.eu ©

Test scope

Test period: 13th to 15th of MayTest activities:• Acquisition

NCA sends data to EXDI via WebUI or standard adapter. EXDI system sends a delivery message. SUBA system sends an acknowledgement message to NCA with the XBRL

validation result NCA receives the messages in the scheduled time periods.

Eurofiling Workshop in Rome, 5th of May 2014 Page 13

Next steps

Voluntary functional tests with NCAs

Rubric

Load tests

IT infrastructure for establishing a supervisory function at the ECB

14

Next steps

First estimation about the load expected for the first remittance date

Number of reported entities 128First remittance date NCAs to EBA/ECB 31.07.2014

Modules reported Number of files (per institute)

Max. file size in MB

Max. number of observations

COREP consolidatedOwn Funds 1 115 177.536Large Exposures 1 264 355.930NSFR 1 9 7.510

Total 512 49.664 69.244.928

*) Estimations based on information of the Deutsche Bundesbank for one big German bank

Rubric

Organisation and approach – Test case preparation

Specification workshops

Specification sign-off Implementation

• Development of software• Definition of test cases (HP ALM)

Tests execution

• Tests runs (HP ALM)• Defects workflow (HP ALM)• Bug fixing

Control point

Metric Target (by priority)Average High Medium Low

A % reviewed requirements 100% 100% 100% 100%

B % requirements test coverage 75% 100% 100% 50%

C % tests run 90% 100% 100% 75%

% tests succeeded 100% 100% 100% 100%

Defect resolution time (working days) 3 3 5

Defects reopened ratio 5% 5% 10%

Control point A Control point B Control point C

Eurofiling Workshop in Rome, 5th of May 2014 Page 15

Next steps

Rubric

www.ecb.europa.eu ©

Key issues and challenges

Load tests Defining scenarios for load tests on XBRL processing via automated test, simulating also peaks

Preparation for next releases

Defining test cases based on more precise user requirements

EBA testing Participating in tests scheduled by EBA

Functional tests Supporting the voluntary functional tests with the NCAs, organized by WG SIM

Eurofiling Workshop in Rome, 5th of May 2014 Page 16

Next steps