non-functional testing at london stock exchange
TRANSCRIPT
![Page 1: Non-Functional Testing at London Stock Exchange](https://reader036.vdocuments.net/reader036/viewer/2022070512/5889c13c1a28abca448b5109/html5/thumbnails/1.jpg)
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 1
Non-Functional Testing at London Stock Exchange29 November 2016Iosif Itkin, Exactpro CEO & Co-FounderLondon Stock Exchange Group
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 2460, +1 415 830 38 49www.exactpro.com
![Page 2: Non-Functional Testing at London Stock Exchange](https://reader036.vdocuments.net/reader036/viewer/2022070512/5889c13c1a28abca448b5109/html5/thumbnails/2.jpg)
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 2
London Stock Exchange Group
Exactpro - Open Access Software Testing
Testing Execution Venues
Efficient Load Generation
Latency Measurement
Non-Functional Testing Principles
Other Ways to Produce Load
Approach to Test Automation
Failover Testing Approach
Passive Software Testing
Research in Tools and Methods of Program Analysis
ContentsContents
![Page 3: Non-Functional Testing at London Stock Exchange](https://reader036.vdocuments.net/reader036/viewer/2022070512/5889c13c1a28abca448b5109/html5/thumbnails/3.jpg)
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 3
London Stock Exchange Group
• A leading diversified international exchange and infrastructure group with assets across the entire exchange value chain
• The Group can trace its history back to 1698• Headquartered in the United Kingdom, with significant global operations, the Group employs approximately 3,500
people• Post trade and risk management services are a significant part of the Group’s business operations• Through FTSE Russell, the Group is a global leader in financial indexing, benchmarking and analytic services with
approximately $10 trillion benchmarked to its indexes
![Page 4: Non-Functional Testing at London Stock Exchange](https://reader036.vdocuments.net/reader036/viewer/2022070512/5889c13c1a28abca448b5109/html5/thumbnails/4.jpg)
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 4
Open Access Software Testing
Exactpro is:
• A specialist firm operating an open access model with focus on functional and operational testing of securities data distribution, trading systems, risk management, market surveillance and post-trade infrastructures.
• A UK company with operations in the US and four QA & software development centres in Russia
• Part of London Stock Exchange Group as of May 29, 2015
• Incorporated in 2009 with 10 people, our company has experienced significant growth as satisfied clients require
more services; now employing over 420 specialists
Clients Worldwide:
![Page 5: Non-Functional Testing at London Stock Exchange](https://reader036.vdocuments.net/reader036/viewer/2022070512/5889c13c1a28abca448b5109/html5/thumbnails/5.jpg)
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 5
Open Access Software Testing
Toronto
New YorkTysons Corner
Montreal
Portland
Seattle
Dubai
Colombo
Johannesburg
Sydney
Singapore
Hong Kong
Moscow
Oslo
ParisMilan
Athens
Vienna
London Amsterdam
A specialist firm operating an open access model with focus on functional and operational testing for global financial markets worldwide.
A UK company with operations in the US and four QA & software development centres in Russia
Part of LSEG Since May 29, 2015
Incorporated in 2009 with 10 people, our company has experienced significant growth as satisfied clients require more services;
Now employing over 400 specialists
![Page 6: Non-Functional Testing at London Stock Exchange](https://reader036.vdocuments.net/reader036/viewer/2022070512/5889c13c1a28abca448b5109/html5/thumbnails/6.jpg)
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 6
Types of Trading Systems
Algo TradingHFT
Brokerageand IDB
Quality Assurance:test planning and
managementProcess audit and test
coverage analysis
Intelligent Management
of Large Data Sets
Test data management
Protocol level testing using FIX/FAST, SOAP, HTTP, ITCH, SWIFT, MQ, SQL, proprietary
binary and text baseddata formats, etc.
Automated regressiontesting
Gathering requirementsand test scenario creation
(human, message & reporting interfaces)
Creating and productizing state-of-the-arttest harnesses
Test automation
Intelligent functional andexploratory testing
Latency & capacitytesting
Market Infrastructures
![Page 7: Non-Functional Testing at London Stock Exchange](https://reader036.vdocuments.net/reader036/viewer/2022070512/5889c13c1a28abca448b5109/html5/thumbnails/7.jpg)
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 7
Execution Venues Testing
Our experience in market infrastructures testing embraces all components and operational processes. Unique tools have been used to test a wide variety of vendor platforms. A holistic integrated automated test solution for trading, clearing, settlement and other systems
![Page 8: Non-Functional Testing at London Stock Exchange](https://reader036.vdocuments.net/reader036/viewer/2022070512/5889c13c1a28abca448b5109/html5/thumbnails/8.jpg)
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 8
Performance Testing Cost Structure
System Under Test
ProductionConnectivity
Test Tools
Infrastructure: ≈ 10 - 100 servers
Infrastructure: thousands of servers
Infrastructure: 1 - 4 servers
![Page 9: Non-Functional Testing at London Stock Exchange](https://reader036.vdocuments.net/reader036/viewer/2022070512/5889c13c1a28abca448b5109/html5/thumbnails/9.jpg)
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 9
The Idea Behind Efficient Load Generation
![Page 10: Non-Functional Testing at London Stock Exchange](https://reader036.vdocuments.net/reader036/viewer/2022070512/5889c13c1a28abca448b5109/html5/thumbnails/10.jpg)
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 10
The Idea Behind Efficient Load Generation
![Page 11: Non-Functional Testing at London Stock Exchange](https://reader036.vdocuments.net/reader036/viewer/2022070512/5889c13c1a28abca448b5109/html5/thumbnails/11.jpg)
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 11
Latency Measurement
![Page 12: Non-Functional Testing at London Stock Exchange](https://reader036.vdocuments.net/reader036/viewer/2022070512/5889c13c1a28abca448b5109/html5/thumbnails/12.jpg)
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 12
Latency Distribution
![Page 13: Non-Functional Testing at London Stock Exchange](https://reader036.vdocuments.net/reader036/viewer/2022070512/5889c13c1a28abca448b5109/html5/thumbnails/13.jpg)
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 13
Question Everything
Do Not Trust:• Test Tools
• Average Metrics and Percentiles
• That Load Shape is Right
• System Requirements
• Anything
![Page 14: Non-Functional Testing at London Stock Exchange](https://reader036.vdocuments.net/reader036/viewer/2022070512/5889c13c1a28abca448b5109/html5/thumbnails/14.jpg)
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 14
Keep Calm
Have No Fear:• Annihilate the System Under Test
• Repeat the Same Tests Many Times
• Load Levels that Exceed Presumed Production
• Make no Assumptions About
Extrapolating Test Results into
More Scalable Environments
![Page 15: Non-Functional Testing at London Stock Exchange](https://reader036.vdocuments.net/reader036/viewer/2022070512/5889c13c1a28abca448b5109/html5/thumbnails/15.jpg)
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 15
Be Confident
Do Not Beg For:• Test Tools
• Confirmation that PROD is Configured as TEST
• Max Load Figure
• Load Scenarios for Existing Production System
• Constant Support
![Page 16: Non-Functional Testing at London Stock Exchange](https://reader036.vdocuments.net/reader036/viewer/2022070512/5889c13c1a28abca448b5109/html5/thumbnails/16.jpg)
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 16
Two Load Generator Types
1.
2. Simulate real trading participants
Adopt to changing market conditions
Check data consistency from member’s point of view
![Page 17: Non-Functional Testing at London Stock Exchange](https://reader036.vdocuments.net/reader036/viewer/2022070512/5889c13c1a28abca448b5109/html5/thumbnails/17.jpg)
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 17
Achieving Diverse and Relevant Load
1. Random load “Pace Maker” 2. Variety of Passive Liquidity
3. “Aggressors” try to interact with the market to increase the number of test cases happening in the environment
4. Analyze the situation in retrospect. Make sure that what happened is correct. Tick checkboxes against tests that were actually executed
Updated test
library
![Page 18: Non-Functional Testing at London Stock Exchange](https://reader036.vdocuments.net/reader036/viewer/2022070512/5889c13c1a28abca448b5109/html5/thumbnails/18.jpg)
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 18
Types of Non-Functional Tests
![Page 19: Non-Functional Testing at London Stock Exchange](https://reader036.vdocuments.net/reader036/viewer/2022070512/5889c13c1a28abca448b5109/html5/thumbnails/19.jpg)
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 19
Failover Tests and Catching In-Flight Transactions
![Page 20: Non-Functional Testing at London Stock Exchange](https://reader036.vdocuments.net/reader036/viewer/2022070512/5889c13c1a28abca448b5109/html5/thumbnails/20.jpg)
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 20
Use Passive Testing Tools to Collect Audit Trails
![Page 21: Non-Functional Testing at London Stock Exchange](https://reader036.vdocuments.net/reader036/viewer/2022070512/5889c13c1a28abca448b5109/html5/thumbnails/21.jpg)
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 21
Questions & Answers
• Competing conference papers go through a review by a number of leading software development and analysis experts and scientists
• The best papers are recommended for their inclusion in Russia’s leading professional and scientific publications
The Fourth TMPA Conference Find out more: www.tmpaconf.org
Renowned experts in program engineering speaking on:
• software test automation• static program analysis• verification• dynamic methods of program analysis• analysis of parallel and distributed systems• testing and analysis of high-availability systems• analysis of hardware and software systems• methods of building quality software• tools for software analysis, testing and verification
![Page 22: Non-Functional Testing at London Stock Exchange](https://reader036.vdocuments.net/reader036/viewer/2022070512/5889c13c1a28abca448b5109/html5/thumbnails/22.jpg)
Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 22
Questions & Answers
https://ru.linkedin.com/in/iosifitkinThe seventh EXTENT conference will take place in London, UK.
Find out more: www.extentconf.com
– LSEG Technology and Quality Assurance– Risk controls and FPGA– Market Surveillance systems– Blockchain and trading technology trends– Artificial intelligence