automated multiplatform compilation and validation of a collaborative repository of examples using...

TRABAJO FIN DE MÁSTER: AUTOMATED MULTIPLATFORM COMPILATION AND VALIDATION OF A COLLABORATIVE REPOSITORY OF

EXAMPLES USING RTI CONNEXT DDS

Author: Ángel Jesús Martínez Bernal Advisor: María Sara Granados Cabeza Granada, December 2014

Content Introduction

Rationale State of the Art

Problem definition

Goals

Development Common Example Structure

Examples Repository

Automation Task

Results

Conclusions

Content Introduction

Rationale State of the Art

Problem definition

Goals

Development Common Example Structure

Examples Repository

Automation Task

Results

Conclusions

Introduction (I) Automation

Automation in Computer Science

Programming Languages

Punched cards

Assembly programming language

Different paradigms (functional programming, logic programming…)

Oriented Object Programming

Scripting

GPU

Introduction (I) Automation

Automation in Computer Science

Programming Languages

Punched cards

Assembly programming language

Different paradigms (functional programming, logic programming…)

Oriented Object Programming

Scripting

GPU

Introduction (II) Continuous Integration

Revision control -> Repositories

Content Introduction

Rationale State of the Art

Problem definition

Goals

Development Common Example Structure

Examples Repository

Automation Task

Results

Conclusions

Rationale - State of the Art (I) Distributed System Communication

Client/Server Paradigm

Rationale - State of the Art (II) Publish/Subscribe Paradigm

Rationale - State of the Art (III) Data Distribution Service (DDS)

OMG Standard

Features:

QoS-enabled

Data-centric

High scalability

More dynamic network topology

Rationale - State of the Art (IV) Data Distribution Service (DDS)

Rationale - State of the Art (V) RTI Connext DDS

Fully compliant with the OMG DDS standard, API and RTPS protocol

Interoperable: languages, operating systems and network types

Low latency and high throughput

Deterministic even under load and at scale

Reliable multicast

Peer-to-peer architecture: no brokers or servers

Non-stop availability.

Different approaches depending on the system.

Rationale - State of the Art (VI)

DDS Bus / RTI DataBus™

Connext

Micro

Pub/Sub API

Small Device Apps

Connext

DDS

Pub/Sub API

DDS Apps

Connext

Messaging

Messaging API

General-Purpose Real-Time Apps

Connext

Integrator

Adapters

Discrete OT & IT Apps/Systems

Administration

Monitoring

Logging

Recording

Replay

Federation

Transformation

Persistence Visualization

Rationale - State of the Art (VII) Continuous Integration (Bamboo)

Local agent

Remote agent

Virtual agent

Content Introduction

Rationale State of the Art

Problem definition

Goals

Development Common Example Structure

Examples Repository

Automation Task

Results

Conclusions

Problem Definition (I) RTI’s user community unmaintainable

Out-of-date examples

Not compiling

Not working in latest versions

Using deprecated features

Only available for one language or platform

Many collaborators Different patterns

More time understanding the example than the feature.

No continuous integration!

Problem Definition (II) QoS used to be

changed programmatically.

Now, XML files are recommended.

We needed to commented them out and create XML profiles. E.G.: Asynchronous

publication:

Problem Definition (III) Use USER_QOS_PROFILES.xml rather than code.

Problem Definition (IV) Needed an automation process able to:

Manage the copyright header

Not necessary for external examples

Build all examples in different platforms

Validate QoS files (XML extension)

Manage <dds> tag

Bundle examples for uploading to a website

Solution needs to be scalable and maintainable.

Content Introduction

Rationale State of the Art

Problem definition

Goals

Development Common Example Structure

Examples Repository

Automation Task

Results

Conclusions

Goals (I) Complete & update the RTI Connext DDS

example repository

Refactor examples to follow a common same pattern

Add new examples

Integrate the examples in the Continuous Integration methodology:

Automated process

Create configurable scripts for scalability

Goals (II) Initial state of the repository

Content Introduction

Rationale State of the Art

Problem definition

Goals

Development Common Example Structure

Examples Repository

Automation Task

Results

Conclusions

Common Example Structure (I) Example Git repository:

https://github.com/rticommunity/rticonnextdds-examples

Outside structure: Example directory:

README.txt /C

README.txt Foo_publisher.c Foo_subscriber.c USER_QOS_PROFILES.xml Foo.idl

/C++ /Java /CS

Create participant

Create publisher

Get type_name

Create topic

Create DataWriter

Narrow DataWriter

Create Instance

Delete data

Common Example Structure (II) Publisher

Modify data

(instance)

Write the data

(instance)

Sleep

Create participant

Create publisher

Get type_name

Create topic

Create DataWriter

Narrow DataWriter

Create Instance

Delete data

Common Example Structure (II)

Modify data

(instance)

Write the data

(instance)

Sleep

Publisher

Create Participant

Create Subscriber

Get type_name

Create topic Create

Listener Create

DataReader

Delete data

Subscriber

Narrow DataReader

Read sample Print the sample

On_data_available

Common Example Structure (III)

Create Participant

Create Subscriber

Get type_name

Create topic Create

Listener Create

DataReader

Delete data

Subscriber

Narrow DataReader

Read sample Print the sample

On_data_available

Common Example Structure (III)

Create Participant

Create Subscriber

Get type_name

Create topic Create

Listener Create

DataReader

Delete data

Subscriber

Narrow DataReader

Read sample Print the sample

On_data_available

Common Example Structure (III)

Create Participant

Create Subscriber

Get type_name

Create topic Create

Listener Create

DataReader

Delete data

Subscriber

Narrow DataReader

Read sample

Print the sample

Common Example Structure (III)

Narrow DataReader

Create Conditions

Create WaitSet

Attach conditions

Create Participant

Create Subscriber

Get type_name

Create topic Create

DataReader Create

Waitset*

Loop: Wait data / printing

Delete Data

Additional Steps WaitSets

Automation process Tasks:

Manage Copyright notice (Check, Add & Delete)

Automated building for different architectures

Windows, Linux or other supported by the tools.

XML validation

Point to Community XSD file (dds tag).

Update DDS version & schema.

Bundle examples

Tools: Bamboo and Perl scripts

Manage Copyright Notice (I) Copies the copyright header to the files under a

folder.

Available for XML and C/C++.

Configurable:

Decide the extensions of the files to process.

Select an option (option_flag): check/copy/delete.

Skip folders or files (e.g.: external examples).

Call syntax: perl ./resources/scripts/copyright.pl <working_directory> <option_flag> <extensions>...

Manage Copyright Notice (II) C Style Copyright Header: Top of the file

Manage Copyright Notice (III) XML Style Copyright Header: after XML definition

Multiplatform Building Call syntax perl ./resources/scripts/compile_[unix|windows].pl <working_directory> <ndds_version> <architecture_c>

Look for IDL files

Call rtiddsgen

Call the compiler

XML Validation Validates all the QoS files with

its corresponding schema version (.xsd).

Call syntax: perl ./resources/scripts/xml_validation.pl <working_directory> <ndds_version>

Validate!

(or error)

xmllint

XML file

URL of the .xsd

file

XML Update Schema location should NOT be a local path, but an

online one:

Three different actions (option_flag): Check whether the <dds> tag has all the 3 attributes.

Add the attributes that the <dds> tag did not have.

Replace the attributes <path_schema> and <ndds_version>.

Call syntax: perl xml_update.pl <working_directory> <option_flag> <ndds_version> <path_schema>

Bundle Examples (I) Bundle the examples in order to upload to a website

Create a .zip & .tar.gz which contains all the examples

Bundle every example separately

Upload all packages & files to a website

Call syntax: perl bundle_examples.pl <working_directory>

<repository_name>

Bundle Examples (II) rticonnextdds-examples

asynchronous_publication

C++

C

Cs

Java

Readme.txt

Batching

builtin_qos_profiles

builtin_topics

coherent_presentation

…

.zip & .tar.gz

.zip & .tar.gz

Bamboo Integration (I) Building steps:

Copyright

XML validation

Compile

Bamboo Integration (II) Shipping steps:

Bamboo Integration Script configuration as Bamboo task

Bamboo Integration (III)

Bamboo Integration (IV)

Content Introduction

Rationale State of Art

Problem definition

Goals

Development Common Example Structure

Examples Repository

Automation Tasks

Results

Conclusions

Results (I) Contribution to the repository

Results (II)

Available soon

Results (III) New examples

Content Introduction

Rationale State of the Art

Problem definition

Goals

Development Common Example Structure

Examples Repository

Automation Task

Results

Conclusions

Conclusions RTI Connext DDS

Easy to create p/s applicatons

Configurable thanks to QoS policies

Big community that needs CI

New Examples

Continuous integration Allow projects to grow easily

Scripts for high scalability

Integrating with different Atlassian’s tools. For instance: JIRA

Future Work Integrate more feature examples

Add the custom content filter example in C#

Evaluate interesting features and concept and create new examples

Update the shipped examples:

Force them to have the same structure

Use a subset of the RTI Community examples repository as shipped examples.

Compile for more platforms

VxWorks, Raspberry Pi, Integrity, AIX, …

TRABAJO FIN DE MÁSTER: AUTOMATED MULTIPLATFORM COMPILATION AND VALIDATION OF A COLLABORATIVE REPOSITORY OF

EXAMPLES USING RTI CONNEXT DDS

Author: Ángel Jesús Martínez Bernal Advisor: María Sara Granados Cabeza Granada, December 2014