SEA-120 Nov 2014

CCSDS System EngineeringArea (SEA): XSG – XML Standards & Guidelines

Peter Shames, SEA AD

20 Nov 2014

SEA-220 Nov 2014

Motivation for this XML Guidelines Project

• CCSDS currently has several different XML based data formats:• DEDSL DTD• NAV NDM XML• SM Service Package• SM&C XTCE & others• SOIS XTEDS• Apologies if I missed any … and others are coming

• These have been developed during the last 8-12 years of CCSDS standards, and they were largely in disjoint domains

• As CCSDS has grown there are now interfaces and overlaps both in content and in terminology, but no standards for namespaces, common terms, or schema construction, nor defined means for establishing and maintaining consistency

• This project proposes to tackle this issue head on by:• Adopt or develop an agreed set of guidelines for XML standards• Derive initial ontology terms from the existing XML (type) schema• Identifying common schema for QUDV and related topics• Using the new common core Ontology and domain extensions for all terms• Doing the work to regularize and formalize the use of terms • Propose, as needed, other new registries in SANA, values, units• Propose a process for managing the evolution and versioning of the set of XML

schemas in the future• Working any issues that arise with the affected WGs

SEA-3

More on XML Issues

• There is no standard for construction and use of names, common terms, or references

• There are no commonly defined registries for: quantities, units, time, and value terminology, agencies, ground stations & antennas, frequency bands, and other common identifiers

• The XML formats may be DTD, or Schema, or other forms, they may be qualified or unqualified

• We are finally establishing CCSDS namespace standard and do have an XML repository in the SANA, but this is just a beginning

20 Nov 2014

SEA-4

Example: Nav NDM XML Schema• Based on existing CCSDS Nav ASCII document

formats• All element names are upper case

• Uses XML Schema and qualified namespace• Little in-line documentation, no in-line constraint

checking• Defines a set of NAVWG common terms and a set of

“CCSDS” common terms in separate schema• ndmxml-R1.5-ccsds-common• ndmxml-R1.5-navwg-common

• Several different schema used to define TDM, OEM, OPM, ADM, AEM, APM document types

• NOTE: CCSDS common terms could be merged with those defined in SmSchemaCommonTypes

20 Nov 2014

SEA-5

Example: CSS SM Schema

• Developed for CCSDS Cross Support Service Management• Uses XML Schema and qualified namespace• References NDM XML schema for their content• Little in-line documentation, no in-line constraint checking• Defines SmSchemaCommonTypes that is included in other SM

schema• SmSchemaCommonTypes-v1.0.0• Defines a lot of CCSDS common types such as CCSDS 401, Space link, and

also SM common terms• Several different schema used to define config profiles, service

agreement, service package, and trajectory prediction• Common type definitions could be split into several sets

• SM specific common terms• SLE / CSTS specific common terms• Space link, RF, Mod, Coding common terms

• NOTE: CCSDS common terms could be merged with those defined in ndmxml-R1.5-ccsds-common

• NOTE: Identify a set of agency / service providers that could be a separate common registry

20 Nov 2014

SEA-6

Example: DEDSL DTD

• Developed for CCSDS Data Entity Dictionary exchange• Uses older XML DTD format, not a Schema• References DTD abstract spec for its content• Little in-line documentation, no in-line constraint checking• Defines some common types that are shared with other XML

standards: integer, real, text, identifier, • One DTD used to define dictionary name, version, language, and

related attribute specifications

• NOTE: Not clear that any of these DTD specifications are in use, need to verify• Could probably be replaced by ISO/IEC 11179-1:2004

20 Nov 2014

SEA-7

XML Schema Guideline Sources

• Google XML Document Format Style Guide, 2008• Topics: design or re-use, schema style (and what to do with DTDs),

namespaces, names & enumerated values, elements, attributes, values, key-value pairs, binary data, processing instructions, representation of XML document instances, elements vs. attributes

• Defines requirements and provides rationales

• ESA Design & Style Guide for XML Data and Schema, 2005• Topics: Project guidelines on use of XML, overview of XML, XML

tools and development environments (IDE), naming, design and style, security & encoding

• Design & style topic covers: design attributes, type, namespace, extensibility, constraints, versioning, global vs local

• Defines rules and recommendations with rationales

• XML Schema Best Practices, From XML-DEV, compiled by U of Cambridge, 2001• Topics: namespace, element vs type, extending schema, extensible

content, global vs local, hide vs expose, variable content containers• Analyzes different approaches, provides recommendations

20 Nov 2014

SEA-8

XML Schema Guideline – Common Ground(Order & text from ESA)

• Rule 1: Use meaningful, familiar and consistent names. • Rule 2: Avoid making abbreviations for names, removing

vowels.• Rule 8: Make two identical copies of all your schemas, where the

copies differ only in the value of elementFormDefault (in one copy set elementFormDefault="qualified", in the other copy set elementFormDefault="unqualified").

• Rule 9: Uniquely identify all schema components with the id attribute.

• Rule 10: If the item is not intended to be an element in instance documents, then define it as a type.

• Rule 11: If the item Gs content is to be reused by other items then define it as a type.

• Rule 12: If the item is intended to be used as an element in instance documents, and it is required that sometimes it be nillable and other times not, then it must be defined as a type.

• Rule 13: If the item is intended to be used as an element in instance documents and other elements are to be allowed to substitute for the element, then it must be declared as an ele- ment.

20 Nov 2014

SEA-9

XML Schema Guideline – Common Ground, contd(Order & text from ESA)

• Rule 17: If you have a targetNamespace, make it the default namespace.

• Rule 18: Do not hard code the identity of an imported schema.• Rule 19: Capture the schema version somewhere in the XML

schema. • Rule 20: Include information in the instance data files, that

makes it possible to determine which version (or versions) of the schema they are compatible with.

• Rule 21: Make older versions of your XML schema available. • Rule 22: In situations where a new version of a schema makes

backwards incompatible changes (e.g., a construct that was valid and meaningful for the previous schema does not validate against the new schema), make sure that older instances will not be accidentally validated against the new version.

20 Nov 2014

SEA-10

XML Schema Guideline – Common Ground(Order & text from Google)

• Attempt to reuse existing XML formats whenever possible, especially those which allow extensions.

• If you are reusing or extending an existing format, make sensible use of the prescribed elements and attributes, especially any that are required.

• Document formats SHOULD be expressed using a schema language. [Rationale: Clarity and machine-checkability.]

• Schemas SHOULD use the "Salami Slice" style (one rule per element). Schemas MAY use the "Russian Doll" style (schema resembles document) if they are short and simple.

• Element names MUST be in a namespace, except when extending pre-existing document types that do not use namespaces. A default namespace SHOULD be used. [Rationale: Namespace-free documents are obsolete]

• Attribute names SHOULD NOT be in a namespace unless they are drawn from a foreign document type or are meant to be used in foreign document types.

• Namespace names are HTTP URIs. Namespace names SHOULD take the form http://example.com/whatever/year, where whatever is a unique value based on the name of the document type, and year is the year the namespace was created.

20 Nov 2014

SEA-11

XML Schema Guideline – Common Ground, cont(Order & text from Google)

• Namespaces MUST NOT be changed unless the semantics of particular elements or attributes has changed in drastically incompatible ways.

• Namespace prefixes SHOULD be short (but not so short that they are likely to be conflict with another project). Single-letter prefixes MUST NOT be used. Prefixes SHOULD contain only lower-case ASCII letters.

• All names MUST use lowerCamelCase. That is, they start with an initial lower-case letter, then each new word within the name starts with an initial capital letter.

• Names MUST contain only ASCII letters and digits. • Published standard abbreviations, if sufficiently well-known,

MAY be employed in constructing names. Ad hoc abbreviations MUST NOT be used.

• All elements MUST contain either nothing, character content, or child elements. Mixed content MUST NOT be used.

• Document formats MUST NOT depend on the order of attributes in a start-tag.

• Elements SHOULD NOT be overloaded with too many attributes (no more than 10 as a rule of thumb). Instead, use child elements to encapsulate closely related attributes.

20 Nov 2014

SEA-12

• Numeric values SHOULD be 32-bit signed integers, 64-bit signed integers, or 64-bit IEEE doubles, all expressed in base 10. These correspond to the XML Schema types xsd:int, xsd:long, and xsd:double respectively. If required in particular cases, xsd:integer (unlimited-precision integer) values MAY also be used.

• Boolean values SHOULD NOT be used (use enumerations instead). If they must be used, they MUST be expressed as true or false, corresponding to a subset of the XML Schema type xsd:boolean.

• Embedded syntax in character content and attribute values SHOULD NOT be used. Syntax in values means XML tools are largely useless. Syntaxes such as dates, URIs, and XPath expressions are exceptions.

• Simple key-value pairs SHOULD be represented with an empty element whose name represents the key, with the value attribute containing the value. Elements that have a value attribute MAY also have a unit attribute to specify the unit of a measured value. For physical measurements, the SI system SHOULD be used.

20 Nov 2014

XML Schema Guideline – Common Ground, cont(Order & text from Google)

SEA-13

XML Schema Guideline – Common Ground, cont(Order & text from Google)

• Binary data MUST NOT be included directly as-is in XML documents, but MUST be encoded using Base64 encoding.

• An attribute named xsi:type with value xs:base64Binary MAY be attached to this element to signal that the Base64 format is in use. [Rationale: Opaque blobs should have decoding instructions attached.]

• Use standard XML parsers, not hand-rolled hacks. • The character encoding used SHOULD be UTF-8. Exceptions

should require extremely compelling circumstances. • Namespaces SHOULD be declared in the root element of a

document wherever possible. The mapping of namespace URIs to prefixes SHOULD remain constant throughout the document, and SHOULD also be used in documentation of the design.

• Well-known prefixes such as html: (for XHTML), dc: (for Dublin Core metadata), and xs: (for XML Schema) should be used for standard namespaces.

20 Nov 2014

SEA-14

XML Schema Guideline – Common Ground, cont(Order & text from Google)

• Attributes are more restrictive than elements, and all designs have some elements, so an all-element design is simplest -- which is not the same as best.

• If something might appear more than once in a data model, use an element rather than introducing attributes with names like foo1, foo2, foo3 ....

• Use elements to represent a piece of information that can be considered an independent object and when the information is related via a parent/child relationship to another piece of information.

• Use elements when data incorporates strict typing or relationship rules.

• If order matters between two pieces of data, use elements for them: attributes are inherently unordered.

• Use common sense and BE CONSISTENT. Design for extensibility. You are gonna need it. [Rationale: Long and painful experience.]

• Break ANY OR ALL of these rules (yes, even the ones that say MUST) rather than create a crude, arbitrary, disgusting mess of a design if that's what following them slavishly would give you.

20 Nov 2014

SEA-15

XML Schema Guideline – Discussion Topics(Order & text from ESA)

• Rule 23: Where minimizing size and coupling of components is of utmost concern then use the so-called Russian Doll design.

• Rule 24: Where your task requires that you make available to instance document authors the option to use element substitution, then use the so-called Salami Slice design.

• Rule 25: The so-called Venetian Blind design is the one to choose when your schemas require the flexibility to turn namespace exposure on or off with a simple switch, and where component reuse is important.

20 Nov 2014

SEA-16

XML Schema Guideline – Discussion topics(Order & text from Google)

• The schema language SHOULD be RELAX NG compact syntax. Embedded Schematron rules MAY be added to the schema for additional fine control. [Rationale: RELAX NG is the most flexible schema language, with very few arbitrary restrictions on designs.]

• Dates should be represented using RFC 3339 format, a subset of both ISO 8601 format and XML Schema xsd:dateTime format. UTC times SHOULD be used rather than local times.

20 Nov 2014

SEA-17

Simple RelaxNG Example(From IBM.com Developers: XML Matters)

• Listing 3: A context-free compact syntax schema

# A library patron example default namespace = "http://some.other.url/ns" namespace foo = "http://home.of.foo/ns" datatypes xsd = "http://www.w3.org/2001/XMLSchema-datatypes"

## Annotation here start = patron patron = name & id-num & book name = element name { xsd:string { pattern = "\w{,10}" } } id-num = element id-num { xsd:string } book = element book

{ ( attribute isbn { text } | attribute title { text } | attribute anonymous { empty }) }*

20 Nov 2014

SEA-18

XML Schema version of Simple RelaxNG Example (From IBM.com Developers: XML Matters)

<?xml version="1.0" encoding="UTF-8"?> <!-- A library patron example --> <grammar xmlns=http://relaxng/ns/structure/1.0

ns="http://some.other.url/ns" datatypeLibrary="http://www.w3.org/2001/XMLSchema-datatypes" xmlns:a="http://relaxng.org/ns/compatibility/annotations/1.0" xmlns:foo="http://home.of.foo/ns">

<a:documentation>Annotation here</a:documentation> <start><ref name="patron"/></start><define name="patron"> <interleave> <ref name="name"/> <ref name="id-num"/> <ref name="book"/> </interleave> </define><define name="name"> <element name="name"> <data type="string"/> <param name="pattern”>\w{,10}</param> </data> </element> </define>

20 Nov 2014

SEA-19

XML Schema version of Simple RelaxNG Example contd(From IBM.com Developers,XML Matters)

<define name="id-num"> <element name="id-num"> <data type="string"/> </element> </define><define name="book"> <zeroOrMore> <element name="book"> <choice> <attribute name="isbn"/> <attribute name="title"/> <attribute name="anonymous"> <empty/> </attribute> </choice> </element> </zeroOrMore> </define> </grammar>

20 Nov 2014

SEA-20

Relax NG and Schematron

• RelaxNG may embed Schematron rules• Schematron is a:

• Way to test XML documents • Rules-based validation language • Way to specify and test statements about your XML

document: elements, attributes, content • Schematron can: do validation, run reports, find

patterns, check elements against a vocabulary• A Schematron “program” is a well-formed XML

document, run by a processor • Schematron specifies, it does not perform

• A Schematron “schema” specifies tests to be made on your XML

• A set of declarations for a process (“test this; tell me that”)

20 Nov 2014

SEA-21

RelaxNG and “normal” Schema

• RelaxNG has two forms, “normal” and “compact”• Compact form is very straightforward, easy to read,

and avoids all of the “<element name="id-num"> <data type="string"/></element>” clutter

• Most tools handle RelaxNG and there are a number of translators from RelaxNG to Schema to DTD and back again (as long as compatible features are used)

• But … RelaxNG is not in use in CCSDS today and there is little familiarity with it

20 Nov 2014

SEA-22

Observations

• Existing CCSDS XML standards were all developed independently and with different formats and styles.

• Different levels of care have been taken to define common terms in any given set of documents.

• There are instances where the same terms are defined in more than one spec and others where the same concept is defined differently.

• There are more and more instances where XML formats defined in one WG are used in another WG, and this is likely to increase.

• Trying to do this work in a SIG, on a shoestring, just isn’t working, it needs to be part of a WG

• Propose making it be a part of the SEA SAWG re-start, along with RASDS and ontologies

20 Nov 2014

SEA-23

XML Notes

• Consider inserting a guideline for users to perform adequate validation of the schema between the source and the sink• Content• Structure• Potentially validity and authentication

• Adhere to Postel’s rule: Be liberal in what you accept, be strict in what you send

• Provide other transformations than just XML, such as HTML, JSON, other styles …

• Consideration of XSD, RelaxNG, ABNF, ASN.1, or other alternate abstract forms • Not about tooling, about compliance, transformability

and validation• Check with Francisco Martinez, John Pietras, qual

vs unqual issues

20 Nov 2014

SEA-24

Proposed Phased Approach

• Identify all current and planned XML related standards.• Pick a key subset of the XML standards and develop schema

structure and content validators, potentially build using COTS tools and Schematron.

• Vet the validation approach with the WGs and do V&V of the capability. Make the result available to the community on the SANA web site (TBC).

• Establish a process for developing similar validators for all CCSDS XML standards.

• Analyze current XML standards to identify places where there are common terms defined, such as NDM XML and CSSM.

• Develop an evaluation suite of common terms, suitable for referencing and a parallel set of modified schema.

• Vet the common terminology approach with the WGs and do V&V of the capability. Make the result available to the community on the SANA web site (TBC).

• Establish a process for developing similar adaptations for all CCSDS XML standards.

• Develop a process for integrating the terms defined in the CCSDS ontology into the updated schema definitions.

20 Nov 2014

SEA-25

Backup

The following slides are from:

• SysML 1.4:Quantities, Units, Dimensions & Values (QUDV) & ISO 80000 Library

Nicolas RouquetteJet Propulsion LaboratoryCalifornia Institute of TechnologyMarch 2014

• Copyright 2013, 2014, California Institute of Technology (“Caltech”)U.S. Government sponsorship acknowledged. All rights reserved.

20 Nov 2014

SEA-26

Using VIM 3rd edition in SysML 1.4 & QUDV

Normative SysML Non-normative QUDV & ISO-80000

1 Quantities and units (30 definitions) language concept

modeling pattern

language concept

modeling pattern

ISO-80000-1 Library

1.1 quantity ✔        1.2 kind of quantity     ✔    1.3 system of quantities     ✔    1.4 base quantity     ✔    1.5 derived quantity     ✔    1.6 International System of Quantities (ISQ)         ✔1.7 quantity dimension     ✔    1.8 quantity of dimension one     ✔    1.9 measurement unit ✔        1.10 base unit     ✔    1.11 derived unit     ✔    1.12 coherent derived unit       ✔  1.13 system of units     ✔    1.14 coherent system of units       ✔  1.15 off-system measurement unit       ✔  1.16 International System of Units (SI)         ✔1.17 multiple of a unit     ✔    1.18 submultiple of a unit     ✔    1.19 quantity value   ✔      1.20 numerical quantity value   ✔      1.21 quantity calculus       ✔  1.22 quantity equation     ✔    1.23 unit equation     ✔    1.24 conversion factor between units     ✔    1.25 numerical value equation   ✔      1.26 ordinal quantity   ✔      1.27 quantity-value scale          1.28 ordinal quantity-value scale   ✔   ✔  1.29 conventional reference scale   ✔   ✔  1.30 nominal property       ✔  

No Coverage in QUDV or SysML

2 Measurement (53 definitions)3 Devices for measurement (12 definitions)4 Properties of measuring devices (31 definitions)5 Measurement standards (etalons) (18 definitions)

http://www.bipm.org/en/publications/guides/vim.html

20 Nov 2014

SEA-27

Systems of Units & Quantity KindsNormative SysML Non-normative QUDV & ISO-80000

Def. Quantities and units (30 definitions) language concept

modeling pattern

language concept

modeling pattern

ISO-80000-1 Library

1.3 system of quantities     ✔    1.6 International System of Quantities (ISQ)         ✔1.13 system of units     ✔    1.16 International System of Units (SI)         ✔

20 Nov 2014

SEA-28

Practical Considerations for Modeling Values in SysML

#1: Recognize the distinct levels of modeling involved

• A Level depends on those abovei < j => Vi depends on Vj

• V4 < V5See VIM3 & ISO 80000-1 foreword:

This first edition of ISO 80000-1 cancels and replaces ISO 31-0:1992 and ISO 1000:1992. The major technical changes from the previous standard are the following: the structure has been changed to emphasize that quantities come first and units then follow;

• V3 < V4, V5Per SysML ValueType

• V2 < V3Per UML TypedElement/Type relationship

• V1 < V2Per UML ValueSpecification/Slot/ StructuralFeature & defaultValue relationships

QuantityKinds

Units

ValueTypes(QuantityKind, Unit)

Value Properties(typed by ValueTypes)

Value Specifications(typed by ValueTypes)

V1

V2

V3

V4

V5

Runtime ValuesV0

20 Nov 2014

SEA-29

Practical Considerations for Modeling Values in SysML

#2: Recognize the alignment with UML

QuantityKinds

Units

ValueTypes(QuantityKind, Unit)

Value Properties(typed by ValueTypes)

Value Specifications(typed by ValueTypes)

V1

V2

V3

V4

V5

Runtime ValuesV0

UML InstanceSpecifications (classifier = SysML or QUDV

Unit or QuantityKind)

UML DataTypes(stereotype = SysML::ValueType

& stereotype tags: quantityKind, unit)

UML Properties(type = <<ValueType>> UML DataType)

UML ValueSpecifications(type = <<ValueType>> UML DataType)

Varies by methodology

20 Nov 2014

SEA-30

Practical Considerations for Modeling Values in SysML

#3: Recognize the implications of DataType modeling in UML

ValueTypes(QuantityKind, Unit)

Value Properties(typed by ValueTypes)

Value Specifications(typed by ValueTypes)

V1

V2

V3

Runtime ValuesV0

UML DataTypes

UML ValueSpecifications

To Be Decided

- Scalar?- Structured?

- Literals?- Expressions?- Intervals?- Instance

Specifications?

Choices made about datatype modeling…

… have implications for modeling values…

… and type checking !

The conformance of a ValueSpecificationto a ValueType depends on these choices!

20 Nov 2014