5/5/2005Toni Räikkönen

Internet based data collection from enterprises using

XML questionnaires and XCola engine

CoRD MeetingMay 11th 2005

5/5/2005 2

Background

Many web based data collection systems developed in Statistics Finland since 2000

Developed using traditional web development methods like ASP, ASP.NET, COM+ etc.

The collection systems are usually quite user friendly consisting of large amount of interaction with the respondent

Automated fills, validations, feedback etc.

5/5/2005 3

Background (cont.)

The development process usually includes a lot of customization and tailoring for a certain survey

The results have typically been “unique” systemsNo automation in development

Requires hard and usually more or less the same work time after time

Separate maintenance cycle for each systemNot very modern way of producing applications

5/5/2005 4

Questions

How to improve the development of web collection systems?

Frameworks?Generic applications?Off-the-shelf applications?

What is the role of XML? etc.

5/5/2005 5

How it all started?

A quick ‘n’ dirty prototype at the end of 2003Used XML4DR to represent a static customized web form

The outcome was quite nice but lacked the dynamics we required for the web surveys

QuestionsCould we extent the use of XML, especially the XML4DR?

Define the customization routines in XML?What sort of application would we need to execute all this?

5/5/2005 6

Early findings

The suitability of pure XML4DR was not adequate enough

We required quite a lot dynamic functionalityXML4DR did not support thisXForms does

But is difficult to execute on the server sideWe later learnt about open source project

Chiba (http://chiba.sourceforge.net/)

5/5/2005 7

What is XCola?

XML based collection application A generic application for web surveys

Typically for surveys aimed to enterprises Meaning, the surveys

conform to each otherare developed using modern techniques and technologies

can be developed “easily”

5/5/2005 8

…and technically?

An engine that reads XML questionnaires as input files and outputs them as XHTML web forms

Typically does a lot of processing for the questionnaires between the input and output phases

Enabling the dynamics for the surveys

5/5/2005 9

Requirements

A generic application The questionnaires purely in XML

No programming, no scriptingDeclarativeMust support customization

Server side implementation Must support at least all the functionality of the older collection systems

5/5/2005 10

Requirements (cont.)

Requirements for the XML formatSupport for all the typical question types

Open questions, multiline questions, multiple choices, drop down lists etc.

Support for open structuresAdding / removing rows or columns

Should be unaware of the presentation logicNo styles, no JavaScript what so ever

5/5/2005 11

Requirements (cont.)

The questionnaires could be stored in XML or in relational databases

Because of our existing infrastructure we decided to store the questionnaires into databases

The database structure must be genericAll the questionnaires will be stored in the same format

The data should be easily mapped back into the XML questionnaires

5/5/2005 12

XML questionnaire

We chose the XML4DR as the base format However as it doesn’t support the required dynamics we modified it quite a bit

Generalized the format to support all the imaginary questionnaire types

Added some extra functionality taken from XFormsIntroduced customization routines

Implemented these with processing instructions and XML templates

5/5/2005 13

The XCola engine

Represents an instance of a web survey Processes the questionnaires and transforms them into web applications

Reads and writes the contents of the XML questionnaire from/to the generic database structure

Supports client and server side validations Makes use of the authentication and authorization framework

5/5/2005 14

Preprocessors

The XML questionnaire contains processing instructions for the XCola engine.

And sometimes also XML templates to support the PIs

The engine reads these instructions and customizes the document accordingly

The preprocessors are implemented using chain of responsibility design pattern.

The processors are executed in a chain one at a time until all instructions are processed

5/5/2005 15

Preprocessors (cont.)

So far the engine supports preprocessing forVisibility settingsPre-FillingRow fetchingDynamic Domain (column) generationDynamic Data (cell) generationDynamic Calculation formula generation

5/5/2005 16

Post processors

Post processing takes place when the user submits the web form

The types of post processorsUpdate the XML document from the web form contents

Update the XPath referencesServer side validationUpdate the dynamic structures in the XML document (e.g. adding / removing rows)

Saving XML document contents into the database

5/5/2005 17

Transformation into a web form

The XML questionnaire is transformed into ASP.NET web form using XSLT

ASP.NET controls are used for input controls Validation controls are used for client side validation User interface related things are brought in using CSS

JavaScript libraries are connected into the web page during the transformation

5/5/2005 18

Pros

One application, no maintenance hell No programming required for the questionnaire Supports all the basic functionality of the web survey Works well on the enterprise surveys

Typically not suitable for interview based surveys Customizable; use just the features needed in the survey

Supports linking of external components in case some custom functionality is required

5/5/2005 19

Cons

Generic solution can not implement any survey dependent functionality

Everything must be generic Seems to fit badly into very hierarchical surveys (with more than two levels)

Some parts of the XML questionnaire are not so trivial

Especially constraints defined in XPath statements can be really hard to understand

5/5/2005 20

Current situation

Couple of in-house test surveys during fall 2004No major problems

In production since January 2005Currently running six different surveys

The smallest with just 200+ respondentsAnd the largest with over 24 000 respondents!

There will probably be 4-5 new surveys during this year

Seems to be the perfect solution for Statistics Finland!

5/5/2005 21

The future

Probably some minor development Some new featuresScalability investigationsetc.

Polish the tools that are needed to administer XCola surveys

5/5/2005 22

XCola in a nutshell

Think about XForms: Most of the typical things you can do with XForms you can do with XCola

Although our solution isn’t so declarative and requires the power of the engine

Executed on the server side Works on every modern browser Extremely simple Easy to implement the questionnaires just in hours