1. 1 A QUANTITATIVE ANALYSIS OF MODEL-DRIVEN CODE GENERATION THROUGH SOFTWARE EXPERIMENTATION Paulo Eduardo Papotti Antonio Francisco do Prado Wanderley Lopes de Souza Carlos Eduardo Cirilo Lus Ferreira Pires CAiSE13 - 25th International Conference on Advanced Information Systems Engineering CAiSE13 Motivation Model-Driven Development (MDD) has been around for a while and has become quite popular Benefits are acknowledged but some organisations are still reluctant to include MDD in their common practices Research can help this by providing evidences of these benefits Reduce development time Facilitate development Avoid mistakes 28/06/2013 2 To avoid misunderstanding: development based on metamodelling and model transformations CAiSE13 Motivation We performed a quantitative study to evaluate the impact of using model- driven code generation (code generation from models via M2C transformations) on the time spent by groups of students to develop software systems by comparing results of manual coding with code generation from models and applying a systematic experimental methodology 28/06/2013 3 CAiSE13 General structure of an experiment 28/06/2013 4 Cause construct Effect construct Treatment Outcome Realm of theory Experiment objective Independent variable Dependent variable Experiment operation Realm of practise CAiSE13 Performed experiment (definition) Experiment conducted in the second half of 2011 following the phases of Wohlin et al. [6] Comparative analysis of the time spent implementing the CRUD (Create, Retrieve, Update, Delete) functions of web systems from models of the application entity classes (UML class diagrams) 28/06/2013 5 Analyse use of mechanisms for code generation from models to construct web applications For the purpose of evaluation With respect to efficiency (time) From the point of view of software developers In the context of CS and CE undergraduate students CAiSE13 Hypotheses formulation Null Hypothesis (H0) No difference in efficiency () between code generation from models and manual coding to build web applications H0: CodeGeneration = Classic CodeGeneration = Classic Alternative Hypothesis (H1) Code generation from models more efficient than manual coding H1: CodeGeneration > Classic CodeGeneration < Classic Alternative Hypothesis (H2) Manual coding more efficient than code generation from models H2: CodeGeneration < Classic CodeGeneration > Classic 28/06/2013 7

2. 2 CAiSE13 Variables selection Independent variables Developed application, development approach, development environment and technologies Application: CRUD part of ProgradWeb system Development environment: Eclipse IDE and MySQL database Development technologies: Java, Hibernate, Java Persistence API (JPA), Java Server Faces (JSF) and XHTML Dependent variable Group efficiency 28/06/2013 8 CAiSE13 Experiment design Subjects selected according to convenience sampling 29 students from 3rd and 4th years of CS and CE UFSCar undergraduate programmes Homogeneous groups based on their experience level 28/06/2013 9 CAiSE13 Experiment design Paired comparison development approach factor (independent variable) with two treatments 28/06/2013 10 Group Code Generation Manual Coding G1 1 2 G2 1 2 G3 2 1 G4 1 2 G5 2 1 G6 2 1 G7 2 1 G8 1 2 G9 1 2 CAiSE13 Preparation Participant's Characterization Form: questionnaire to assess knowledge on the technologies and concepts Consent Form: objectives and confidentiality, and freedom to withdraw Task Description: task to be performed Support Material: roadmap of steps to be used Data Collection Form: to record the start time and end time of each activity Evaluation Form: multiple-choice questions to evaluate and compare the two development approaches 28/06/2013 11 CAiSE13 Execution results 28/06/2013 12 TProjTEntity ClassesTPersistenceTmanaged BeansTConvertersTWeb PagesTotalManualCodingG1 0 h 14 min 0 h 32 min 0 h 7 min 0 h 25 min 0 h 6 min 0 h 44 min 2 h 8 minG2 0 h 6 min 0 h 38 min 0 h 3 min 0 h 13 min 0 h 4 min 0 h 59 min 2 h 3 minG3 0 h 7 min 0 h 59 min 0 h 3 min 0 h 8 min 0 h 3 min 0 h 37 min 1 h 57 minG4 0 h 8 min 1 h 8 min 0 h 8 min 0 h 14 min 0 h 6 min 0 h 15 min 1 h 59 minG5 0 h 7 min 0 h 49 min 0 h 8 min 0 h 15 min 0 h 6 min 0 h 40 min 2 h 5 minG6 0 h 8 min 0 h 41 min 0 h 13 min 0 h 25 min 0 h 7 min 0 h 33 min 2 h 7 minG7 0 h 13 min 0 h 33 min 0 h 7 min 0 h 32 min 0 h 8 min 0 h 22 min 1 h 55 minG8 0 h 5 min 0 h 30 min 0 h 10 min 0 h 10 min 0 h 2 min 1 h 2 min 1 h 59 minG9 0 h 8 min 1 h 7 min 0 h 9 min 0 h 8 min 0 h 6 min 0 h 29 min 2 h 7 minAverage 0 h 8 min 0 h 46 min 0 h 7 min 0 h 16 min 0 h 5 min 0 h 37 min 2 h 2 minCodeGenerationG1 0 h 7 min 0 h 4 min 0 h 4 min 0 h 15 minG2 0 h 8 min 0 h 3 min 0 h 1 min 0 h 12 minG3 0 h 1 min 0 h 7 min 0 h 1 min 0 h 9 minG4 0 h 3 min 0 h 4 min 0 h 1 min 0 h 8 minG5 0 h 2 min 0 h 1 min 0 h 1 min 0 h 4 minG6 0 h 8 min 0 h 2 min 0 h 3 min 0 h 13 minG7 0 h 10 min 0 h 2 min 0 h 1 min 0 h 13 minG8 0 h 6 min 0 h 2 min 0 h 1 min 0 h 9 minG9 0 h 9 min 0 h 6 min 0 h 3 min 0 h 18 minAverage 0 h 6 min 0 h 3 min 0 h 1 min 0 h 11 minCAiSE13 Dispersion of collected data 28/06/2013 13 3. 3 CAiSE13 Data set normality verification 28/06/2013 14 Timespent(minutes) Timespent(minutes) CAiSE13 Results analysis Hypothesis testing (Paired t-test) t0 = 65.30667 From the Students t-distribution = 6.8 x 1012 , t0.0275,8 = 2,6899, t3.4 x 10 -12 ,8 = 65.2154 |t0| > t3.4x10 -12 ,8 the null hypothesis H0 can be rejected with a significance level of 6.8x10-12 % (probability of observation by chance) 28/06/2013 15 CAiSE13 Discussion of results Difference in development time spent by the groups was due to the development approach they used during the experiment, and not by any accident or mistakes in the collection of samples Evidence that hypothesis H1 can be validated instead of H2 Generation from models usually costs less time than manual coding (under the conditions of our experiment) Results of this study are limited to the scope of CRUD web applications implemented by software developers in a university environment 28/06/2013 16 CAiSE13 Difficulties faced by participants 28/06/2013 17 CAiSE13 Perceived advantages of code generation 28/06/2013 18 CAiSE13 Validity threats 1. Internal Validity (interference) Homogeneous groups with respect to experience level Random assignment of manual coding and code generation 2. External Validity (generalisation) Third and Fourth year (experienced) undergraduate students 3. Conclusion Validity (conclusions from statistics) Statistical analysis of results 4. Construct Validity (measurements representative for concepts) Experiment as similar as possible to working situation 28/06/2013 19 Cause construct Effect construct Treatment Outcome Independent variable Dependent variable 1 2 3 4 4. 4 CAiSE13 Final remarks Performed a quantitative analysis of an experiment to evaluate mechanisms for code generation from models Experiment executed in a university environment (teaching laboratory) Results confirm the benefits of model-driven code generation Experiment can be reproduced download from * Future work Evaluate effectiveness by analysing implementation faults New similar experiment in an industrial environment Other experiments with code generation (e.g., other applications) 28/06/2013 20 * www.dc.ufscar.br/~paulo.papotti/ EXPERIMENT.zip