reverse engineering & architecture · pdf filereverse engineering & architecture...

1 © NOKIA 2000, 2002 Name v.ppt/ 26 April 2002 / Riva

Reverse Engineering & Architecture Reconstruction

Claudio RivaSoftware Architecture Group

Software Technology LaboratoryNokia Research Center

[email protected]

Helsinki, 19 February 2003


Contents• Introduction to reverse engineering

• Software Maintenance• Definition of terms

• The basics of reverse engineering:• Source code models and formats• Phase 1: Extraction• Phase 2: Visualisation• Phase 3: Abstraction• Phase 4: Presentation

• Demonstration on the Venice case• Architecture Reconstruction• Nokia Case Study• Dynamic Analysis• Bibliography


What is a Legacy System ?A legacy system is a piece of software that:

• you have inherited, and• is valuable for you

… but legacy systems are not a bad thing...

Common problems of a legacy system:• original developers are no longer available• missing or outdated documentation• extensive patches or modifications have been made• internal architecture of the system is unclear• further development is expensive and difficult


Software Maintenance

Software maintenance is the “modification of a software product after delivery to correct faults, to improve performance or other attributes, or to adapt the product to a changed environment” [ANSI/IEEE Std. 729-1983]

Corrective maintenance (17%)fixing reported errors in the software

Adaptive maintenance (18%)adapting the software to a newenvironment

Perfective maintenance (65%)implementing new functional ornon-functional requirements


Definitions

• "Forward Engineering is the traditional process of moving from high-level abstractions and logical, implementation-independent designs to the physical implementation of a system"

• "Reverse Engineering is the process of analysing a subject system to q identify the system's components and their interrelationshipsq create representations of the system in another form or at an higher level of

abstraction"

• "Reengineering … is the examination and alteration of a subject system to reconstitute it in a new form and the subsequent implementation of the new form.”

[Chik90]


Reverse and Reengineering

Requirements RequirementsForward Engineering Re

vers

e En

gine

erin

g

Software System

Design models

Reengineering

abst

ract

ion


Level of abstractionsDomain model: describe the commonality and variability of requirements for

software components within a given Domain.

Requirements: A software system has to satisfy all the functional and non-functional requirements that are specified in the problem domain

Feature: is a “coherent and identifiable bundle of system functionality” . Features are usually mapped in architectural concepts, like subsystems or components.

Architecture: is the collection of design decisions that enable to implement the functional requirements and to satisfy the quality parameters of the system. It is a clear specification of the elements that build the system and their interactions.

Design: the building blocks are internally designed and implemented in one or more programming languages. The design specifies how the components satisfy their architectural specifications. At this level, we deal with design entities like classes, functions, abstract data types, data structures, user defined types and their interactions.

Code: the code is the artefact at the lowest level of abstraction. The behaviour of a piece of code is expressed in a programming language.

abst

ract

ion

Prob

lem

Dom

ain

Solu

tion

Dom

ain


Goals of Reverse Engineering

Cope with complexity and size• need techniques to understand large and complex systems

Generate alternate views• automatically generate different ways to look at the system

Recover lost information• extract what changes have been made

Generate higher abstractions• extract the "big picture" of the system

Facilitate reuse• detect stable and reusable artefacts of the system


Reverse Engineering: key techniques

• Re-documentation is the creation or revision of a semantically equivalent representation of a subject system within the same relative abstraction level

• pretty printers• diagram generators• cross-reference listing generators

• Design recovery is the creation of design abstractions (beyond those directly obtainable by examining the system) from a combination of code, domain knowledge, existing documentation, personal experience, deduction or fuzzy reasoning

• static and dynamic analysers• software metrics• browsers• visualisation and abstraction tools


Goals of Reengineering

Improve the system• identify and remove defects• performance• enhance the quality of the implementation

Adapt the system to new scenarios• porting• monolithic towards product family


Reengineering: key techniques

• Restructuring is the transformation from one representation form to another at the same relative abstraction level, while preserving the system's external behaviour.

• Data Engineering is the process of analysing and reorganising the data structures in a system to make it more understandable

• Re-factoring is restructuring within an object-oriented context• renaming/moving methods/classes etc. [Fowler99]


Approach to Reverse & Re-engineering

SourceCode

NewSource Code

Source CodeModel New model

0) Requirements1) Model capture2) Abstraction3) Presentation4) Reorganization5) Change propagation

(1)

(4)

(3)

(5)

(0)

High levelmodel

(2)


Tools for RE

• Tools are necessary • to cope large amount of data, and• to enable the repeatability of the methods.

• Reengineering vs. forward engineering• Forward engineering tools are chosen deliberately• Reengineering tools must integrate with what’s already in place

• Tools integration• commercial, academic and ad hoc tools• use of a common repository• APIs and interoperability standards


Basic Tool Architecture

Model CaptureTools

Repository

Source CodeSystemDocumentation

ConfigurationManagement System System Experts

Model AnalysisTools

Refinement


Tools• FAMOOS Environment for reengineering OO systems [FAMOOS]

• CodeCrawler [Demeyer99], Duploc [Ducasse99], Goose [Ciupke99], Sniff+ [Sniff]

• Rigi• programmable visual environment for program understanding [Rigi]

• Personal Bookshelf (SWAG toolkit)• web-based paradigm for the presentation and navigation of information representing large

software systems [PBS]

• Dali• a Software Architecture Reconstruction Workbench [Kazman96]

• Datrix• source code assessment (software metrics) [Datrix]

• RedHat Source Navigator• Programming Environment and source code browser [SourceNavigator]

• Front-end-art• Front-end for C/C++ [FrontEndArt]

• DMS Software Reengineering Toolkit• Reengineering based on code transformation [DMS]


The basics of reverse engineering


Where is the model ?import java.applet.Applet;import java.awt.Graphics;

public class HelloWorld extends Applet{

public void paint(Graphics g) {g.drawString("Hello world!", 50, 25);

}}

+paint() : void

HelloWorld

Applet

+drawString() : void

Graphics

The UML class diagram used during design


Some issues of UML for RE

• The UML language is not precise

• Some of the UML concepts have no direct mappings to the implementation

• UML lacks some concepts that are necessary to model the source code

Aggregation...

Composition

Class Attribute

Generalization= Inheritance

Operation = Method

Invocation

Access

UML Reengineering

Square

Circle

Inheritance orGeneralisation ?

[Demeyer99]


Requirements for the RE source code model

• Language independent• We need to support a variety of languages (C/C++, Java, Smalltalk)

• Domain independent• Each domain has its own terminology, we need to adapt our tools to what we find in

the system (differently from forward engineering).

• Extensible• We need to change/extend the model during the RE activity

• Easy to use• RE is a kind of “exploratory” activity. We need flexible and easy to use tools.

• Scalable• Think of an unknown system containing thousands classes

• Different levels of abstractions• Abstract Syntax Trees, Architectural models,


Attributed Typed Directed Hierarchical Graphs• RE information is well-represented with relational graphs defined as a

set of • Concepts: software artefacts (e.g. classes, methods, subsystems)• Relationships: relationships among the artefacts (e.g. inheritance)

• The basic elements of the graphs are:• Nodes: model artefacts (e.g. classes, methods, subsystems)• Arcs: relationships between the artefacts (e.g. invocation)• Attributes: key-value pairs per node/edge

• A schema defines the allowed types of nodes and arcs

Line: 15File: foo.c

Function: foo

Function: print

Line: 85File: print.c

invocation

Line: 25File: foo.c

Variable: a

access


Again, where is the model ?import java.applet.Applet;import java.awt.Graphics;

public class HelloWorld extends Applet{

public void paint(Graphics g) {g.drawString("Hello world!", 50, 25);

}}

Class: HelloWorld

Method: paint

Class: Applet Class: Graphics

Method: drawString

inherit

has_method

has_methodinvocation


The Schema of a Model

• The schema of the graph defines the types of nodes and edges.

• The schema evolves during the RE activity by including new concepts that become relevant or that are discovered during the process.

• There are a set of basic schemas to start with, e.g. FAMIX, DMM

Class

call

inheritanceMethod

has_methodsuperclass

subclass

caller

caleethe schema of the previous example


The FAMIX model for OO systems

• The FAMIX model was developed during the FAMOOS project with thefollowing goals in mind:

• To create an exchange mechanism for RE tools• To decouple the model capture from the model analysis• To be language independent and extensible• To support different programming languages via plug-ins

FAMIXCommonFormat

Language Plug-ins

Languages PrototypesC++

Ada

Java

Smalltalk

Grouping

Metrics

Heuristics

Visualization


The FAMIX core model• Entities: class, method, attribute

• Relationships: inheritance, access, invocation

superclass

subclassClass

MethodAttribute

Invocation Access

Inheritance

candidates

belongsToClass

invokedBy

belongsToClass

accessedIn

accesses


The Dagstuhl Middle Model (DMM)


The format of the graph files

• The format has to be easy to generate, read and also suitable for manual inspections.

• Textual representations: • RSF: Rigi Standard Format

• Unstructured RSF for plain graphs• Structured RSF for hierarchical graphs

• GXL: Graph eXchange Language• Plain graphs• Hierarchical graphs

• Prolog facts

• The choice of the file format plays an important role because we need to be able to manipulate the facts in a very easy way (e.g. with a text editor) and to convert them from one format to another one.


The unstructured RSF format

• This format is used to describe a plain graph without any hierarchy. The format is a text file with a list of tuples with the following syntax:

<verb> <subject> <object>

where verb can be any arc defined in the Rigi domain or the built-in verb type. Subject and object are respectively the source and destination of the arc. The nodes must have a unique name.

type Applet Classtype HelloWorld Classtype Graphics Classtype HelloWorld.paint(Graphics) Methodtype Graphics.drawString(String,int,int) Methodinherit HelloWorld Applethas_method HelloWorld HelloWorld.paint(Graphics)has_method Graphics Graphics.drawString(String,int,int)invocation HelloWorld.paint(Graphics)

Graphics.drawString(String,int,int)

The definition of a node

The definition of an arc


The structured RSF format• The data about the hierarchy is stored in the “level” arcs. The source of the

arc represents the parent and the destinations of the arc represents its child. The syntax is:

level <parent> <child>

• In order to be visible, all the top nodes must be children of the “Root” node.

level Root <top node>

type Root Syntactictype HelloWorld Classtype Graphics Classtype HelloWorld.paint(Graphics) Methodtype Graphics.drawString(String,int,int) Methodlevel Root HelloWorldlevel Root Graphicslevel HelloWorld HelloWorld.paint(Graphics)level Graphics Graphics.drawString(String,int,int)


Approach to Reverse & Re-engineering

SourceCode

NewSource Code

Source CodeModel New model

0) Requirements1) Model capture2) Abstraction3) Presentation4) Reorganization5) Change propagation

(1)

(4)

(3)

(5)

(0)

High levelmodel

(2)


Phase 1: Extraction• Problem:

• The source code level contains too many details.• The system is to complex to be examined at once

• Goal:• To extract a source code model of the system that contains only the most relevant

information

• Actions:• To select/define the schema of the model we want to extract• To select/create/configure the source code analysers that extract a source code

model

• Input: • Source code• Software artefacts (such as documents, mapping tables, databases)

• Output: • a set of concepts and relationships

• Tools:• “grep”, scripts (Perl, Tcl), front-ends (Source Navigator, Sniff+, Front-End-Art)


Example: Source Navigator -> FAMIX


The FAMIX Schema

Class

FileDirectory

GlobalVariable

TypeDef

Macro

AttributeMethodFunction

expansion

use_type

expansion

use_type

access access

access access

Function

Method

Class

has_method has_attribute

has_class

access

invocation

invocation

contain_file

contain_dir

Class

GlobalVariable

TypeDef

Macro

Function

containcontain

contain

containcontain_file

include


Source Code Analysers

• Exact parsers• Front End Art (http://www.frontendart.com)• CPPX - Open Source C++ Fact Extractor (http://swag.uwaterloo.ca/~cppx)• MSVisual C++ plug-in• C++ Front End from Edison Design Group (http://www.edg.com)

• Fuzzy parsers• RedHat SourceNavigator (http://sources.redhat.com/sourcenav)• SNiFF+ (http://www.windriver.com/products/sniff_plus)


Examining the models

• The source code models extracted by the analysers are large and complex.

• A Java system of 18KLOC contains 1600 nodes and 5700 arcs

• To examine the models we use two different techniques• Visualisation: to create a graphical representation of the models• Abstraction: to rise the level of abstraction in the model

• The goals are: • Reduction of the complexity• To generate different views of the model• Re-documentation of the system


Phase 2: Visualisation

• Goal: • To create a graphical representation of the relational model extracted from

source code

• Actions:• To visualise all the nodes and relationships of the model• To manipulate the graph in order to understand the underlying information• To create multiple views of the data in the model

• With the graph metaphor, queries on the model are translated in visual operations:

• Navigation: to browse the information in the model• Layout: to show particular relationships• Filtering: to focus on interesting data sets


Overview of Rigi

• Rigi is an interactive, visual and programmable environment to support the tasks of program comprehension and reverse engineering.

• Rigi is basically a graph editor useful for • Visualising software structures• Discovering and creating abstractions• Navigating the software models

• Extensible through the Tcl interface to support most of the graph based reverse engineering tasks.

• Developed by the Department of Computer Science of the University of Victoria

[Rigi]


The Graph Metaphor of Rigi

• The software artefacts and their relationships are represented with hierarchical attributed typed graphs

• Nodes represent software artefacts, such as functions, subsystems, classes, variables, directories, files.

• Arcs represent relationships among the software artefacts, such as function calls, attribute accesses, inheritances, directory containments.

• The Rigi domain defines the colours and types of nodes and edges that are allowed in the graph.

• A set of Rigi domains are already available for common programming languages and systems (C/C++, Java). New domains can be easily derived from the basic ones.


The Rigi Domain for FAMIXGraphs are composed of nodes and arcs.

Arcs are directed from the bottom of the node to the top of the node.

Nodes and arcs can be of various types (distinguished by colours).

CollapseSystem

DirectoryFile

ClassMethodAttributeFunction

Nodes

compositelevel

contain_dirhas_methodhas_attribute

invocationaccess

decl_fun

Arcs E.g. Method A invokes the Function B

Method A

Function B

invocation

The FAMIX domain:


The model of HelloWorld as a Rigi graph


A snapshot of Rigi


Phase 3: Abstraction

• Goal:• To rise the level of abstraction in the model in order to simplify its

comprehension and to show architecturally relevant information.

• Actions:• To group a set of concepts into a high level concept• To map a set of source code concepts to a high level relationship

• Input: • The source code model• The design documentation of the system and interviews with experts

• Output:• A high level model of the system


Grouping• Grouping is the process of selecting a set of nodes (concepts) and

collapse them into one node. • Method:

1) Identify a cluster of concepts that:- implement the same functionality- belong to the same concept in the domain model- belong to the same component

2) Group the cluster in a new entity3) Give a name to the new entity

• This process can be automated by applying common grouping patterns or by writing a Tcl script in Rigi.


Grouping

g()

c

c.m1()

var

c.m2()

contain contain

call

access

g() c+

var

composite

composite

C+

composite

composite

Rigi will take care of maintaining the correct composite arcs


Grouping the classes

• Grouping the methods and attributes of a class in a new class node

• High level dependencies are maintained by Rigi


Phase 4 - Presentation

• Goal: • To create a graphical or textual representation of the architectural model

• Actions: • To define a particular architectural view from the model (e.g. component

view, task view, development view)• To present the view with a particular notation

• Some of the possible formats are:• Rigi hierarchical graphs: for navigating the models and analysing the

dependencies• UML diagrams: for communicating and distributing the diagrams to the

architects• HTML reports: for re-documenting the architecture with hyperlinked text


Component View


Navigating the component view


Dependencies of “Load” function


The Portable Bookshelf showing Linux FS

http://www.swag.uwaterloo.ca/swagkit/


UML-like diagrams from Visio


UML diagrams in Rational Rose


HTML + SVG reports


Graph based visualization


Demonstration on the Venice case


Demonstration with the Venice case

• Case study:• Venice: graphical application written in Java (~18KLOC)

• The demo shows: 1. How to extract the source code model2. How to navigate the raw model3. How to select and filter nodes4. How to group elements using the standard grouping scripts5. How to group elements according to the design documentation6. How to navigate the high-level model

• Pre-requisites:• The user has already created the SourceNavigator project and calculated

the cross-references.


Ex 1. How to extract the source code model

1. From the Plug-ins/Famix menu, select “Extract from Source Navigator”

2. Select the project file3. Select the RSF output file4. Select the concepts to extract

(in the case of Java: Class, Attribute, Method, Directory, Inheritance, Access & Invocation)


Ex 1. How to extract the source code model

1. From the Domain button, select the “famix” domain

2. Select the command “File/Load Graph from RSF”

3. Open the RSF file “venice/venice.rsf”

4. The graph is visualised in the main window


Ex 2. How to navigate the raw model

1. Select the Filter/By Node Type item

2. Select all the node types except “Class”

3. Apply the filtering

4. From the Layout menu, select the Sugyiama layout

It’s the class diagram of Venice


Ex 3. How to select and filter nodes

1. From the previous graph, select the node “Command”

2. Select the arctype “any” by pressing the Arctype button

3. Select the menu item Select/Incoming nodes

4. Press and hold <shift> and re-select the node “Command”

5. Select the menu item Filter/Complement6. Select Layout/Spring

All the subclasses of the “Command” class are visible


Ex 4. How to group elements with scripts

1. Load the raw model

2. Select “Collapse meth & attr in classes”

3. Filter out all nodes except classes

4. Apply the spring layout Methods and attributes have been

collapsed in Class nodes


Ex 4. How to group elements with scripts

1. Create the previous model

2. Select “Collapse classes in files”

3. Filter out all nodes except files

4. Apply the spring layout Classes have been collapsed in File nodes. The

spring layout reveals the possible clusters


Ex 5. A snapshot of the design documentation

• 4 major subsystems:• ModelStorage• Application• Commands• GUI

• External libraries• JFrame• JAZZ


Ex 5. How to group according to the design docs

1. Create the previous model2. Filter out all the arcs and nodes except “File” nodes3. Group nodes according to the following rules:

• Naming conventions• The intended design• Proximity

How to group manually:1) Select the nodes (e.g.

with the mouse –multiple selections with shift)

2) Select the menu item Edit/Collapse

3) Select the new node and rename it


Ex 5. The resulting high-level model

With the library node Without the library node


Ex 5. Comparing the concrete structurewith the intended design

• The two structures are pretty similar except for two differences:

• The commands subsystem depends on Application

• The JazzVisualisation subsystem is not present in the original design


Ex 6. How to navigate the high-level model

• Select the Commands and Application nodes

• Filter out the complement of the selection

• Expand all the nodes that have a dependency with the node “Application”

• Filter out all the nodes that are not clients of “Application”

• Two methods of the class “Command” are responsible for the wrong dependency


Architecture Reconstruction


Description of a Software Architecture

“ The description of the software architecture should communicate the essential decisions that have been taken in the design of the software system ”

Categories of design decisions:• Concepts: the way we think of a system, its architectural style• Architecturally significant requirements: the major concerns that

should be addressed by a proper software architecture• Structure: components and their relationships• Texture: architecturally significant design decisions at the

implementation level, such as design patterns, coding conventions and policies.

Multiple architectural views are typically used to communicate the different aspects of an architecture (like the “4+1” model).

[Ran]


Architecture Reconstruction• What is it ?

Architecture reconstruction concerns with the task of recovering the pastdesign decisions that have been taken during the development of a software system.

• Why ?• Quality assurance upon intake, acquisition, merger• The architecture is unclear and not well documented• Original developers have left and documentation is outdated• The size and complexity of the system• Supporting Software Evolution

• Conducting informed software modifications

• What for ?• To increase our comprehension of a software system by creating mental

models of its as-implemented architecture.


Architecture Reconstruction Goals

• To extract the concrete architecture from the implementation (as opposed to the intended architecture).

• To identify the past design decisions (reverse architecting) and the new ones originated from the evolution (arch discovery).

• To analyse the dependencies among the major artefacts of the architecture (such as components, tasks, files).

• To re-document the architecture and to communicate it to the developers.

• To develop a repeatable process applicable at each build of the system.

• To gather the information for architectural conformance checking.


The output of reconstruction• Concepts: the way we think of the system, the terminology of the

reconstruction, the key architectural concepts instantiated in the other views

• Multiple views:• Component view: describing the major components, their interfaces and their

logical relationships • Task/Process view: describing the task allocation of the architectural entities and

showing the inter task communications • Development view: describing the organisation of the source code files and their

relationships (for example, include dependencies) • Deployment view: describing the physical location of components in the processing

units.• Feature view: describing the run-time implementation of a feature at a high level

of abstraction.• Organizational view: describing the organization of the development activities

(projects, programs, sites).

• Two types of analyses: static and dynamic


The Process

3. Abstraction

1. Definition of Architectural Concepts2. Data gathering

Develop

Sources of information:- Documentation- Experts- Source Code

ArchitecturallySignifcantConcepts

Extract

Low levelmodel

Source Code

1. Model Refinement2. Composition rules3. View Selection

ArchitecturalModel

DomainKnowledge Hierarchical graphs

UML logical diagramsMessage sequence chartsHTML reports

4. Visualisation


Reverse Architecting Environment


Nimeta

• Nimeta is an extension of Rigi to support the activity of architecture reconstruction.

• Nimeta supports the following features:• Convertion of graphs to GXL, Prolog, Dot, Rational Rose and HTML format• Support for the Prolog interpreter• Support for Python scripts and Relational Algerbra• Grouping based on TCL scripts• Creation of multiple views of the model

• Plug-ins:• FAMIX: SourceNavigator extractor and scripts for basic groupings• ISA: scripts for creating architectural views of ISA based systems• Other Nokia systems


Phase 1 - Definition of architectural concepts

• The goal of this phase is to recover and clarify the architecturally significant concepts that build the system:

• the building blocks of the system • the communication infrastructure that enables the components to interact

at runtime

• The outcome is a meta-model that describes: • the types of architectural concepts and their relations• the mappings between the high-level concepts and their implementation

• We need to answer the following questions:• What elements of the system are essential for implementing the features ?• What terminology is used by architects when talking about the system ?• What are the “bricks” of the system ? How do they communicate ? How are

they implemented ?


Phase 1 – Talk with the architects

• The architecture of a system is often in the people’s minds rather on the papers.

• A practical approach is:

1. Define a set of use cases that cover the key features of the system

2. Meet the architects and programmers in half day sessions

3. Ask them to explain with their own words how the selected scenarios are implemented in the system. Drawing Message Sequence Charts is typically the best and more familiar communication mean during the discussion.

4. Try to avoid: • Slide shows• Reading documents• The details (Details are usually well documented on papers and are in the code, the

big picture is the one missing…)• Discussion among the architects how the system should be implemented.

• The “First contact pattern” of [Demeyer02] could be useful in this phase


Phase 1 – An MSC drawn during a session

My Server

Application Server Sys1 API Sw Bus BusManager Server Proxy

ManagerSystem 2

Proxy

function call

::Connect

Client

Sys1.Open()

Sys2.Send()

Message

Send_msg()

Recv_Msg

Response

Response

Response

Response

Responsereturn

System 2System 1My System

My System API


Phase 1 - Examples of high level concepts

• Components:• Subsystems, applications, servers, classes, software busses, tasks, processes,

queues.

• Communication: • Function calls, RPCs, pipes, sockets, software busses, shared memory.

• Textures (micro-architectures and design patterns) should also be considered at this stage because they hide interaction patterns that are architecturally significant for the reconstruction.


Phase 1 – Example of Architectural concepts

Applicationclient

Server

*

*

*

1..*

server

server

Service

*

1..*

*

caller

**

client

1


Phase 1 – Example of Communication

ApplicationServiceServer

Software Bus

Register/UnRegister

SendMessageToServer

SubscribeEvents


Phase 1 – Schema of Architectural Concepts

File

Proxy

Task

Application

Directory

Service

contain_file

contain_dir

Server

Entity

message,event

message

call

message,event

includecontain

Interface

Functon

call

define

Source Code Concepts

Architecturally Relevant Concepts


Phase 1 - Architectural Concepts

Entities:• Directory, File, Application, AppDelegate, Server, Interface

Relationships:• Application/AppDelegate/Server message Server• Application/AppDelegate/Server subscribeEvents Server• Application/AppDelegate invocation AppDelegate• Server contain_interface Interface• Task contain_object App/Server• Directory implement Server• Directory contain_dir Directory• Directory contain_file File


Wrong selection of architectural concepts (1)

• The system is based on CORBA

• The model on the left has been generated with the FAMIX schema

• All the nodes seem to be loosely coupled but…

… the problem is that this model shows only the low-level dependencies (e.g. function calls) that are not relevant for this case!!


Wrong selection of architectural concepts (2)• High level dependencies are hidden by the

CORBA communication infrastructure (ORB)• The solution has been to:

• Restrict the analysis to the Implementation classes

• Collapse the Impl, Stub and Skel in one single entity

Client Impl

Stub Skel

ORB

Automatically generated

Function call

CORBA call

The graph shows the real dependencies among the CORBA classes


Phase 2 – Extraction strategies

• There are two strategies for analysing the source files and for extracting the relevant information:

1. Extracting the typical programming languages constructs from code• Schema already exist for typical OO languages• There are plenty of front-ends that can analyse the source code

2. Extracting domain specific information from code and other artefacts• The schema is defined in the Phase 1• Code pattern detectors and ad-hoc analysers (e.g. written in Perl)

• The two strategies are usually combined in order to extract the complete source code model of a system.


Strategy 1: Extracting typical constructs

• The main steps for creating a source code model based on the typical programming languages constructs.

1. Select/define a schema for the model. There are several choices: Famix model, Dagstuhl Middle Model (DMM). The schema can be a subset or an extension of the standard ones.

2. Choice a front-end for parsing the source code and extracting the source code model. Front-ends like Source Navigator, Sniff+, Columbus.

3. Configure the front-end in order to extract the source code model. The front-ends typically provide APIs for accessing the internal symbol tables.

4. Store the source code model in a standard format that can be imported by the visualisation and analysis tool.


The FAMIX Schema

Class

FileDirectory

GlobalVariable

TypeDef

Macro

AttributeMethodFunction

expansion

use_type

expansion

use_type

access access

access access

Function

Method

Class

has_method has_attribute

has_class

access

invocation

invocation

contain_file

contain_dir

Class

GlobalVariable

TypeDef

Macro

Function

containcontain

contain

containcontain_file

include


Strategy 2: extracting domain specific data

• The approach is to detect code patterns that represent high-level domain specific concepts or relationships to be represented in the model. The steps are:

1. Select the high level concepts that cannot be inferred with Strategy 1

2. Identify how the concepts are implemented in the system

3. Develop an approach for automatic the extraction. There are three common methods: • The UNIX “grep”: the simplest tool• A script in Perl or other scripting languages: it is a refined version of

“grep”• As a complement to the API of front-ends (e.g. SourceNavigator)

...

if (/^.*send_message\s*\(\s*(\S+)\s*,.+,.+\)/) {

...


Phase 3 -Abstraction• Abstraction is the process of enriching the model with domain specific

information that will lead to a high-level view of the system with the following goals in mind:

• Information hiding, encapsulation, compactness• Easy navigation of the model• Architecturally relevance

• The main goal with abstraction is to create a model that is architecturally significant for the developers:

• Showing important dependencies• Hiding details• Showing the “big picture” of the system

• The major issues with abstraction are:• Selecting the correct level of abstraction for showing the data• Selecting the appropriate views• Classifying and grouping the concepts in a correct way


Binary Relational Algebra

• Binary relational algebra provides us a set of axioms to manipulate the relational data and to formally specify the graph transformations.

• We rely on Tarski Relational Algebra:• Operators: ∪ , ∩, \, -1, °, id, R+, R*

• Usage: • Transform architectures (e.g. for abstraction)• Analyze architectures (e.g. conformance checking)

• Some advantages are (over other approaches like SQL, Prolog): • Formal specification of transformation rules• Support for the transitive closure• Scalable

• Approaches: • Relational Partition Algebra (Philips)• Grok (University of Waterloo)• Algebra + Python scripts (Nokia)


Example: the lift operation

• We use the lift operation to lift the low-level use edges up in the hierarchy:

H = invocation ↑↑↑↑ contain

invocation = relsIn['invocation']contain = relsIn['has_method']

H = fullLift(invocation, contain)

relsOut['level'] = containrelsOut['composite'] = HrelsOut['inherit'] = relsIn['inherit']relsOut['invocation'] = relsIn['invocation']

A

e

B D

C

f g

h i

Define the usage relationship

Print the high-level arcs as ‘composite’ arcs

Calculate the full lift


The containment relations

• The high-level contain relations are usually provided in a separate file.They relations are extracted from the design documents or prepared by the architects.

• The contain relations define the mappings between low-level concepts (such as classes, functions) and high level ones (components, subsystems)

• The file has to be selected as “Factsbase 2” in Nimeta

ssContainFile GUI_s GUI.javassContainFile GUI_s MenuToolBar.javassContainFile Application_s Application.javassContainFile Application_s Applet.javassContainFile Application_s AppModule.javassContainFile FileCommands_s SaveFileCommand.javassContainFile FileCommands_s LoadCommand.javahighContain Commands_s FileCommands_s highContain Commands_s VisualisationCommands_s

Low-level contain relation (subsystem to file)

High level contain relation (subsystem to subsystem)


Comparing the concrete structurewith the intended design

• The two structures are rather similar except for two differences:

• The commands subsystem depends on Application

• The JazzVisualisation subsystem is not present in the original design


4 - Presentation

Development view

Task view


Nokia Case Study


Nokia Mobile Phones (NMP)

• The world’s largest mobile phone manufacturer.

• Is a large company (over 1000 sw developers)

• Has dozens of products belonging to several families in the portfolio

• Has to achieve fast product development (several new products every year).

• Must continuously add new feature to the products to encourage replacement.

• Highly dynamic domain and fast changing requirements

• More than100 Millions mobile devices per year


Software Architecture Issues

• Variance factors: handset category, telecom protocol, UI, OS services, customer/country customisation

• The code base is pretty large (excess of a million lines).

• Component-based paradigm.

• Hard real time constraints (e.g. for placing a call).

• Memory and energy consumption must be minimized.

• Products are organised in families to achieve reusability• -> Need for a robust architecture• -> this generates many dependencies across platforms and products

• Development process is• Geographically distributed, and• Concurrent


Migrating from copy/paste to a Product family• Big impact on the organization (probably more than on the code itself)• Develop a precise process and take it in use

• Introduce bureaucracy

• Migration towards a component based architecture• “platformization” of assets

• Key aspects related to architecture recovery:1. Reference architecture2. Empowered architects (orthogonal to the projects) with the role of:

- preserving the architectural integrity (we can’t implement that feature)- being the experts from the requirements till the post-release bugs- assessing the impact of new requirements- controlling the implementation of feature areas

2. Logical architecture3. Keep control of the software platform4. Conformance checking -> architecture policeman


Product family architecture


Some issues

• Products share the common software infrastructure (OS, scheduling, kernel, development environment)

• Features are typically developed independently and concurrently by different development teams. A set of features are combined together to create a particular product and they are presented in the user interface (UI). This approach puts a strong pressure on the integration phase and in the design of the UI.

• Architectural reqs are handled at different levels. They are typically detected in the products and then moved higher according to their importance.

• Features are moving up and down according to their relevance• The high level architectures are inherited and instantiated with variation

by the low architectures.• Keep the machine up and running to produce a target output of 80

products per year


Architecture recovery process

Logical ViewReferenceArchitecture

SourceCode

Code AnalysisPerl Script

SourceCodeModel

RelationalAlgebra &

Python

ArchitecturalViews Visualisation

HTML Report 3D Graph ToolUML-Rose DiagramUML-Visio Diagram


reference architecture = architectural concepts• A product family usually includes product that are built with the same

architectural style.

• The style contains all rules that are relevant at the architectural level• Types of components allowed

• wheels, doors, engines, windscreens• applications, servers, delegate applications

• Communication rules between such component types • wheels have to be tied to the framework and ensure that the movement of the engine gets

transmitted to the asphalt, engine should be connected with a fuel tank, etc.• applications can ask services from servers with asynchronous messages, servers directly

manage hardware and software resources, etc.

• This is contained in the reference architecture document.• Heuristic: the reference architecture contains all that everyone needs to know about

architectural rules of the product family.

• Changes to the reference architecture are carefully assessed• Yes, it is costly to change them!!

• There are tutorials for newcomers (just like a driving course)


The reference architecture in detail

• Architecturally significant requirements• They do not strictly belong to the reference architecture, but we keep them

there.

• Architectural rules• Allowed types of components, allowed relationships, described in UML

(despite its shortcomings)

• Communication infrastructure• Implementation of the relationships between the components

• Simplified logical view of the family• Runtime architecture

• Allocation of processes to threads• Division of tasks at OS level• Feature interaction models


The logical view = intended architecture

• The logical view describes:• The components, their interfaces and their owners• The hierarchical organization of the components in packages• The logical dependencies among components and packages

• Discovery of the existing assets

• History: • Version 1: UML diagrams manually created (using “grep”)• Version 2: component inventory realized by interviewing the architects

• The final component inventory is a database that contains information about the components in the family.


The output of the recovery• Concepts: the way we think of the system, the terminology of the reconstruction,

the key architectural concepts instantiated in the other views • Multiple views:

• Component view: describing the major components, their interfaces and their logical relationships

• Task/Process view: describing the task allocation of the architectural entities and showing the inter task communications

• Development view: describing the organisation of the source code files and theirrelationships (for example, include dependencies)

• Deployment view: describing the physical location of components in the processing units.

• Feature view: describing the run-time implementation of a feature at a high level of abstraction.

• Organizational view: describing the organization of the development activities (projects, programs, sites).

• Two types of analyses: static and dynamic


Dynamic Analysis


The Feature View• The system features are understood by managers, UI specifiers,

analysts, architects, designers, testers, programmers, …

• Features represent an unifying element

• From a feature engineering perspective, the goal of reverse engineering is to discover how the features have been implemented, what are they interactions, or what are the necessary componentsfor their execution.

• This task requires to gather information at any level of abstraction but it is focused on a specific goal.

• We need to create the feature view in order to understand how the features of the products are implemented and how they can be reused among several products.


Why do we need the feature view ?

• A feature is coherent and identifiable piece of system functionality • Features are typically developed independently and concurrently by

different development teams. A set of features are combined together to create a particular product and they are presented in the user interface (UI). This approach puts a strong pressure on the integration phase and in the design of the UI.

• To avoid costly delays in the integration phase (due to feature interaction problems), 1) to identify and specify the possible feature interactions as early as possible during the design phase and 2) to analyse the implementation of the features (and their interaction) during the development of the product.

• Features represent assets that we want to reuse in other products of the family.


Our approach1) Select a feature to analyse and define a use case

2) Execute the use case on the instrumented software and collect the traces

3) Create the feature view by combining the traces with the high-level static view (typically the component or the task view)

4) Abstract the feature view by detecting interaction patterns

5) Navigate the feature view by expanding/collapsing the participants and messages of the MSC, by filtering un-relevant information and by slicing the static views with particular dynamic scenarios.

• Abstraction of the MSC according to the static analysis

• Visualisation of the MSC with support for Horizontal and vertical abstraction


The Dynamic Analysis Process

Visualisation

Static Model

Execute ascenario

Instrumentedsystem

Traces

Visualisaton tool:Rigi + MSc

Extract/Filter Abstract Present


The abstractions in the MSCA B C

M 1M 2

M 3M 4

M 9M 1 0

M 1 1

M 1 2

M 5

M 6

M 7

M 8

A B C

A B C

M 5

M 6

M 7

M 8

M 1 -M 4

M 9 -M 1 2

O r ig in a l M S C H o r iz o n ta l a b s tr a c t io n

V e r t ic a l a b s tr a c t io n

A B C

M 1 -M 4

M 7

M 9 -M 1 2

H o r iz o n ta l a n d V e r i tc a la b s tr a c t io n s

M 1 0

M 1 1

M 7

M 2

M 3


Reqs for the Dynamic Analysis

• Architecture and programming language independence

• Generation of high-level views of the structure and behaviour

• Interactive manipulation and synchronisation of static and dynamic views

• Abstraction plays a key role for reducing the complexity of the views.

• Support for an iterative process of data analysis


Reverse Architecting Environment


Horizontal abstractions


Opening the boxes


Vertical abstractions


An exampleMSC_COMMANDS

FILE

CANVAS

DIALOGS

DATA_STRUCTURES

EXECUTION

GUI

ms c_env _get

exec_initia lizeMs c

TREE_CREATE

HASH_CREATE

guis e l_initBindings

exec_Command

f ile_loadUs eCas e

ADD_PARTICIPANTS

ADD_MESSAGES

exec_initia lDraw

gui_initia lDraw

exec_ge tVis iblePartic ipants

tree_children

DRAW_PARTICIPANTS

exec_ge tVis ibleMes s ages

tree_children

exec_f ilte rMes s ages ToDis play

FILTERING_MESSAGES

DRAW_MESSAGES

gui_s caleCanv as ToFit

exec_ge tVis ibleMes s ages

tree_children

exec_f ilte rMes s ages ToDis play

FILTERING_MESSAGES

exec_ge tVis iblePartic ipants

tree_children

ms c_env _s et

GUI

CANVAS

DIALOGS

FILE

DATA_S TRUCTURES COMMANDS

EXECUTION

Model: hava_only_func.rsfAbstraction: hava_architecture.tclScenario: feature_load.msc


Key Conferences for RE

• The International Conference on Software Maintenance (ICSM)• The European Conference on Software Maintenance and Reengineering

(CSMR)• The Working Conference on Reverse Engineering (WCRE)• The International Workshop on Program Comprehension (IWPC)

• usually co-located with the International Conference on Software Engineering (ICSE)

• The Workshop on Program Analysis for Software Tools and Engineering (PASTE)

• Reengineering Forum• http://reengineer.org/

• The European Conference on Software Maintenance and Reengineering (CSMR) 2004 will be held in Tampere on 24-26 March 2004


Books on the topic

Christine Hofmeister, Robert Nord, Dilip Soni, Applied Software Architecture

Len Bass, Paul Clements, Rick Kazman, Ken Bass,Software Architecture in Practice

Paul Clements (Editor), Felix Bachmann, Len Bass, David Garlan, James Ivers, Reed Little, RobertNord, Judith Stafford Documenting Software Architectures: Views and Beyond

Mehdi Jazayeri, Alexander Ran, Frank Van DerLinden, Software Architecture for Product Families: Principles and Practice

Serge Demeyer, Stephane Ducasse, Oscar Nierstrasz, Object Oriented Reengineering Patterns


Bibliography[Chik90] Chikofsky E. J. and Cross J. H. II., Reverse Engineering and Design Recover: A

Taxanomy. IEEE Software, Jan. 1990.

[Ciupke99] Ciupke O., Automatic Detection of Design Problems, Proceedings of the 30th

Technology of Object-Oriented Languages and Systems International Conference USA (TOOLS), IEEE Computer Society Press, 1999.

[Bowman99] Bowman T., Holt R. C., and Brewster N. V., Linux as a Case Study: Its Extracted Software Architecture, Proc. of the International Conference on Software Engineering (ICSE '99), Los Angeles, May 16-22,1999.

[Datrix] Datrix, Bell Canada, http://www.iro.umontreal.ca/labs/gelo/datrix

[Davis95a] Alan M. Davis, 201 Principles of Software Development, MgGraw-Hill, 1995.

[Demeyer99] Demeyer S., Ducasse S. and Tichelaar S., Why Unified is not Universal. UML Shortcomings for Coping with Round-trip Engineering, Proceedings of The Second International Conference on The Unified Modeling Language (UML '99), Bernhard Rumpe(Ed.), LNCS ????, Springer-Verlag, Kaiserslautern, Germany, October, 1999.


Bibliography[Demeyer99] Demeyer S., Ducasse S., Lanza M., A Hybrid Reverse Engineering Approach

Combining Metrics and Program Visualization, Proceedings of the 8th Working Conference on Reverse Engineering (WCRE ’99), IEEE, 1999.

[DMS] DMS Software Reengineering Toolkit, http://www.semdesigns.com

[Ducasse99] Ducasse S., Rieger M. and Demeyer S., A Language Independent Approach for Detecting Duplicated Code, International Conference on software Maintenance (ICSM '99), Oxford, England, 1999.

[FAMIX] Demeyer S., Tichelaar S. and Steyaert P., FAMIX 2.0 - The FAMOOS Information Exchange Model, technical report, University of Berne, July, 1999, http://www.iam.unibe.ch/~famoos/FAMIX

[FAMOOS] FAMOOS, Framework-based Approach for Mastering Object-Oriented Software Evolution, ESPRIT Project 21975, http://www.iam.unibe.ch/~famoos/

[Fowler99] M. fowler, Refactoring Improving the design of existing code, Addison Wesley, 1999.

[Favre] J. M. Favre, A New Approach to Software Exploration: back-packing with GSEE, Proceedings of the European Conference on Maintenance and Reengineering 2002, CSMR 2002, Budapest, March 2002.


Bibliography[Freijs94] Freijs L.M.G., Krikhaar R.L. and van Ommering R.C., A relational approach to Software Architecture

Analysis, Software Practice & Experience, 28(3), April 1994, pp. 371-400

[FrontEndArt] http://www.frontendart.com

[Murphy] Gail C. Murphy and David Notkin, Reengineering with Reflexion Models: A Case Study, IEEE Computer 30, 8, pp. 29-36, 1997.

[GXL] R. C. Holt, A. Winter, A. Schürr, GXL: Towards a Standard Exchange Format, Proceedings of the 7th Working Conference on Reverse Engineering (WCRE 2000), Brisbane, Australia, 23-25 November, 2000. http://www.gupro.de/GXL

[Gahmy] H. M. Gahmy, R. C. Holt and J. R. Cordy, Wins and Losses of Algebraic Transformations of Software Architecture, ASE 2001: Automated Software Engineering, San Diego, California, November 26-29, 2001.

[Hof] Hofmeister C., Nord R.L. and Soni D., Describing Software Architecture with UML, Proc. of the 1st

Working IFIP Conference on Software Architecture, Kluwer Academic Publishers, 1999.

[Kazman96] Kazman R., Tool Support for Architecture Analysis and Design, Joint Proceedings of the SIGSOFT '96 Workshops (ISAW-2), ACM, 1996, pp. 94-97.

[Kazman99] Guo G. Y., Atlee J. M. and Kazman R., A Software Architecture Reconstruction Method, TC2 First Working IFIP Conference on Software Architecture (WICSA 1), 22-24 Feb, 1999, San Antonio, Texas, in Software Architecture by P. Donohoe, Kluwer Academic Publisher, pp. 15-33.

[Krikhaar99] Krikhaar R., Postman A., Sellink A., Stroucken M., Verhoef C., A Two-phase Process for Software Architecture Improvement, Proceedings of the International Conference on Software Maintenance, (ICSM '99), IEEE Computer Society Press, 1999, pp. 371-380.


Bibliography[Lehman85] Lehman M. M. and Belday L., Program Evolution – Processes of Software Change, London

Academic Press, 1985

[PBS] The Software Bookshelf, P. Finnigan, R. Holt, I. Kalas, S. Kerr, K. Kontogiannis, H. Mueller, J. Mylopoulos, S. Perelgut, M. Stanley, and K. Wong. IBM Systems Journal, Vol. 36, No. 4, pp. 564-593, November 1997. http://swag.uwaterloo.ca/pbs/

[Ran] Ran A., “ARES Conceptual Framework for Software Architecture” in M. Jazayeri, A. Ran, F. van derLinden (eds.), Software Architecture for Product Families Principles and Practice, Addison Wesley, 2000.

[Rigi] Rigi: a visual tool for understanding legacy systems, University of Victoria, http://www.rigi.csc.uvic.ca/

[Riva00] C. Riva, Reverse Architecting: an Industrial Experience Report, Proceedings of the 7th Working Conference on Reverse Engineering, Brisbane (WCRE2000), Australia, 23-25 November, 2000

[Sniff] Sniff+, Take Five Software, http://www.takefive.com

[SourceNavigator] RedHat Source Navigator, http://sources.redhat.com/sourcenav/

[Stewart99] David B. Stewart, 30 Pitfalls for Real-Time Software Developers, Embedded Systems Programming, VOL. 12 NO. 10 October 1999.

reverse engineering & architecture · pdf filereverse engineering & architecture...

Documents