distributed monitoring & mining

Distributed Monitoring & MiningOMSE PRACTICUM II FINAL PRESENTATIONJUNE 6, 2013

THOMAS MOONEY SHAILESH SHIMPI AHMED OSMAN ISAAC PENDERGASSREQUIREMENTS ENGINEER QA MANAGER/ TEST

ENGINEER ARCHITECT

PROJECT MANAGER

“”

… as software systems become more complex, automated analysis may be the only way that we can keep up with their size and scope.

Our Purpose

Project Status Architecture Google Prediction Engine Application Demonstration Project Review What’s next?

To augment current collaboration tools by collecting and modeling historical project data to alert project stakeholders when signs of trouble are detected.

Project Status Deliverables

Planned Project Proposal Software Project Management Plan (SPMP) Concept of Operations (ConOps) Software Requirements Specification (SRS) Software Architecture Document (SAD) Software Quality Assurance Plan (SQAP) User Manual Metrics Document Prediction Application

Project Status (cont.) Deliverables

Completed Project Proposal Software Project Management Plan (SPMP) Concept of Operations (ConOps) Software Requirements Specification (SRS) Software Architecture Document (SAD) Software Quality Assurance Plan (SQAP) User Manual Metrics Document Prediction Application

Architecture

Architectural Highlights – Goals and Constraints Useful to future architects and designers Written with MS .NET, but deployed with open source technologies

(Mono, MySQL). Interfaces with 3rd party APIs (Google, Assembla). Minimize the impact of anticipated changes

Changes to the Assembla API Changes to the Google API Changes to the metrics used to construct the predictive model Changes to the content of the prediction report.

Architecture – Components

Google Prediction Engine What is Google Predictive API ?

The Prediction API provides pattern-matching and machine learning capabilities. Given a set of data examples to train against.

It is using supervised learning It uses classification and regression pattern recognition https://developers.google.com/prediction/docs/getting-started

Why do we use it? We need to classify projects either in risk or in the right path Free to use (for 6 months) and has many API flavors (.NET, python …)

Google Prediction EngineHow is it working?

Create Training Data (Most Important)

Upload the training data to Google Cloud

Train the model from the data file

Start to get the prediction by API

Status Milestones Completed

Average Closed Tickets

Average Open Tickets

AROT ATP OTV MTBT MT

GOOD 2 2 2 0 3 0 0.000189 0GOOD 1 0 0 0 0 0 0 0

WARNING 8 8.857143 9 0.142857 2.955882 1.573232 0.645121 0

WARNING 6 0.75 1 0.25 3 0 0.001831 0

Demonstration

Project Review      Model Fidelity Probably the most important aspect How do we define success? Is it constant?

DMMs Criterion: 75% of completed milestones, completed on-time.

Project Review      Metrics Selection The “right” metrics are not always available. Extra care and, in some cases, additional infrastructure is required

in the gathering of the metrics and their handling during analysis. Extending the system to support more repositories increases this

challenge due to differing formats. Repository usage does not always reflect serious development

effort.

We must choose metrics based on data that has a reasonable probability of being recorded in order to build models that are useful in determining user-project status.

Project Review A Look at the Metrics Total Number of Milestones Opened Tickets per Milestone Closed Tickets per Milestone Abandoned Tickets per Milestone Average Ticket Priority Opened Tickets per Day Mean Time Between Tickets Mean Time Between High Priority Tickets

Project Review (cont.)      Organization Roles assigned on volunteer basis Vote would be cast in the event two members wanted same role Every voice held the same weight

Design and Development Infrastructure Documents housed on Skydrive Wiki Contains references to all documents and project information Mirrored code repository Continuous Integration Server

Project Review (cont.)      Communication Decided to perform duties in distributed manner

The DMM Team has never met face to face

Email supplanted forums to become main medium of communication

Relatively few misunderstandings

Always held meetings for issues that required "complex" communications

What's Next?

Increase metric relevancy and quantity

Incorporate auto-notification scheme

Provide more reporting options

Provide mechanism for user defined analysis

Questions?

DMM Home Page

Assembla Login Screen

Select Project Space

Analysis Report

PDF Report