1. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. 1 Forking and coordination in multi platform development a case study April 9, 2014 Anh Nguyen-Duc Audris Mockus John Palframan Randy Hackbarth

2. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 2 Outline Introduction Objective Terminology Context Research approach Results Implications Future work 3. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 3 Introduction Rapid growth in the popularity of cross-platform software Skype, with 299 million users, is available on Windows, Linux, Mac, Nokia, Blackberry, iPhone, Android, iPad, Kindle, PS, and TV The market for cross-platform mobile development tools is forecast to reach US$8.2 billion by 2016 The existing literature: highly pragmatic with technical proposals, but few empirical studies 4. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 4 Objective To understand challenges and solutions of development and coordination for multi-platform systems In multi-platform development: 1. What are the main development challenges? What fraction of work is redundant? 2. Why do codebases diverge? By how much? 3. What are the cross-platform coordination practices? What is the coordination overhead? 4. Is the amount of parallel development related to coordination overhead? 5. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 5 Terminology Project is a set of codebases that serve for the same product Fork is a clone of a (whole or a part of) VCS repository that has subsequent development independent of the original project Modification request (MR) is a piece of work (a story, an epic, a task or a bug) that is tracked in issue tracking system, such as JIRA Commit is a set of changes to one or more files traced by Version Control Systems (VCSs). Commits to the VCS are associated with a MR identifier in the considered projects Related files are files that had identical content in the past (based on their version history in all forked repositories) Cloning MR 6. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 6 Context Spark family Started in 2005 Functionality: signaling, session manager, media, audio and video manager Hardware (Flare, iPad, iPhone, Mac, Samsung devices,1XC, VDI, ACA, AAC, IPO and LYNC) Software (iOS, android, windows) Protocol (SIP, H323, WebRTC) 7. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 7 Research method - Data collection Qualitative: 10 semi-structured interviews with different project roles, such as product manager, technical leader, scrum master and developers Projects: H323 96x1, H323 B189, Phone X6, Client platform, Flare Windows, Flare iOS and Flare Android Quantitative 232065 files from 43 codebases across Git and SVN 32008 code-associated MRs from Jira 8. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 8 Research method Data analysis Open coding Identify concepts Selective coding Measure of some concepts Logistic regression 9. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 9 Results Outline The extent of code divergence Development challenges posed by divergence Current coordination practices to cope with divergence The amount of coordination effort The reasons for divergence The amount of redundant work 10. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 10 Results Development challenges posed by divergence The extent of code divergence Development challenges posed by divergence Current coordination practices to cope with divergence The amount of coordination effort The reasons for divergence The amount of redundant work 11. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 11 Challenge 1: Increase in Technical Debt Developers create technical debt to hit important deadlines Isolated development increases technical debt as changes will eventually be required to merge Work-around bug fixes solution would need to be redone on other platforms We migrate software from platform to platform, or team to team, and this builds up technical debt. We need to continuously invest in software it is getting old even if you don't touch it. 12. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 12 Challenge 2: Duplicate development effort Feature implementation and bug fixes are duplicated across platforms Redundant code related to third-party components are more critical and harder to solve Duplicated files diverge over time Current porting and merging process limits ability to detect redundant work Some files are generated by multiple packages. This is redundant but also makes it difficult to know for sure which version of the files end up on which phone release. ... 13. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 13 Challenge 3: Redundant testing effort High level testing teams (such as SV, PV) independently create their own test cases Replication of test cases related to third-party components are more critical The PrA team now only tests for PrA code and PrB team only tests PrB code. If there is a cross delivery, both teams have to test other product as well. Hence, both test teams get hit by the same amount. 14. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 14 Challenge 4: Lack of knowledge of the whole system Large and complicated set of codebases, different set of hardware, software and protocols Organizational boundaries among teams inhibit knowledge flow Limited understanding of who does what" and who knows what" across platforms Once you move outside the core, you have different platforms, schedules, and product managers, etc. In many cases working with another project is like working with a different company so the orchestration of it all becomes problematic." 15. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 15 The reasons for divergence The extent of code divergence Development challenges posed by divergence Current coordination practices to cope with divergence The amount of coordination effort The reasons for divergence The amount of redundant work 16. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 16 Reason 1: Schedule driven Different release schedules Different development iterations the next delivery date of PrA is [Date X] while the coming software delivery date of PrB is later at [Date Y]. Therefore, while PrB is still in development phases with testing and debugging, PrA needs to be stable for delivery. 17. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 17 Reason 2: Feature variety Differences in: Hardware designs User interaction Required features Development languages if we are working for a H.323 release and we know that a release from another project introduces a major feature then probably we want to branch out before that. This is a purely risk management activity. 18. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 18 Reason 3: Coordination overload Sharing codebases requires more coordination effort Informal & formal communication Establishment of meetings Forking delays coordination needs across platforms Historically there were separate teams working on codebase PrF. Later in the project, they had been communicating too much and (hence) the codebases got separated. 19. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 19 Reason 4: Organizational differences Management needs total control of resources if they are responsible for product delivery in their organization Codebases are forked for outsourced development teams we have been divided into different teams, as the managers were divided into the pure desktop and windows team and the purely mobile teams the organization split in the different ways from 20. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 20 Results Outline The extent of code divergence Development challenges posed by divergence Current coordination practices to cope with divergence The amount of coordination effort The reasons for divergence The amount of redundant work 21. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 21 Definitions for Measures of Code Divergence R2 The extent of code divergence R1: The reasons for divergence Parallel development Cross platform development NParaFile: Files changed in parallel FracPara: Fraction of parallel files FracRelatedFile: Fraction of related files AvgFork: Avg. No. forks related to files 22. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 22 Illustration for parallel development F1 a F1 b F1 c F1 d F2 a F3 a F3 b F3 c F2 b Proj1 Proj2 Proj3 time Related files: F1-F2, F1-F3, F2-F3 Files changed in parallel: F1-F2, F1-F3 Related forks of Proj1: Proj2, Proj3 Fraction of Para development 23. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 23 How much did codebases diverge? No files under parallel development: code being moved around 1 10 100 1000 10000 100000 1000000 No.offiles Project The extent of code changes across codebases Files in project Cloned files Files in parallel In median, each project has 49.25% files being copied, 4% of files under parallel development 24. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 24 How much did codebases diverge? 1 10 100 1000 10000 100000 1000000 No.offiles Project The extent of code changes across codebases Files in project Cloned files Files in parallel Top four codebases with more than 40% files under parallel development: ngue_flareandroid.git, cpgit.git, phonex6, contextengine.git 25. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 25 The Amount of Redundant Work Redundant works MR based Commit based NRedundantMR NRedundantComm 26. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 26 Illustration for Redundant Work NRedundantMR(Proj2) = 2x(3 - 1)/3 (for F2) + 1x(2 - 1)/2( for F3) = 1.83 NRedundantComm(Proj2) = 3x(3 - 1)/3 (for F2) + 2x(2 - 1)/2( for F3) = 3 MR1 MR2 MR3 MR4 F1(1) F2(3) F3(2) F4(3) F5(1) Proj1 Proj2 Proj3 Cloning Association Divergence Filename(Commit) Modification request In median, each file associated with 0.82 redundant MR and 4.33 redundant Commits 27. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 27 Coordination mechanism 1: Cloning MR ticket Cloned MR links: tracks an issue occurs in application, roots in engine codebases tracks a duplicated issue among engine codebases coordinate trouble shooting activities across codebases coordinate bug fix activities across codebases We take a ticket from one platform branch, cloning them to another platform branch and linking them to the application feature. Basically it helps us to track back the issues 28. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 28 Coordination mechanism 2: MR review meeting Weekly meeting includes project managers, technical managers and lead developers from each participating project Cross-project MR meetings help creating a culture of regular informal communication to clarify conflict issues, cloning issues every time there is an open code inspection everyone in the team can participate . . .We have a mandatory of minimum inspection from two reviewers when submitting our code . . . . 29. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 29 Coordination mechanism 3: Role of coordinators Technical staff who address cross-platform issues, e.g.: changes in tools or process buyback planning dependencies between projects Appointed staff responsible for examining all components being delivered to other platforms . . there is a Spark engine team who look on both sides, Windows and iOS to check for a bug across the platform . . . 30. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 30 Coordination overhead: MR cloning in Spark Top four codebases with more than 60% code MRs being cloned: dca/library, webrtcengine_gme.git, ngue_flareandroid.git, fa.git Fraction of cloned MRs over total code associated MRs In median, a project has 31% of code associated MR being cloned 31. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 31 Is the amount of parallel development related to coordination overhead? R2 Extent of code divergence R5 Extent of coordination effort R6 Extent of redundant work 32. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 32 Likelihood of an MR to be cloned The amount of redundant work vs. coordination effort H1: The fraction of redundant MRs The probability of a cloned MR H2: The fraction of redundant Commits The probability of a cloned MR The extent of parallel development vs. coordination effort H3: The fraction of file changed in parallel The probability of a cloned MR Other context factors vs. coordination effort H4: Project age The probability of a cloned MR 33. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 33 Likelihood of an MR to be cloned Variables Coeff. Median 3rd Quart OR FracRedundantMR 0.4092** 0.82 1.69 1.505 FracFileParallel -0.307** 0.04 0.15 0.735 Duration -0.0007** 1109.5 1452.5 0.999 Logistic regression: Model: glm(cbind(NoClonedMR,NoMR-NoClonedMR)~ log(FracRedundantMR+1) + log(FracFileParallel+1) + Duration) Model summary: Null deviance: 4686.3; Residual deviance: 3692.9 Explain c.a 21% variation H1: accepted H2: accepted H3: reversed relationship H4: reversed relationship 34. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 34 Summary 1. What are the main development challenges? Technical debt, Redundant work and Coordination knowledge across codebases What fraction of work is redundant? In median, each file associated with 0.82 redundant MR and 4.33 redundant Commits 2. Why do codebases diverge? Schedule difference, technical variation, coordination overload and organizational difference By how much? In median, each project has 49.25% files being copied, 4% of files under parallel development 3. What are the cross-platform coordination practices? MR cloning, cross platform MR review and role of coordinators What is the coordination overhead? In median, a project has 31% of code associated MR being cloned. 4. Is the amount of parallel development related to coordination overhead? Amount of parallel development decreases likelihood of MR cloning Amount of redundant MR increases likelihood of MR cloning 35. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 35 Implications The primary value of a version control system is to allow parallel development on forks to delay coordination needs Establish formal processes for cloning MRs across codebases, such as periodic review of cross-platform issues Coordination between testing team and development team to reduce the number of duplicated test cases Automatic detection of code divergence and propagating change notifications to relevant developers 36. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 36 Future work Measure and impact of code divergence How much has a file or a project diverged? Is divergence associated with fault proneness? Is divergence associated with coordination overhead? How to organize development? Are forks more likely along the organizational boundaries? Are cross-platform coordinators also the top contributors to each fork? Are forks with more developers associated with more faults? Effective coordination practices? 37. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. 37 Questions ? 38. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 38 Backup 1: Measure definition 39. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 39 Back up 2 Quantitative description Median: 347 MRs Max: 5192 MRs (sip96x1.git) Median: 77 MRs Max: 1017 MRs (sip96x1.git) Median: 30% Max: 82.93% (dca/library) Median: 3.84% Max: 72.21% (ngue_flareandroid.git/) Median: 1101 days Max: 1858 days (dca/library) Median: 2368 files Max: 132050 files (advd_) Median: 597 files Max: 21828 files (summit_) Median: 112 files Max: 12918 files (summit_) 40. Avaya - Proprietary. Use pursuant to your signed agreement or Avaya policy. Avaya Proprietary. Use pursuant to your signed agreement or Avaya policy. 40 Back up 3: Qualitative Scheme Growing technical debt Duplicate development effort Redundant testing effort Lack of knowledge of whole system Schedule difference Feature variety Coordination overload Organization difference Cloning MR ticket Cross platform coordinator MR review meeting Development challenges Reason for divergence Coordination mechanisms Multi- platform development