Towards A Better Code Completion System by APIGrouping, Filtering, and Popularity-Based Ranking

Daqing HouElectrical and Computer Engineering

Clarkson University, Potsdam, NY USA 13699dhou@clarkson.edu

David M. PletcherMathematics and Computer Science

Clarkson University, Potsdam, NY USA 13699pletchdm@clarkson.edu

ABSTRACTNowadays, programmers spend much of their workday dealing withcode libraries and frameworks that are bloated with APIs. Onecommon way of interacting with APIs is through Code Completioninside the code editor. By default, Code Completion presents ina popup pane, in alphabetical order or by relevance, all accessiblemembers available in the apparent type and supertypes of a receiverexpression. This default behavior for Code Completion should andcan be further improved because (1) not all public methods are APIsand presenting non-API public members to a programmer is mis-leading, (2) certain APIs are meant to be accessible only in somelimited contexts but not others, (3) the alphabetical order separatesotherwise logically related APIs, making it hard to see their con-nection and to work with, and (4) commonly used APIs are oftenpresented long after much less used APIs due to suboptimal APIsorting strategies. BCC (Better Code Completion) addresses theseproblems by enhancing Code Completion so that programmers cancontrol how specific API elements should be sorted, filtered, andgrouped. We report our preliminary validation results from testingBCC with Java projects that make use of the AWT/Swing APIs.For one large project, the BCC approach reduces by over ninetypercent the total number of APIs that a programmer would have toscroll through using Eclipse’s Code Completion before settling onthe desired ones.

Categories and Subject DescriptorsD.2.3 [Software Engineering]: Coding Techniques and Tools—Object-oriented programming, Program editors; D.2.6 [SoftwareEngineering]: Programming Environments—Integrated environ-ments

General TermsAlgorithms, Documentation, Experimentation, Measurement

1. INTRODUCTIONLarge application frameworks and libraries have clearly contributed

substantially to programmer productivity and software quality. At

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.RSSE ’10, May 4, 2010, Cape Town, South AfricaCopyright 2010 ACM 978-1-60558-974-9/10/05 ...$10.00.

the same time, this kind of reuse has also created a massive amountof information and artefacts that many programmers have to dealwith. For example, the Standard Edition of the Java DevelopmentKit version 1.6 (i.e., Java SE 6) has 3,777 classes and interfaces.Over the past several years, the standard API library of Java SDKhas grown nearly linearly, on average at a rate of 356 classes andinterfaces per year [10]. As another example, in a keynote speech 1,Charles Petzold estimated that “[in the] .NET Framework 2.0[,][t]abulating only MSCORLIB.DLL and those assemblies that beginwith word System, [there are] over 5,000 public classes that in-clude over 45,000 public methods and 15,000 public properties, notcounting those methods and properties that are inherited and notoverridden.” In addition to the standard, core SDK APIs, program-mers also need to learn to use third-party, domain-specific APIs.Finally, on top of the core APIs, there are the vast amount of de-rived, nonetheless equally critical, information and artefacts suchas documentation, tutorials, wizards and recipes, and sample codeand applications for programmer consumption.

Therefore, modern programmers must master the necessary skillsto effectively work with this vast amount of information artefactsand to overcome potential learning barriers [6]. They must learn toavoid both information overloading and information scarcity. Thisis far from a trivial task, and many programmers rely on tools andfeatures to help manage these APIs and artifacts, such as the IDEs(Integrated Development Environments). Drawing from featuressuch as automated refactorings and code-assist operations, modernIDEs have certainly changed the way software is developed. IDEsare particularly critical in helping programmers deal with the pro-liferation of APIs in libraries and frameworks.

Figure 1: Code Completion can be invoked automatically or bypressing CTRL-SPACE.

One of such features that has become standard in IDEs is CodeCompletion. Code Completion regularly saves IDE users time whenthey work with difficult-to-remember or unfamiliar parts of APIs.Obviously, Code Completion as currently implemented in IDEs

1http://www.charlespetzold.com/etc/DoesVisualStudioRotTheMind.html (All URLs ver-ified on Jan. 31, 2010.)

(a) Sorting alphabetically by name all completionproposals available for the receiver.

(b) Moving two relevant proposals to the top andsorting the remaining proposals alphabetically.

Figure 2: Eclipse Code Completion (ECC, as of version 3.4) sorts completion proposals either alphabetically or by relevance.

such as Eclipse can still be improved. In this paper, we report onour experience with “Better Code Completion” (BCC), a researchprototype that demonstrates the feasibility of several such improve-ments. More specifically, we show that ranking APIs based onusage frequencies (popularity ranking) can reduce the number ofAPIs that a programmer has to browse in order to select the onethat she or he needs. BCC allows programmers to specify rules tocontrol the presentation of APIs during Code Completion. Withrules, BCC can be customized to provide a more intuitive, logicalordering and grouping of the list of completion proposals, to fil-ter out proposals that are not intended for programmers to see, andto show a proposal only in certain specified contexts. (Currently,rules are specified with a simple grammar and stored in a text file.The details will be described elsewhere.) BCC is developed as anEclipse plugin [8].

2. BACKGROUND AND MOTIVATIONThe Code Completion displays a list of class members that can

be accessed from or invoked upon a specified “receiver” object in-stance. An example in Java from the Eclipse IDE is shown in Fig-ure 1. In Figure 1, a local variable of type java.awt.Graphics2Dnamed g has had Code Completion invoked upon it by typing “g.”.Typically, hitting “.” after either an identifier, expression, or the“this” or “super” keywords invokes the code-completion process.During this process, Code Completion in Eclipse computes a listof “completion proposals”, which are either method invocations orfield accesses that could be used to complete the current Java ex-pression. If the user continues to type after the initial “.”, the listof proposals is automatically filtered such that only those whosenames begin with the typed-in letters are displayed. As shown inFigure 1, the completion proposals are listed one per line, in orderof member name, showing the member name and argument namesand types (if applicable), return type (or declared type for a field),and the first type in the type hierarchy on the path from the receivertype to java.lang.Object where the member was declared.

As of version 3.4, ECC (Eclipse’s Code Completion) has twosorting methods for completion proposals. The first, as shown inFigure 2a, is to sort the completion proposals by member name, inalphabetical order. This is usually ineffective for objects with largeAPIs, as this sorting ignores the level in the hierarchy at whichthe member is declared or overridden at. However, an importantadvantage of the alphabetical order is its familiarity and good pre-dictability (in the sense that every time Code Completion is invokedfor the same type, the list of completion proposals remains in thesame order). Any improvement to Code Completion must some-

how preserve this advantage.The second method is sorting by relevance. The completion pro-

posal computer assigns each proposal a relevance score during thecomputation process. However, relevance only comes into playwhen the expected return type of the completion invocation targetis non-trivial. For instance, if the completion process is invoked onthe right side of an assignment statement, the type expected on theleft side of the assignment statement is used to boost the relevancescores of proposals that return or resolve to that type. Proposalsthat are not relevant as described above fall into alphabetical order.An example of sorting by relevance is shown in Figure 2b.

With Code Completion, programmers are freed from having toremember all the specific details about each API. Instead, theyrely on Code Completion as a just-in-time reminder to help re-call and access these details only as they are needed. In this way,they can focus on higher level, more important design informa-tion such as relevant classes and type hierarchies. For example,for Java 2D graphics, a programmer may need to remember onlygenerally that java.awt.Graphics2D contains various APIsfor painting, such as drawOval() and drawString(), butshe or he does not have to remember the exact names, parame-ters, or the detailed semantics for each of these APIs. These detailscan be discovered using Code Completion on the fly. In particu-lar, Code Completion allows a programmer to conveniently browseand choose the relevant APIs, and offers easy access to the asso-ciated documentation, all in the context where the programmer isactively coding. Thus, Code Completion helps programmers avoidswitching work contexts and the ensued interruptions to their train-of-thoughts. In this way, Code Completion supports programmersin best utilizing their brain power so that they can focus on moreimportant information, handle larger problems, and work more ef-fectively. Over time, Code Completion also helps a programmerincrementally get familiar with the APIs that she or he is using.

Two typical scenarios can be identified when a programmer usesCode Completion to complete an API usage expression. In the firstscenario, the programmer may already know the exact spelling, ora good portion of the prefix of the API name to be used. In thiscase, she or he can just type the name or prefix as usual withoutmajor interruption from Code Completion. In the second scenario,the programmer may know only a receiver expression and a roughidea of what is needed to achieve with the receiver. In this case,she or he may rely on Code Completion as a quick, within-contextalternative to searching and browsing documentation, as shown bythe JavaDoc in the right portion of Figure 2a. We believe that thesecond scenario is the main target for optimizing Code Completion.

(a) BCC sorts proposals by position of declarationtype in type hierarchy and then alphabetically byname within type.

(b) BCC sorts proposals by popularity.

Figure 3: BCC sorts proposals by the type hierarchy or by popularity ranking.

Since Code Completion is an interface through which program-mers view and access APIs on a daily basis, it is important to de-sign it to be highly usable. The usability issue is especially crucialwhen the number of APIs to be viewed by a programmer becomestoo large (say, more than twenty). In such circumstances, it can bebeneficial if a programmer can see the needed APIs as early as pos-sible. While reaching this goal, it is necessary to keep the program-mer remaining oriented in the API space, which can be achievedby adding some kind of logical structures into APIs. It can also behelpful if Code Completion can filter out APIs that are irrelevant tothe programmer’s current task. In these ways, the number of APIsthat a programmer has to spend time on is reduced.

3. BCC: BETTER CODE COMPLETIONAt the core of any type-based Code Completion system is an en-

gine capable of computing the static type of a receiver expressionthat a programmer is currently working on. All APIs from this typeand its supertypes that are available for use in the current code con-text, which are also known as completion proposals, are presentedto the programmer in a popup pane in a certain order.

The presentation of the popup pane can be customized by filter-ing and sorting the list of completion proposals in different ways.Other than the two sorting methods introduced in Section 2, thedefault ECC offers very limited options for user customization.The goal for BCC is to provide programmers with more controlover how code completion proposals can be grouped, filtered, andsorted. The main design ideas of BCC are described as follows. 2

3.1 Grouping APIs.First, BCC allows programmers to specify which methods should

belong to the same group. Methods of the same group will alwaysbe displayed together in the Code Completion popup pane. For ex-ample, the add()/getComponent()/remove() methods in java.awt.Co-ntainer can be grouped together logically as they control how childcomponents are added or removed from a container widget. Agroup can span multiple classes in the same type hierarchy. A spe-cial kind of grouping is by the type where APIs are defined (group-ing by types).2The BCC implementation for the most part is a straightforwardcustomization of the modular design of ECC. Thus in this paper wewill focus mainly on the design ideas.

We hypothesize that the grouping mechanism make it more ef-fective for programmers to learn and use APIs than without as itcan be used to present APIs logically. Moreover, we believe that byadding logical structures into the API space, grouping helps main-tain the predictability of the popup pane. This is important becausebefore any new way of sorting completion proposals other than thealphabetical order can be used, it must ensure that the popup panedoes not become too chaotic to the programmers.

3.2 Sorting APIs.Second, BCC provides two more options for sorting completion

proposals than ECC.

• Type-hierachy-based sorting.BCC sorts the list of completion proposals in the order fromthe declared type to the root in the type hierarchy (java.lang.-Object), then alphabetically within each type. An exampleof this behavior is shown in Figure 3a, where methods fromJButton are shown at the top of the list before AbstractButton,unlike the default Code Completion ECC shown in Figure 2,which mixes up methods without considering their relativepositions in the type hierarchy.

• Popularity-based sorting.The frequencies, or popularity by which APIs have been in-voked statically in source code can be used to sort APIs.The more frequent an API is used, the earlier it should ap-pear in the popup pane. API use frequencies can be dynami-cally computed or statically configured. Static configurationcan be based on information gathered from “representative”projects that use the same APIs. We are currently experi-menting to determine which approach, or combination of ap-proaches may yield the best possible solution.An example of this behavior is shown in Figure 3b, wheretwo methods are shown at the top of the list before othermethods for JButton and AbstractButton. This is becausethese two methods were used and their popularity has beentaken into account when BCC sorts the APIs.

3.3 Filtering APIs.Finally, BCC also allows users to defined context-sensitive filters

to filter out completion proposals that is certainly irrelevant in thecurrent coding context. Three types of filters can be applied.

(a) BCC filters out public method Upda-teUI().

(b) BCC filters out public method Con-tainer.add() for JLabel.

(c) BCC keeps add() available for JPanel.

Figure 4: BCC API filters.

• Private Filter.The class javax.swing.JComponent contains the public methodupdateUI(). Although public, this method is not intended tobe directly accessed by client code. It is made public be-cause the Swing artifact(s) invoking this method are locatedoutside the package javax.swing.BCC can be configured to filter out such non-API publicmethods (Figure 4a).

• Private Filter With Receiver Exceptions.Single inheritance in Java has been shown to generally leadto cleaner design, but it can have negative consequences too.One example is that class javax.swing.JComponent inheritsfrom java.awt.Container. The designers probably made thisdecision to avoid the code duplication that would result fromhaving each “true container” class redefine the exact sameContainer methods. However, this means that non-containerclasses like JLabel also inherit from Container, even thoughJLabel is probably never intended to be a container. Thisfilter makes methods inaccessible (Figure 4b) except uponcertain subclasses (e.g., JPanel in Figure 4c).

• Subclass Only.The public paint() method in class java.awt.Component shouldgenerally be called by an instance’s own repaint() method, orfrom within a subclass of Component that wishes to extendpaint(). With BCC, paint() can be made accessible only fromwithin subclasses of Component.

4. PRELIMINARY VALIDATION AND DIS-CUSSION

Table 1: Java Projects Used in BCC Validation.

Projects #API Calls #Unique CallsLAPIS 1944 562OpenSwing 6865 721SweetHome3D 4368 674Swing Tutorial Examples 2629 547SwingX 8233 1250jEdit 5339 734jide-common 11032 1007zeus 1036 261netbeans 151567 2928

We have tried to empirically answer two validation questionsabout BCC. Our first validation question is whether each of BCC’scustomization strategies outlined previously in Section 3 may ac-tually lead to any improvement over the default ECC. Our secondvalidation question is which combination of BCC’s customization

strategies may yield the highest improvement. The second questionis the most important as it will help determine how BCC can be de-ployed in practice. We use the rank of an API in the popup pane asthe metric of improvement, with a value of 1 for the top API, 2 forthe second API, and so on. If the adoption of a new strategy leadsto statistically smaller overall API ranks for all projects than thoseof ECC, then it is considered a better strategy.

For our validation, we decided to test BCC against the Java AWT/Swing APIs. Table 1 depicts the nine projects used during our val-idation. These include GUI frameworks that extend Swing andAWT, as well as applications that use Swing and AWT. For eachproject, we gathered the number of Swing/AWT API calls as wellas the number of distinct APIs called by the project. In particu-lar, jide-common, OpenSwing, SwingX, and the Zeus Java SwingComponents library are UI frameworks that invoke a total of 27166times of Swing and AWT methods. jEdit, LAPIS, SweetHome3D,and Sun’s Swing Tutorial Examples are applications that make useof Swing and AWT by invoking a total of 14280 times of Swingand AWT methods.

We configured BCC with different combinations of strategies.For each configuration of BCC, we ran it over each validation projectand gathered the total ranks. This is done by a program that visitseach method call and programatically invoking BCC to performcode completion, which, as a result, computes an ordered list ofcompletion proposals. The position of the method to be called inthat list is added to the total rank for the project. Table 2 shows apart of the results of our validation so far. The first column lists theprojects. The second and third columns contain the ranking resultsfor ECC’s alphabetical order and ECC’s by-relevance order. Thenext four columns are for four of BCC’s configurations.

Table 2 shows that all of BCC strategies significantly improveover ECC’s alphabetical order and by-relevance order. In particu-lar, the “ranking only” column used only popularity-based rankingstrategy (the ranking is computed dynamically; every time an APIis used, its use count, or ranking increments by 1). Table 2 showsthat in total, the adoption of this strategy alone reduces 80.19% overECC’s alphabetical order and 74.10% over the by-relevance order.

Table 2 also shows that the combination of type-hierarchy-basedorder, filtering, and dynamic ranking yields the highest percent-age of overall rank reduction (the right most column, 88.22% and84.60%). (The “type” strategy represents “type-hierarchy-basedsorting” described previously. When combined with the type-hiearchy-based order, the popularity-based order has a higher priority.) En-couraged by this result, we applied this combination to a largerproject, netbeans, and the percentage of reduction goes even higher(the bottom section of Table 2, 91.96% and 90.23%).

Note that Table 2 shows only part of our overall results. For ex-ample, we tested the individual effects of the type-hierarchy-basedsorting and the user-defined grouping on ranking. These results

Table 2: Partial results of running ECC and BCC over validation projects. The “ranking only” column used only popularity-based rankingstrategy, and the “type” strategy represents “type-hierarchy-based sorting”. netbeans is added only recently.

Projects alphabetical relevance ranking only ranking+filtering ranking+type ranking+type+filteringjEdit 619294 469516 104375 84436 72549 61102Jide-common 824731 568139 163536 135446 133308 114654LAPIS 182356 132981 49037 38494 27831 23895OpenSwing 884303 711841 158490 129528 115456 98952SweetHome3D 370844 286319 75136 58720 47438 39157Swing Tutorial Examples 395301 340554 71544 57918 35188 31136SwingX 698256 522218 164982 135875 123036 105812zeus 174136 141989 34844 27535 16556 14137Total 4149221 3173557 821944 667952 571362 488845% reduction from alpha 23.51% 80.19% 83.90% 86.23% 88.22%% reduction from relevance 74.10% 78.95% 82.00% 84.60%netbeans 23924204 19706197 1924340% reduction from alpha 17.63% 91.96%% reduction from relevance 90.23%

are not shown. However, it is clear from Table 2 that the type-hierarchy-based sorting adds additional value to the popularity-basedsorting strategy.

We are experimenting with the static configuration of the popularity-based ranking, hoping that a universally effective popularity-basedorder can be found. But it seems that a combination of static anddynamic popularity-based approaches and the other grouping andfiltering strategies would lead to the ultimate best configuration.

5. RELATED WORKThe two most closely related work are [9] and [1]. Both incor-

porate additional information and knowledge (program history andAPI go-togetherness, respectively) to help API consumers more ef-fectively select APIs. Both work try to reduce the number of APIsthat a programmer has to go through before settling on the one sheor he needs. However, none of them have adequately addressedthe issue of how to present the recommended APIs in the origi-nal Code Completion popup pane, or the related potential usabilityproblems. New contributions of BCC include type-hierarchy-basedsorting, grouping, and filtering.

API patterns can be useful information aids to programmers inthe form of an additional view showing the recommended APIs [1,11]. Such tools can be helpful to a programmer engaging in thebroader design exploration task, which requires deep thought andreflection, and tends to take longer time than using Code Comple-tion to insert a method call. For design exploration, design quality,rather than coding speed and work efficiency, is of paramount im-portance. Although some developers may use Code Completion fordesign exploration, it is not necessarily the best tool for that task.

Other related tools that are aimed at improving coding efficiencyin the code editor include keyword programming, abbreviation basedcompletion, and automated method completion. Keyword program-ming takes words that may appear in an API as input to create anexpression [7]. In this way, it frees a programer from remember-ing the specific API names and may reduce a programmer’s mem-ory load. Abbreviation based completion speeds up coding effi-ciency by using abbreviated input to query syntactic valid codesnippets [2]. Automated Method Completion exploits code similar-ity to recommend code templates similar to what the programmeris working on in the code editor [3].

BCC uses API popularity to sort the APIs for Code Completion.API popularity has been proposed for informing the consumption

and production of APIs [5, 4].BCC’s API filtering mechanism filters out public methods that

are not APIs as well as APIs that are meant to be used in only lim-ited contexts, such as in a subclass. Interestingly, the issue of non-API public methods seems to be important enough that a proposalto extend Java language with the so-called “superpackage” modu-larity mechanism is being worked on as part of the Java communityprocess 3.

6. REFERENCES[1] M. Bruch, M. Monperrus, and M. Mezini. Learning from

Examples to Improve Code Completion Systems. InESEC/SIGSOFT FSE, pages 213–222, 2009.

[2] S. Han, D. Wallace, and R. Miller. Code Completion FromAbbreviated Input. In ASE’09, 2009.

[3] R. Hill and J. Rideout. Automatic Method Completion. InASE’04, pages 228–235, 2004.

[4] R. Holmes and R. J. Walker. Informing Eclipse APIProduction and Consumption. In ETX’07, pages 70–74,2007.

[5] R. Holmes and R. J. Walker. A Newbie’s Guide to EclipseAPIs. In MSR ’08, pages 149–152, 2008.

[6] A. J. Ko, B. A. Myers, and H. H. Aung. Six LearningBarriers in End-User Programming Systems. In VL/HCC,pages 199–206, 2004.

[7] G. Little and R. C. Miller. Keyword Programming in Java.Autom. Softw. Eng., 16(1):37–71, 2009.

[8] D. M. Pletcher and D. Hou. BCC: Enhancing CodeCompletion for Better API Usability. In ICSM’09, pages393–394, 2009. Tool Demonstration.

[9] R. Robbes and M. Lanza. How Program History CanImprove Code Completion. In ASE’08, pages 317–326, 2008.

[10] Y. Ye, Y. Yamamoto, K. Nakakoji, Y. Nishinaka, andM. Asada. Searching the Library and Asking the Peers:Learning to Use Java APIs on Demand. In PPPJ, pages41–50, 2007.

[11] H. Zhong, T. Xie, L. Zhang, J. Pei, and H. Mei. MAPO:Mining and Recommending API Usage Patterns. In ECOOP,pages 318–343, 2009.

3JSR 294: Improved Modularity Support In the Java ProgrammingLanguage. http://jcp.org/en/jsr/detail?id=294