A preliminary analysis of AAAI-99 submissions

Devika SubramanianRice University

Distribution of 400 submitted papers by area

Distribution by area (submitted papers)

17%

13%

13%

13%

11%

8%

6%

5%

4%

4% 4% 2%multiagents/agents

KR

reasoning/unc

learning

planning

vision/robotics

search

interfaces

NL

IR/IE

neural

misc

Trends in submissions

Multi-agent/agents, KR, reasoning/UAI, planning, learning are the largest constituencies.

Robotics papers are coming into AAAI; however these are mobile robot papers.

Neural computation papers are not a large part of the submissions.

Overall acceptance statistics Before SPC meeting:

– Accepts (105)– Undecided (50)– Rejects (255)

After SPC meeting– Accepts (104)– Conditional accepts (5)– Rejects (290)

15 papers accepted with 1 A/2M or R. 3 papers rejected with 2 A/1 strong R. Every other accepted paper had two As. 32 accepted papers had 3 As! (just under 30%)

Distribution of 109 accepted papers by area

Distribution by area (accepted papers)6%

7%

8%

7%

10%

6%12%

7%

7%

13%

10%

7%multiagents/agents

KR

reasoning/unc

learning

planning

vision/robotics

search

interfaces

NL

IR/IE

neural

misc

Trends in accepted papers

Search, planning, neural computation, and information extraction/information retrieval have higher than average acceptance rates.

Data on information extraction/information retrieval and neural computation unreliable because of small sample size.

Other trends in papers

AAAI still attracts the best work in planning, constraint satisfaction, search, multi-agent systems and KR.

Other trends in papers

The best work in machine learning, uncertainty, KDD, neural computation, natural language and Web agents (not including multi-agent systems) is not being submitted to the conference.

The work in mobile robotics submitted to the conference is not competitive with the work represented at the top vision/robotics conferences..

Any new ideas in the papers?

Proverb: a system that solves NY Times crosswords. A tour-de-force integration of ideas in AI.

Hybrid approaches to collaborative filtering. New extensions to Graphplan and Satplan. Active learning: analysis and implementation. Integrating the fields of constraint satisfaction and

classical planning.

Remarks gathered from SPC

Number of NLP papers submitted to AAAI is increasing; they are of much better quality than in years past. Still not the very best papers (which tend to go to ACL), but solid work at the intersection of statistics/machine learning and NLP.

Neural computation papers should be reviewed on a special track as they were this year (with two special SPC members Giles and Sun) even though the actual number of submissions from that community was small this year.

More remarks from the SPC Need to develop mechanisms to get the best work in

machine learning, UAI, agents, KDD, robotics and neural computation to be submitted to conference.

Many were concerned about the fact that most accepted papers were incremental advances.– signs of a maturing field? Or self-selection among

submissions because of reputation as archival conference? Mechanisms to detect and encourage revolutionary

work among the submissions not working well.– reflects fundamental split in community on what a

significant result is, and about the extent of evaluation needed to “prove” that an idea works.

Some SPC statistics

SPC Papers Accepts %Acceptkautzh 20 9 45kollerd 20 8 40weldd 20 7 35thruns 18 6 33.33hirschbergj 18 6 33.33sunr 15 5 33.33gilesl 16 5 31.25kaelblingl 16 5 31.25morgensternl 14 4 28.57spectorl 18 5 27.78steinl 19 5 26.32mcallesterd 20 5 25crawfordj 20 5 25forbusk 19 4 21.05boutilierc 19 4 21.05piatetskyshapirog 15 3 20porterb 17 3 17.65nayakp 19 3 15.79tambem 19 3 15.79konoligek 19 3 15.79greinerr 20 3 15velosom 18 1 5.56

Acceptance rates vary widely among SPC.Partially explains variation in acceptance rates across sub-areas.Mean=26.37stdev=10.22

0

5

10

15

20

25

30

Number of PC

0 to10

10 to19

20 to29

30 to39

40 to49

50 to59

60 to69

70 to100

Acceptance % ranges

PC statistics for AAAI-99

110 PC members

SPC statistics for AAAI-98SPC Total Accepts %acceptsfinint 13 8 61.54porterb 18 10 55.56knoblockc 15 8 53.33moorej 11 5 45.45etheringtond 18 8 44.44crawfordj 20 8 40freudere 20 8 40hendlerj 17 6 35.29epsteins 20 7 35tambem 19 6 31.58gratchj 16 5 31.25myersk 16 5 31.25greinerr 13 4 30.77boutilierc 17 5 29.41konoligek 15 4 26.67horvitze 19 5 26.32suttonr 19 5 26.32biswasg 16 4 25forbusk 16 4 25borgidaa 17 4 23.53bonissonep 13 3 23.08morgensternl 18 4 22.22lochbaumk 15 3 20pereiraf 20 4 20simmonsr 18 3 16.67subramaniand 18 3 16.67goldszmidtm 13 2 15.38hirshh 20 3 15

Acceptance ratesvary widely.

Mean=30.95stdev=12.33

0

10

20

30

40

50

60

Number of PC

0 to 9 10 to19

20 to29

30 to39

40 to49

50 to59

60 to69

70 to79

89 to100

%accepts

PC statistics for AAAI-98

204 PC members

Issues to consider for AAAI-2000 and beyond

Size of SPC and PC to get consistent reviewing standards. Ways of attracting “work-in-progress” to AAAI and to set

good evaluation guidelines for them. Reconsider need for an SPC meeting. Paper assignment to reviewers benefits from a manual

component. Consider providing electronic access to paper abstracts and to assignment software so chairs can teleconference and do reviewer assignment.

Consider accepting few papers for plenary presentation (say 20) and have all papers presented at poster sessions.

How to exist and cooperate/compete with speciality conferences.