Soloway’s Rainfall Problem has become Harder

Simon Faculty of Science and Information technology

University of Newcastle Australia

simon@newcastle.edu.au

Abstract— Discussing the use of plans in programming and in programming education, Soloway describes a programming task that has come to be known as the rainfall problem. This problem was used in a number of empirical experiments in the 1980s and 1990s, and was generally recognised as being quite difficult for student programmers. This paper reports that when the rainfall problem was recently used as an examination question in an introductory programming course, the students performed extremely poorly on it. These students are presumably no better than the many students who have been set this problem in the past, but it also appears that the problem has become harder than it was 20 years ago. For example, the problem assumes that loop-controlled keyboard input is standard, whereas in many programming environments nowadays the standard has become event-driven GUI input. As a consequence of this change, students are less likely to be familiar with loop-controlled keyboard input and with the use of a sentinel to terminate input, another feature of the rainfall problem.

While there is potential value in comparing the performance of today’s students with that in the literature of past decades, it is important to consider changes in technology that might impose a different level of challenge on the same problem.

Keywords—programming education; Soloway; rainfall

I. INTRODUCTION After more than 40 years of research into programming

education, it would be good to think that we have learnt something, and as a consequence improved the way we teach programming; yet there is much current literature indicating that students can find it difficult learning to program [1, 2, 3, 4]. This does not necessarily mean that programming is not being taught well. It seems reasonable to believe that programming is now taught to a significantly higher proportion of the population than it was 40 years ago. Perhaps at that time programming was taught just to a suitable ‘elite’; and even then there is evidence that some found it a difficult topic to learn [5, 6, 7]. Even if advances in programming education have helped to reduce students’ difficulty in grasping the topic, it is possible that the reduction has been offset by the increase in the numbers of students setting out to learn programming.

One way of exploring this line of thought is to test students on the same questions that they have been tested on

in the past, to establish whether their performance is better, the same, or worse.

To that end, it was decided to include in a recent introductory programming exam a specific problem that was described in the computing education literature in 1986 and has since been used on several other occasions.

In a paper explaining the use of plans and goals in programming, Soloway [8] describes a problem as follows: “Read in integers that represent daily rainfall, and print out the average daily rainfall; if the input value of rainfall is less than zero, prompt the user for a new rainfall”. He then goes on to explain how this problem would be solved by use of plans:

• a valid data entry plan (which loops while entered numbers are negative) is nested within

• a sentinel-controlled counter loop plan merged with a sentinel-controlled running total loop plan, which is abutted to (appended with)

• a guarded division plan (to ensure that division is not attempted if the divisor is zero), which is then abutted to (appended with)

• an output plan [8]. Soloway goes on to explain the sorts of error that

students might make when attempting to solve this problem.

Eight years later, Ebrahimi [9] used a variant of this rainfall problem in an experiment, and listed others who had used it in the intervening years [10, 11, 12]. In Ebrahimi’s version of the problem, the students were required to output the minimum and maximum rainfall values as well as the average.

In Ebrahimi’s experiment, as with the earlier ones he mentions, students undertook the task while working at the computer, and were subsequently interviewed to help determine the causes of any errors in their work. Although three-quarters of the students were in advanced programming courses, their work did indeed include errors, and the focus of the experiment was to identify the types of error that students made in recognising which plans to use and how to combine them.

The plan-oriented approach to programming education espoused by Soloway and Ebrahimi is by no means universally adopted, although it is still in current use [13]. It

2013 Learning and Teaching in Computing and Engineering

978-0-7695-4960-6/13 $26.00 © 2013 IEEE

DOI 10.1109/LaTiCE.2013.44

130

2013 Learning and Teaching in Computing and Engineering

978-0-7695-4960-6/13 $26.00 © 2013 IEEE

DOI 10.1109/LaTiCE.2013.44

130

is not the approach taken in the introductory programming course that is the subject of this paper. However, there is still potential value in determining how today’s students would fare when given the same problem to solve.

II. METHODOLOGY A version of the rainfall problem was included in the

final examination of two offerings of an introductory programming course at the University of Newcastle. The course was taught using C# in the Visual Studio environment. One of the offerings in question was at an Australian campus of the University; the other was at an overseas campus.

While C# in Visual Studio can be taught as a console-based programming language, the approach in this course has been to work from the outset with graphical user interfaces, with controls such as text boxes for input and output, and with the program being event-driven rather than serially executed. For example, to input a number the user will enter the number in a text box, then click on a button marked ‘Accept’ or something comparable. Each click of the button launches an event-handling method in which the individual input is processed.

An immediate consequence of this event-driven approach is that input is accepted not in a single loop but in a series of calls to an event-handling method. The concept of a loop to accept and process input is no longer meaningful to the students. A further consequence is that input is generally terminated not by entering a sentinel value but by clicking a different button, perhaps marked ‘Done’ or ‘Calculate’ or some such. Therefore some modifications were required to the rainfall question in order for it to appear anything like the sort of programming that the students had been learning and practicing.

The most obvious change was to replace the input loop with an array already containing the data. This immediately led to the question of what to do with negative numbers in the data. In the original formulation of the problem, negative numbers were to be ignored, with looping to continue until an acceptable value was input. For the array-based version, it was felt that while a negative number was clearly a data entry error, it presumably represented an actual day, so students were asked to regard it as a day with zero rainfall. That is, while no rainfall would be registered for this day, the day itself would be counted and used in calculating the average rainfall.

Fig. 1 shows the question as it was used in the examination, with the various further clarifications that it was thought wise to include.

III. RESULTS The exam in which this question was used was taken by

149 students at a domestic campus and an overseas campus of the University of Newcastle. Student performance on the question was poor: students averaged 52% on the exam overall, but only 23% on this question.

Fig. 2 shows students’ marks on the rainfall question plotted against their overall marks for the exam. The shape of the plot is as might be expected, with a tendency for the better-performing students overall to gain a better mark on this question; but the number of students scoring zero for the rainfall question, even when they scored more than 50% on the exam, is somewhat remarkable. Of the 149 students, 54 either failed to answer the question or wrote answers that included none of the components expected in a reasonable answer. This leaves 95 students who wrote answers with some recognisable components of the expected answer.

In case this might seem somewhat harsh, the marking rubric for the answers is presented below. Perhaps some readers will feel that they might have marked the question differently, in a way that would lead to better results. The question was worth 10 marks, allocated as follows:

(1) total initialised to 0 (3) correct loop (not a for loop), index initialised,

correct stopping condition, index incremented) (1) in loop, rainfalls >0 (or >=0) added to total (2) after loop, avoiding division by zero

A program has a one-dimensional array of integers called iRainfall, which is used to record the rainfall each day. For example, if iRainfall[0] is 15 and iRainfall[1] is 0, there was 15mm of rain on the first day and no rain on the second day.

Negative rainfall values are data entry errors, and should be ignored.

A rainfall value of 9999 is used to indicate that no more rainfall figures have been registered beyond that element of the array; the last actual rainfall value recorded is in the element immediately before the 9999.

The number of days represented in the array is open-ended: it might be just a few days, or even none; it might be a month; it might be several years. The number of days is determined solely by the location in the array of the 9999 entry.

The array pictured here shows that rainfall was recorded for five days, with falls of 15mm, 0, 0, 5mm, and 2mm.

i 0 1 2 3 4 5 …

iRainfall[i] 15 0 –53 5 2 9999 …

Write a function method to find and return the average rainfall over all the days represented in the array. A day with negative rainfall is still counted as a day, but with a rainfall of zero. The following code might help.

private int AverageRainfall(int[] iRainfall) { . . . }

Figure 1: The rainfall question as used in the exam

131131

(2) else calculate average, being sure to divide by the correct number

(1) return average.

A loop is a principal component of the expected solution. A while loop is the obvious choice for a possibly empty data stream, but this construct was chosen by only 24 students, with a further three using do loops. Nineteen of these students correctly terminated the loop on reaching the sentinel; others looped to the end of the array, or to an index of 9999.

Fifty-seven students used for loops to process the data. This is perhaps not surprising, as during the course the for loop was presented as the principal way of traversing an array. Only 14 of these students correctly terminated the loop on encountering the sentinel. Some used break to terminate the loop (something that they had not been taught in this course). Others used the knowledge that while the middle component of the for structure is typically used to indicate a stopping value for the loop index, in reality it is simply a boolean expression used to determine whether looping should continue; and thus it can be used to express more complex conditions such as whether the sentinel has been encountered:

for (int i=1; iRainfall[i] != 9999; i++)

Most of the remaining students who used for loops iterated through the full length of the array, although some took this to be 5 (the number of entries preceding the sentinel in the example) or 9999 or 10000 (confusing the sentinel with the array size).

Only 53 students initialised the rainfall sum variable before entering the loop – and one more initialised it repeatedly, in the body of the loop.

Only 32 students counted the valid days while iterating through the array.

Of the 95 students whose answers went some way towards addressing the question, fully 60 took reasonable steps to exclude negative rainfall values from the calculation, and one more wrote appropriate statements, but outside the loop.

Not one student placed a guard around the division to avoid a division-by-zero error, although one student actually appended this note to his solution:

Please note: no error check to make sure dAverage isn’t calculated if the first index contains 9999 is put in as it wasn’t asked for. This code assumes at least one day of rainfall was recorded.

It would probably have been easier to write the code than to compose this note!

Finally, 30 students took steps to avoid integer division when calculating the average, although three of these did this by calling the Math.Average method (incorrectly) rather than by doing an explicit division.

Table 1 summarises these observations about which components were used by how many students.

IV. DISCUSSION In the experiment of Ebrahimi [9], the focus was on the

types and numbers of errors that were found in his students’ code; in the exam reported in this paper, the focus was on individual students, and indeed on their marks for the question. Therefore while it is certainly possible to address the commonality of the errors themselves, there can be no direct comparison between Ebrahimi’s findings and those from this exam.

It is clear, and indeed it was expected when the exam was written, that this question would be among the most difficult ones on the examination paper. The following discussion will address why this is so, and expand on the previously discussed differences between this version of the question and the versions used by Soloway [8] and Ebrahimi [9].

A. Sentinel-terminated keyboard input Thirty years ago, data entry in introductory computer

programming was almost exclusively a matter of serial input by way of the keyboard. Soloway [8] wrote “for example, programmers have a template-like structure for reading in a stream of integers, summing them, and stopping when a sentinel value (99999) is input”. A sentinel value was used because in many programming languages the type of the input had to be predetermined by the programmer; it would

Figure 2: Student marks for the rainfall question (out of 10) against their

marks for the whole exam (out of 100)

TABLE 1: STUDENT SOLUTIONS SUMMARISED

Students 149 100% No (pertinent) code 54 36% while or do loop 24 16% for loop 57 38% loop terminated by sentinel 33 22% sum initialised 53 36% days counted within loop 32 21% negative values ignored 60 40% guarded division for average 0 0% integer division avoided 30 20%

132132

not have made sense, for example, to prompt the user with “Enter the next value, or type ‘Done’ if there are no more”, because the data entry loop would have been programmed to expect integer input, and would crash if a non-integer string were entered.

The use of a sentinel was no longer mandated when languages moved to simple string input, with the programmer choosing when and how to parse that input; and it has probably never applied in teaching contexts such as event-driven form-based programming, which make no use of loops to accept serial streams of input.

Event-driven programming is by no means the only current approach that does not use serial keyboard input. In the objects-first approach, such as with BlueJ [14], individual methods are called directly, with appropriate arguments, long before any considerations of input and output. In the media computation approach of Guzdial and Ericson [15] all of the early input consists of image or sound files, and keyboard input is a very late consideration.

For these reasons, it was decided for this exam question to describe the numbers as being in an array, and to ask the students to process them in that array.

B. Choice of loop structure When while loops are introduced in an introductory

programming course, it is often in the context of serial input. If serial input is not a consideration, the while statement is unlikely to receive a great deal of emphasis.

When arrays are introduced in an introductory programming course, their introduction will often go hand in hand with the for loop. So long as the intention is to traverse the full array, this is the obvious looping construct in many traditional approaches to the course. In a first programming course, 12 weeks long, it is possible that this is the only looping construct ever used to traverse an array.

In this context, it is not surprising that only 16% of the students used a while or do loop, while 38% chose a for loop, with most of those using the loop to fully traverse the array.

C. Loop termination The decision to replace the serial input version of the

problem with an array-based version had the unintended ramification that there are probably empty entries beyond the sentinel – although this was not made explicit in the question. It was expected that students would terminate the loop when encountering the sentinel, but many traversed the whole array. Others completely misunderstood the role of the sentinel, taking it to be the index at which to terminate the traversal.

D. Counting valid values In the original formulation of the rainfall problem the

valid data entries had to be counted. In the array-based formulation – especially once it was decided to count negative values as days with zero rainfall – the count can be

trivially determined as the index of the last entry before the sentinel. Not many students observed and took advantage of this simplification.

E. Summing valid values Almost all of the students who programmed a loop

included a statement to add the rainfall values to a progressive total. However, only 36% of the class initialised the progressive total. With some, this is presumably an oversight. However, many programming languages auto-initialise numerical values to zero. While students are often taught not to rely on this, and to explicitly initialise any variable before using its value, it is quite possible that some students overlooked this step because they know that the variable will automatically start at zero.

F. Ignoring negative values Forty percent of the students – 63% of those who wrote

some pertinent code – took reasonable steps to ignore negative rainfall values. However, once again this is a different problem from that in the original formulation of the problem. In that version the expectation is that the program will loop, doing nothing but asking for and reading new values, until a non-negative value is entered. In the new version, a negative value is to be taken as zero but nevertheless counted as a day.

The means used to ignore the negative values show the usual variety of beginning students. Some altered the negative array elements themselves, setting them to zero, then proceeded to add them to the total. Others explicitly added zero to the total (if (iRainfall[i] < 0) iTotal = iTotal + 0;) while still others used the common beginners’ device of a non-null statement with a null effect (if (iRainfall[i] < 0) iTotal = iTotal;).

G. Guarded division As mentioned in the previous section, not one student

placed a guard around the division, although one wrote a note indicating that he was aware of the possibility of a zero divisor.

Soloway [8] writes “an expert programmer knows that division must be protected since a run-time error will occur if division by zero takes place”. When Soloway was writing, many computer programming courses dealt primarily with the processing of small amounts of numerical data, and finding an average would have been a common operation. In that context, students would have been taught to check for a zero divisor.

Courses today have many focuses besides the processing of numbers, and are perhaps less likely to explain this particular problem and its solution. Those that do cover the possibility might well recommend a solution in terms of exception handling rather than explicit checks, if the course in question proceeds as far as exception handling. And finally, many current programming languages these days

133133

don’t crash on division by zero, instead returning an ‘infinity’ token such as NaN (Not a Number).

H. Avoiding integer division The possibility of pure integer division is discussed by

Ebrahimi [9], but not categorised as one of the errors he was counting. This is certainly something that was explicitly taught in this course, beginning with such apparent puzzles such as the fact that 1/2 is not 0.5 but 0. Yet it appears that many students might have relegated this knowledge to something that they might think to check only if it becomes obvious that a program is returning incorrect results, as only 20% of them took explicit steps to avoid finding the quotient of two integers.

I. The plans involved in the solution While the array-based rainfall question was clearly not

the same as the question set by Soloway [8], it was used in some expectation that it was comparable. Recalling that the original context for this question was one of plans, it is now evident that the solutions are quite different.

In particular, in Soloway’s solution the valid data entry plan is nested entirely within the merged sentinel-controlled counter loop plan and sentinel-controlled running total loop plan. That is, there is a neat self-contained while loop, within the outer loop, whose sole purpose is to ensure a non-negative value with which to proceed. In our solution, on the other hand, the plan to ignore zero values is merged with the counting and summing loops, because the counter is required to advance while the total remains unaltered. In addition, the data entry plan is replaced with an array traversal plan, which arguably places more demand on a student’s cognitive load.

Soloway [8] writes that “plan merging is an exceedingly difficult activity to carry out effectively”. As an aside, perhaps this is one reason why design by plans has not been universally adopted. Regardless, it would seem to be obvious that if merging two plans is exceedingly difficult, merging three must be more difficult still.

J. Comparison between students then and now While Ebrahimi [9] counted student errors of various

types, this work focuses on a number of specific errors. While Ebrahimi’s principal concern was to identify errors in the formulation, combination, or implementation of plans, the errors of concern in this research were at a somewhat lower level. Nevertheless, some comparison is still possible. Ebrahimi writes:

Guard IF This feature was the most common source of error … Students neglected to take into account … the need to check for special situations, such as division by zero.

Initialization

… Students frequently overlooked the initialization of variables because of the assumption of automatic initialization by compilers …

Loops Students had difficulty with what type of loop to use, how to terminate the loop, and the structure of the loop [9].

It is clear from the observations summarised in Table 1 that the students in this recent examination demonstrated exactly the same common errors. It would be easy to conclude that little has changed in programming education in the intervening years.

One possible reason for the lack of change, alluded to earlier, is that far more students are setting out to learn programming now than in the past. Teaching approaches and techniques might indeed have improved over the decades; but if at the same time programming has changed from a minority course to some kind of default choice for the masses, the improvements might not have been enough to cope with this change in the cohort.

However, it is important not to lose sight of the fact that these common errors concern plans, or standard solution steps, that were undoubtedly more prevalent in the programming of 30 years ago than they are in the many styles of programming taught in today’s classes. Perhaps the students are being better taught today, but not in the style of decades gone by.

V. CONCLUSION The decision to use the rainfall question was based upon

the notion that it would be interesting to see whether students’ capabilities had changed since the question was first studied. This turned out not to be practical, for several reasons. One reason is that Ebrahimi [9] set the question as a computer-based experiment, which is very different from using it as a question in a written examination. A second reason is that the focus and teaching context of programming courses have changed sufficiently since the 1980s that the question no longer resonates with the sort of task students are asked to do while learning to program. A third reason is that programming languages and environments have changed over time to the extent that in some courses the rainfall question can no longer reasonably be asked in its original form. The solution to the altered form of the question no longer even uses the same plans as the original, which renders impossible a detailed performance comparison of today’s students with those of the 1980s.

Nevertheless, some form of comparison has been found possible, and the general conclusion is that large numbers of students are still making the same sorts of error that they were making 30 years ago.

A further conclusion from the work, and perhaps a more useful one, is that while it might seem a good idea to re-use questions from the computing education literature of years gone by, a great deal of consideration should first be given to whether the same question really is the same question, or

134134

whether circumstances have changed enough to make it a markedly different question with a markedly different solution.

ACKNOWLEDGMENT This work is partially supported by a grant from the

Australian Government’s Office for Learning and Teaching.

REFERENCES [1] R. Lister, E. S. Adams, S. Fitzgerald, W. Fone, J. Hamer, M.

Lindholm, R. McCartney, E. Moström, K. Sanders, O. Seppälä, B. Simon, and L. Thomas, “A multi-national study of reading and tracing skills in novice programmers”, inroads – The SIGCSE Bulletin 36 (4) 119-150, 2004.

[2] A. McGettrick, R. Boyle, R. Ibbett, J. Lloyd, L. Lovegrove, and K. Mander, “Grand challenges in computing: education – a summary”, The Computer Journal 48(1) 42-48, 2005.

[3] J. Sheard, Simon, M. Hamilton, and J. Lönnberg, “Analysis of research into the teaching and learning of programming,” International Workshop on Computing Education (ICER 2008), Berkeley, San Francisco, USA, 93-104, 2009.

[4] R. Lister, T. Clear, Simon, D. J. Bouvier, P. Carter, A. Eckerdal, J. Jacková, M. Lopez, R. McCartney, P. Robbins, O. Seppälä, and E. Thompson, “Naturally occurring data as research instrument: analyzing examination responses to study the novice programmer”, inroads – The SIGCSE Bulletin 41(4), 156-173, 2010.

[5] J. B. H. du Boulay, “Some difficulties of learning to program,” Journal of Educational Computing Research 2(1): 57-73, 1986.

[6] R. D. Pea, “Language-independent conceptual “bugs” in novice programming,” Journal of Educational Computing Research 2(1): 25-36, 1986.

[7] E. Soloway and J. C. Spohrer, “Some difficulties of learning to program,” in Studying the Novice Programmer, E. Soloway and J. C. Spohrer, Eds. Lawrence Erlbaum Associates, Hillsdale, NJ, 283-299, 1989.

[8] E. Soloway, “Learning to program = learning to construct mechanisms and explanations,” Communications of the ACM 29(9) 850–858, 1986.

[9] A. Ebrahimi, “Novice programmer errors: language constructs and plan composition,” International Journal of Human-Computer Studies 41, 457–480, 1994.

[10] W. I. Johnson, Intention-based diagnosis of novice programming errors, Los Altos, CA, USA, Morgan Kaufman, 1986.

[11] J. Bonar and R. Cunningham, “Bridge: tutoring the programming process,” in Intelligent Tutoring Systems: Lessons Learned, J. Psotka, D. Massay, and S. Mutter, Eds. 1988.

[12] D. Frye, D. Littman, and E. Soloway, “The next wave of problems in ITS,” in Intelligent Tutoring Systems: Lessons Learned, J. Psotka, D. Massay, and S. Mutter, Eds. 1988.

[13] M. Hu, M. Winikoff, and S. Cranefield, “A process for novice programing using goals and plans,” Fifteenth Australasian Computing Education Conference (ACE 2013), Adelaide, Australia, 3-12, 2013.

[14] D. J. Barnes and M. Kölling, Objects first with Java: a practical introduction using BlueJ, Pearson Education, 2012.

[15] M. Guzdial and B. Ericson, Introduction to Computing and Programming in Python – a Multimedia Approach: Pearson Education, 2012.

135135