1

ACTION, AGENCIES, ADVOCACY, AND ANALYSTS’ Victor patthews and Leonard Apedaile

Generally speaking, human beings dislike admitting f laws in their actions. The complexities of regional development provide ample opportunity for flawed action. As public intervention in the process o f economic and social development is on the upswing, serious attention must be paid to evaluations i n order to minimize f laws in the programs. This paper examines three constraints to evaluation: friction generated b y data collection; absence of precise goals and modus operandi coupled with continuous program modification; and role Fonflict within the evaluation team. The difficulties with data lie i n the definition of regional boun- daries and hostility of the people resulting from normlessness and gouern- ment involvement. Regional survey research centers are suggested to mitigate the problem. The second constraint erists because of the sensitivity of program administrators to critical evaluation. T h e y react with vague, general, and illusory goals, discounting and negation of results, and the concept of a pilot study from which “lessolrs learned always just i fy a program.” Long-run conditioiiing and public relations m a y be the only effective policies to relieve this constraint. The third coiistraint of tole conflict has always existed but is particularly important when changes in programs derive ,from their evaluations. The solution is for our pro- fessions to insist on full public disclosure of results coupled with triyearly turnover of evaluation personxel. De facon gdne‘l-ale, les humains de‘testent auouer leurs fautes. Le dduelop- pemeiit regional i tant tr6s complese, beaucoup d’actions enterprises pounont s’avkrer dhfectueuses ou ne‘fastes. L’nteruention publique dans le processus de ddueloppemeiit dconomique et social se faisant peu fre‘quente, une attention sphciale et constante doit Ctre apporte‘e aux dvaluations visant. ci mininriser les errertrs dans les programmes. Cet article examine trois contraintes propres ci l’e‘valuation: obstacles creds par la cueilette des donne‘es; absence de buts prds et de modus operandi joints ci une modification continuelle d1r programme; et conflit de r8le dans I’e‘quipe d’dvaluation. Les difficirltds avec les donnhes sont lie‘es d la ddlimi- tation des rdgions et it l’hostilith des gens rdsultant de l’absence complBte de normes et de l’ingdrence gouvernementale d a m la vie prive‘e. Pour re‘soudre le problPme, on suggkre des centres d’enqudtes et de recherches re‘gioiraux. La seconde contrainte vient d u fait que les administrateurs de programme sont sensibles ci une &valuation critique. 11s rdpondent auec des buts vagues, gkndraur et illusoires; ils ne nient pas les re‘sultats mais n’acceptent pas le concept d’une dtude pilote 6u “Les rdsultats obtenus justifient toujours un plan d’ame‘lioration.” U n conditionnement ci long terme et des relations exte‘rieures peuvent t tre les seules politiques efficaces pour parer ci cette contrainte. La t r o i s i h e , due au conflit de rrjle, a toujours eristh mais elle est particu- likrement importante quand les modifications de programme de‘riuent de

1 Theresearch for this paper is under The University of Alberta’s contract number 55-08037 with the Alberta Human Resources Development Authority. An earlier version of this paper was read at meetings of the Society for the Study of Social Problems in Washington. D.C., August. 1970.

VICTOR MATHEWS is Assistant Professor, Department of Sociology, and LEONARD APEDAILE is Associate Professor, Department of Agricultural Economics and Rural Sociology, The University of Alberta, Edmonton.

2

Ieurs Cualuations. La solution serait que Ies associations professionnelles insistent sur une divulgation complhte des thstultats au public ainsi qu’un renouuellement du personnel d’iualuation tous Ies trois ans.

With the growing complexity of an industrial society has come an increase in the responsibility of governments for the welfare of their population. One result of this responsibility has been public intervention in regional economic and social development processes. This intervention has been the subject of some controversy with respect to orientation, goals, means, and consequences. Legislators and treasury officials have insisted more and more upon tangible results to justify both public spending and public imposition upon the private lives of persons within the intervention areas. These concerns have led to research into the impact of public interventions in economic and social processes.

The role of the researcher within the intervention may be described as pseudo-legitimate; that is to say, the researcher must be an outsider to evaluate objectively, but an insider to understand the program and to properly gain access to a full range of information. This role may threaten program personnel because the researcher’s recommendations may terminate the program and his actions cannot be controlled at the local level.

The position of the researcher within the program is frequently made all the more tenuous by serious constraints within the research environment. We suggest that the researcher evaluating a social action type of regional development program meet three constraints in his interaction with the program and its operating agencies. The first constraint is the friction generated by data collection from social and economic units within the region. The second results from the lack of precise goals and modus operandi for the program and continual program modifications. Ideally these modifications stem from feedback to program officials from evaluation recommendations but realistically are probably politically motivated. The third con- straint involves the researcher’s own inner conflicts arising from his dual role as a scientist-evaluator and as a concerned citizen.

Social action generally focuses upon a region because that region has been deemed in some way disadvantaged and, therefore, in need of change. When dealing with traditional social problems-alcoholism, drug abuse, racial strife, etc.-criteria for defining the social problem are relatively concrete and are based upon a consensual and ex- plicit rationale [3, 1970; 9, 1965; 14, 19661.

However, selection of a region for a social action program need not be based upon an explicit rationale, and the criteria by which such regions are selected are not usually consensual. Selection is made primarily by a group with power that perceives the region as

3

deviating from a norm and expects the application of a social action program to induce conformity. Fanfare usually accompanies the an- nouncement of a social action program for a particular region. A label is applied to the region in much the same manner as a label is applied to an individual. It appears (based upon author obser- vation) that in labeling a region deviant one implies the negation of the rights of that region in much the same way that the labeling process affects individuals. For example, in discussing stigma Goffman [5, 19631 states that “the implication . . . is that the stigmatized individual is a person who can be approached by strangers at will, providing only that they are sympathetic to the plight of persons of his kind.”

Because of this process of defining the region and of the gross data needs at the regional designation stage, social action areas are frequently over researched and, therefore, hostile to evaluators. These conditions occur when specific, reliable information about the region must be gathered from sources within the region.

Social action programs depend upon the perceived needs of the region. Many different agencies participate in a social action program, and each agency needs to collect data relative to its unique interests. It is puzzling that public records and documents (census data, em- ployment information, income distributions, education records, etc.) are often ignored, that there is a lack of coordination among agencies collecting information, and that there is an almost paranoid hoarding of collected data [8, 19653. The evaluator who approaches such a region frequently finds a populus with definite attitudes toward participating in the data collection process, a populus to whom the evaluator’s neutrality implies a lack of empathy which appears as condescension. The nature of the first constraint in evaluative re- search is, in one word, hostility.

This hostility arises from two frequently interrelated sources: normlessness and government involvement. Normlessness occurs when residents of the community become aware that their way of life is under consideration and is the focus of change, but they have no concrete reference point for that change. Frequent subjection to surveys that ask identical or similar questions contributes to norm- lessness. It is also fostered by the failure of resident participation to be of any perceived value to the residents or to the social action program.

Government involvement in social action programs generates hos- tility for the following reasons:

(a) The rationale for a particular question, or the general justi- fication for a survey, is frequently summed up by the phrase “government wants to know”; the government is treated as

4

a nonrational entity that moves in mysterious ways to perform or fail to perform its tasks.

(b) The community social action program assumes political over- tones creating tension and division in the community.

(c) Individuals may be opposed to the political party in power. (d) Individuals may resent government involvement in their social

and economic lives. (e) A social action program imposed upon a community from the

top causes resentment; the local people often have little or no say in whether or not the program is wanted. The program may thus be perceived as “prying” into private lives.

( f ) Finally, the lower economic class of the community-those to whom the social action program is primarily oriented- depends upon the government for support.

Government involvement and normlessness result in false answers to questions and in a propensity to avoid giving any required infor- mation. Attempting to collect data under these circumstances is diffi- cult. Either individuals will provide the “pat” or necessary answer to a question to avoid trouble with the agencies involved, or they will view the attempt to collect data as “government prying” and will be secretive and illusive about the requested information.

These difficulties with the collection af program information should be understood and overcome prior to making evaluative state- ments. Attention should be focused on data collection methods other than direct respondent contact for determining the reliability and validity of respondent-provided information. Although the success of evaluation frequently requires the use of unobtrusive data collec- tion techniques [18, 19661, the evaluators must be careful to remain anonymous while collecting data by these means. Discovery may result in their being labeled as spies and may increase hostility within the region.?

The second constraint is imposed by the valuations and recom- mendations for change made by the evaluator. The evaluator should realize that the members of social action agencies, especially those involved in a controversial social action program, are aware of any disputes and are anxious to maintain their positions within the bureau- cracy. To these individuals the evaluator poses a threat, for if a project is judged to be unsuccessful, the positions of its program members may be jeopardized. When confronted with this situation, members of a social action program may use certain techniques to avoid re- sponsibility for the program. They may be vague and general about the

2 The evaluatm of a recreational development in Alberta used a campsite overnight without identifying himself. When the campsite attendants discovered the visit. he was accused of spying. and a user survey which depended on the attendants’ co- operation was jeopardized.

5

goals of the program and the methods by which these goals may be obtained. For example:

The major objective . . . is to promote change, through demon- stration and research, in the character of human service rendered by governmental, private and public institutions by making such services as effective and economic as possible so that its recipients are more highly motivated to participate in community affairs [17, 1963, p. 111.

Or with respect to the recent Alberta,/DREE agreement for the Lesser Slave Lake area:

Provision made for incentives to primary and secondary industry to locate in an area with resource potential thus creating job opportunities especially if these were coupled with educational and social adjustment programs enabling people of the area to take advantage of the opportunities created [sic] [lo, 19703.

Grandiose ideals present .problems to the implementor of the social action program and may serve to discourage program implementation. How does an implementor “promote change in the character of human service”? In the absence of consensual means for attaining this goal, a multitude of programs could be contrived. Indeed, simply because a program must be implemented, attention may be focused upon some visible facet of the community which may or may not be related to the ideal expressed by the goal. For example, the goal of promoting the general well-being of the community might be interpreted as re- moving people from the welfare roll.

How can the success of a program with such content be evaluated? The empiricist finds that there is no precise definition of the informa- tion sought by the agency and that the objectives of the agency are in a continual state of flux. If the evaluator ultimately fails to under- stand the interrelationship between the general ideals and the specific projects of the program, his competency may be questioned. Even when the interrelationship is understood (although not necessarily accepted as valid), certain problems arise. As long as the evaluations produced by the research support the social action program, very few problems arise. The existence of supportive evaluation, even of an extremely limited nature, enables the initiators of the program to claim success in attaining their goals. To be objective, however, the evaluator must ask two questions: “What are the necessary variables and considerations pertaining to the success of a given project?” and “HOW is it possible to attribute ‘success’ o r even ‘failure’ to a given project?’’ For without specified objectives, specified means for obtaining these objectives, and specified criteria through which success or failure can be assessed, a clear-cut, viable position regard- ing evaluation cannot be taken. Yet evaluation is an essential element

6

in the social action process and must occur! Suppose that the evaluation has been completed and does not

generally support the program. That part which endorses one aspect of the social action program is generalized by the administration to support all aspects and participants. Conversely, the evaluation which rejects any or all aspects of a program is heavily discounted or even negated. Evaluation findings may be neutralized or negated by using many techniques, three of which follow.

A first technique involves the suggestion that the purpose and orientation of the program were either unknown to or not completely understood by the evaluator. The evaluation of a social action project in Alberta provides an example. After much searching through the records of the program and questioning of the staff, the presented consensus suggested that no written statement of the purpose of the project or of its plan of attack was available. Considerable time was devoted to ascertaining and understanding the goals of the project. The statement of aims finally produced was approved by the staff involved in the project, and the subsequent evaluation was based upon this statement. The evaluation, although generally favorable, expressed doubts about several aspects of the project. At the formal presentation of the evaluation results, one of the head administrators of the program produced a statement of the orientation and goals of the project, a statement which had been formulated before the research was initiated but the existence of which earlier had been denied by this same individual.3 The fact that the evaluator was unaware of this statement enabled this individual to suggest careless- ness and lack of thoroughness in the assessment of the project. Con- sequently, the findings were negated by pointing out that the evalu- ation had not even addressed itself to the orientation and means of the project as originally conceived.

A second means of negation involves the selection of variables that define the focus of the evaluation research. The evaluator is a prisoner of his science and must rely upon the established theoretical knowledge of his discipline. The number of possible (or actually occurring) consequences and the selection of variables is primarily a function of the evaluator’s mode of perception. This perception, of course, depends to a certain extent upon the nature of the project; but because the evaluator is seldom a participant in the social action program from its inception, it is primarily a function of the current attitudes of the researcher and the current state of the development

3 According to the evaluator the actual contents of thls document were not made known to hlm. It mleht hive been an early statement uf the general nature of the Program, in which case the contents were vague and did not dellneate specific goals and means. or it might have been a current statement. In any case. the problems of communlcation within the hierarchy for social action programs may result in such misunderstandings.

7

of his science. When the evaluation report is written, the considera- tions, the focus, and the variables selected by the evaluator present an image of the evaluator himself. If the focus of the initiators differs from that of the researcher, it may justly be pointed out that the re- searcher’s conclusions are invalid because he did not consider certain aspects of the program that were important to its policy makers; that is, certain variables were excluded while others were included. Although a given type of consideration (i.e., a theoretical evaluation) may be useful, what may have been desired was a practical rather than theoretical evaluation or a clientele study rather than a region- wide evaluation. The criterion for success of evaluation is not the degree of accuracy in its interpretation, but the degree to which the conclusions are plausible to its recipients [lS, 19701.

A third technique of negation may be called the “pilot study approach.” Establishing a pilot program enables program admin- istrators to ascertain the effectiveness of a particular type of action. Labeling the project “pilot” suggests experimental ion and innovation. A pilot social action program can be based upon a number of pilot projects, such as in the Alberta Census Division 14 program.’ This approach is extremely safe for program personnel. Even if a project is a total failure, one can always maintain that experience and know- ledge were gleaned from the failure-experience such as working with the social action process and knowledge about the types of projects or techniques which are ineffective. Old pilot projects typically fade away through piecemeal defunding.

What are the implications of the pilot approach? The program cannot accomplish any major goals because it lacks the full support of the social action program. Secondly, the staff need not concern themselves unduly with success or failure-their positions are rela- tively secure in that failure does not imply deficiencies in their performances-but they can rather focus on problems in the inter- pretation of the social action process. Thirdly, it is not unusual for a five-year social action program to mean five years spent modifying five different pilot projects of limited scope, funding, and impact. The limited successes and widespread failures of pilot projects may result from “self-fulfilling prophecies” [ 13, 19571. Finally, the approach runs a heavy risk of cheating the participants in the program. Initia- tion of social action involves people who participate in good faith without understanding the limited commitment of the program to their problems.

~~

4 This program was initiated in 1964 and was subsequently funded for nearly $7 million to 1971. The authors of this article are part of a team to design an evaluation SYStem and to evaluate the program.

8

In short, the second constraint on the researchers is imposed by action within the program, the agencies involved, and the analytic methods employed. Despite objectivity and accuracy of evaluation, insecurity of the agency, imprecise goals, and pilot program action encourage discrediting tactics against the evaluator and his product. On the other hand, acceptance of the evaluative results and their incorporation into the program soon create a vested interest for the researcher. The second horn of this dilemma is the third constraint imposed upon the evaluator.

This third constraint involves the dual role of the evaluator as a scientist-researcher and as a concerned and responsible citizen. There is a vast literature dealing with this topic including Mannheim [ll, 19431, Weber [19, 19461, and Merton [13, 19571. This constraint is concerned primarily with the role of the evaluator as advocate. One of the distinctions between the pure and the’ applied scientist is drawn in terms of purpose, the former being concerned with the description and understanding of the social world, the latter with control of that world [6, 1961, p. 741. As an evaluator the researcher is partially responsible for making recommendations for changes in addition to questioning the nature of the organization. These recommendations may be of the pure “if . . . then” type without advocacy. The evalua- tion game, however, seldom allows the researcher to escape advocacy.

In evaluating a particular project, the researcher may note that a certain policy is detracting from the effectiveness of the project. His evaluation reports may state that if such a policy is continued, certain results can be anticipated. The initiators of the project may then request recommendations from the evaluator regarding alter- native policies which will yield the desired effect. At this point, however, the trap is sprung, and the researcher who wants to avoid advocacy is caught. Refusal to recommend alternatives implies criti- cism without willingness to make positive contributions. The image of hesitation in commitment and avoidance of reality is reinforced, the role of the evaluator is negated, and the policy is continued.

On the other hand, the researcher may suggest that certain policies have effects similar to those desired by the initiators. The real danger is that recommendations may be accepted as truth and adopted without due consideration of the policy perspective. The researcher may then receive the dubious honor of evaluating his own suggestions, thus experiencing role conflict and compromising his objectivity.

When the evaluator does become involved as advocate, a project’s failure, for whatever reason, to realize the goals as perceived by its instigators may be attributed to the evaluator. If recommendations were made, their validity may be questioned. If the evaluator re-

9

fused to make recommendations, failure may be attributed to his lack of cooperation.

Advocacy may be the most fundamental precepts of a social action program. As mentioned earlier, both regions and people become the focus for social action when they fail to conform to a particular definition of success or well-being. According to Marris and Rein [12, 1967, p. 1891, a program may be self-defeating when “the restora- tion of individual dignity is taken as a psychological, social and eco- nomic problem, inherent in those who are demoralized, rather than as a moral problem, inherent in the society which humiliates them.”

What are the alternatives for an evaluator who questions the basis upon which a social action program has been developed, who ques- tions the manner in which people are being treated, who questions the assumptions made about what people should be like or must be- come, who questions the spending of tax money with little or no results, but whose comments and evaluations are ignored? The simplest alternative is to avoid involvement, to remain aloof, and to contemplate problems as abstractions. Is this philosophical stance responsible? Despite the long debate over this issue, the search for the most appropriate role still continues to inhibit those involved in social action research.

The constraints described and illustrated are the negative side of evaluation. Generally speaking, human beings dislike admitting flaws in their actions, and the complexities of regional development provide ample opportunity for flawed action. This potential for error is pre- cisely why ongoing evaluation of public programs is essential. By getting people to admit that programs are usually less than perfect, the researcher can do much through personal contacts to remove or lessen the impact of the constraints imposed upon him. Certainly the success of evaluation will be enhanced through a better understanding of the nature of these restrictions by all parties involved in the social action process [7, 19671.

A few ideas on ways to ease the constraints on evaluation research may serve to summarize. Hostility from a region’s inhabitants could be parried by planting a survey research center in the region. This center would be established before the development plans were im- plemented and could later assume a monitoring role, perhaps as part of a regional government after the public development program termi- nated. Such a center would be more efficient in the collection, pro- cessing, and dissemination of information than would agencies for whom such activity is only secondary. The survey research center could serve to coordinate requests for information, not only for the intervention program but also for administrative and demographic

10

purposes (i.e., regarding education, health, crime, welfare). The pres- ence of this center could relieve some of the tension for the social action researcher in that the data would hopefully be of unques- tionable quality and independent of both the intervention program and the researcher.

Goals and objectives must be continually emphasized as prereq- uisite to any program action. Unfortunately, we have no suggestions on how to overcome the illusory rankings and goal statements often made by policy officials. Nevertheless, more quantifying is required before development plans are implemented.

The constraint imposed by our understanding of economic, social, and psychological behavior will bow only to a long-term solution. Research funding in the agriculture sector has been traditionally con- centrated in problems of the firm, such as production and, more re- cently, marketing. There is a paucity of research activity involving rural policy issues. A basic theoretical understanding of problems gained through research will always govern the extent to which pub- lic action is successful and the extent to which evaluations will justly consider appropriate variables.

The advocacy issue has existed since the birth of philosophy. The issue persists when the researcher must choose whether or not to disclose findings and when the evaluation procedure permits a high degree of subjectivity. In the former case, we suggest that our pro- fessions should not undertake the evaluation of public development interventions without the understanding that results will be fully disclosed as they are concluded. On the second issue, evaluations should be as systematic as possible, thereby minimizing the impor- tance of role conflict for the researcher in ongoing evaluations. Finally, directors of evaluation should realistically avoid spending more than three or four years on one development intervention.

References 1. Becker, Howard S. Outsiders: Studies in the Sociology o f Deviance. New

York: The Free Press, 1963. 2. Brager, George. “Influencine Institutional Change Through A Demonstration

Project: The Case of the Schools,” cited in Marris and Rein, Dilemmas of Social Reform. New York: Atherton Press, 1969.

3. Davis, F. James. Social Problems. New York: The Free Press, 1970. 4. Gerth, H. H. and C. Wright Mills. From Mar Weber: Essays in Sociology.

New York: Oxford University Press, 1958. 5. Goffman, Erving. Stigma. Englewood Cliffs, N.J.: Prentice-Hall, Inc., 1963. 6. Greenwood, Ernest. “The Practice of Science and the Science of Practice,”

in Warren G. Bennis, Kenneth D. Benne, and Robert Chin, eds, The Plan- ning of Change. New York: Holt, Rinehart, and Winston, 1961.

7. Hackler, James C. “Evaluation of Delinquency Prevention Programs: Ideals and Compromises,” Federal Probation, XXXI (March, 1969), 22-26.

8. Herman, Melvin. “Problems and Evaluation,” The American Child, XLVII (March, 1965).

9. Horton, Paul B. and Gerald Leslie. The Sociology of Social Problems, rev. ed. New York: Appleton-Century-Crofts, 1965.

11

10. Human Resources Development Authority, Government of the Province of Alberta. “News Release.” Edmonton, May 13, 1970.

11. Mannheim. Karl. “Roots of the Crisis in Evaluation,” in Warren G. Bennis, Kenneth D. Benne, and Robert Chin. eds., The Planning of Change. New York: Holt, Rinehart and Winston, 1961.

12. Marris. Peter and Martin Rein, Dilemmas of Social Reform. New York: Atherton Press, 1967.

13. Merton, Robert K. Social Theory and Social Structure. Glencoe: The Free Press, 1957.

14. Merton, Robert K. and Robert A. Nisbet. Contemporary Social Problems, rev. ed. New York: Harcourt, Brace & World, Inc., 1966.

15. Moynihan, Daniel P. Maximum Feasible Misunderstanding. New York: The Free Press, 1969.

16. Nettler, Gwynn. Explanations. New York: McGraw-Hill, 1970. 17. Philadelphia Council for Community Advancement. Annual Report to the

Ford Foundation, cited in Marris and Rein, Dilemmas of Social Reform. New York: Atherton Press, 1963.

18. Webb, Eugene J., Donald T. Campbell, Richard D. Schwartz, and Lee Sechrest. Unobtrusive Measures: Nonreactive Research in Social Sciences. Chicago: Rand McNally & Company, 1966.

19. Weber, Max, “Science as a Vocation,” in H. H. Gerth and C. Wright Mills eds., From Mas Weber: Essays in Sociology. New York: Oxford Univer- sity Press, 1946.