Managing a Health Project: HIV/AIDS in Thailand. Complex Program Evaluation

PADM 522

Professor Rivera

The case study 1998-2003 Project jointly undertaken by

The Japan International Cooperation Agency (JICA) In conjunction with the Thai Ministry of Public Health

(MOPH) Phayao Province is located in

Northern Thailand Population estimated at about

515,700 Mostly farming region Province about 2,500 sq. mi. Nearest city – Bangkok

~ 450 miles away

Phayao Province – Nine districts with the highest rate of HIV/AIDS in Thailand

1. Mueang Phayao 2. Chun 3. Chiang Kham 4. Chiang Muan 5. Dok Khamtai 6. Pong 7. Mae Chai 8. Phu Sang 9. Phu Kamyao

Principal Routes of Transmission Gay population

Intravenous Drug Users

Female prostitutes

Male customers

Spouses

Children/Infants

Young women leaving rural areas for commercial sex industry in the cities, to help with meager incomes at home; return to Phayao without apparent symptoms Erroneous beliefs about disease transmission Stigmatizing attitudes toward homosexuals more generally cast on most victims of HIV/AIDS—underscored need for education

Consider that there are asymptomatic patients—illness may not show for years HIV-infected persons often show no physical

symptoms Symptoms do not show for 9 to 10 years in

some Initial exposure to disease are often mistaken for

flu or other illnesses During this asymptomatic stage, virus is

multiplying, infecting, and severely weakening the immune system

Turning Point in the Case Study Japanese International Cooperation Agency (JICA)

Officer in Charge, Dr. Tani, received a proposal for a modification to an already agreed-upon treatment-centered project

The proposal was from Thai project leader Dr. Phannee, advocating another or a different type of intervention

Dr. Phannee argued that fighting HIV/AIDS required a multidimensional response, in particular the front-loading of training courses for administrative and clinical staff working through Phayao. Why did she insist on this?

Community Assessment Course Course module was designed to:

Change the attitudes of health staff Enhance their capacity for critical thinking Go into the community to talk to potential beneficiaries

of the project Work collaboratively with non-health sectors Acquire skills for data collection and analysis

Strategy was to change the perspective of health staff, from passive administrators to active practitioners, to change culturally-ingrained attitudes, and to make the engagement of HIV/AIDS victims in the program.

Dr. Phannee’s PerspectiveDr Phannee believed the following: “Health staff do not assess community needs or take

action by themselves.” She believed in decentralized decision making and

action Manpower development was the highest priority of the

project, in her view. Do you agree with her assessment? Do you agree that manpower development and training had to come first, before the program implementation JICA wanted?

Concerns of the Japanese Team Urgent need for HIV/AIDS patients to have access to care Promotion of preventative measures locally Wanted action on HIV/AIDS intervention as originally

planned, rather than broadening of the project to include manpower development.

Wanted technical improvement in prevention and care Prevention included strengthening public awareness by

increasing availability of condoms to prevent transmission from infected mothers to children

Providing care and advice to asymptomatic patients in early stages of infection

Increasing available care at health centers by increasing the technical (rather than cultural) skills of clinical staff

The Controversy The Japanese saw an urgent need for direct intervention

because patients had little access to care and support in Phayao

If manpower training was top priority, there might be a suspension or postponement of other activities, with the possibility of increased rates of infection. The disease might spread more quickly and widely in the community. Conditions might worsen with less funding allocated for each patient

Nonetheless, Dr. Phannee prevailed. Dr. Tani and other consultation team members were

surprised at the drastic changes Dr. Phannee urged on the program. However, under instructions from headquarters, the consultation team added the interventions for health manpower development

Consequently, Dr. Phannee started to implement the course entitled “Community Assessment Training” for her staff.

Following the First Consultation, 1999 – 2001 The consultation teams of both the Japanese experts and

JICA decided to add interventions for health manpower development to the project—the Community Assessment Training

The project design changed, and manpower development became the first priority in the interventions, postponing JICA’s preferred interventions.

Besides adding health manpower development, some other changes and adjustments took place several times during the project.

The Thai and Japanese team held numerous discussions about the project’s approach to issues of HIV/AIDS.

Final Evaluation, Summer 2002 JICA sent an evaluation team in the summer of

2002, led by Dr. Ishi. The aim of the evaluation was to assess the

extent of achievement of the aims of project by asking how far, and how efficiently and effectively, the project purpose(s) had been achieved.

How to assess the health manpower development intervention in this context?

Final Evaluation, Summer 2002The Project’s Accomplishments Standardization of care and creation multilevel networks for

care and prevention resumed in 2001 after delay due to the manpower development/training effort.

Developed a counselors’ network and strengthened pre/post test counseling to improve prevention and better serve people diagnosed as PHA.

As a result, HIV related examinations became faster and more accurate.

For prevention, the project installed condom vending machines and analyzed used of the machines with social marketing methodology.

The Project also studied the sexual behavior of young people—the local political and cultural economy of the national sex industry (heavily used by some Japanese tourists).

Final Evaluation, Summer 2002On the other hand, the evaluation team argued that there should have been a focus on improving

care and prevention of HIV/AIDS, rather than broadening the project scope to cover health manpower development; questioned whether the course was essential for alleviating the problems surrounding HIV/AIDS in Phayao. It might have been needed for Thailand, but should not have been included in the project.

found that the decision in 1999 to make health manpower development the top priority intervention had delayed actual improvements in health service for HIV/AIDS.

Found, overall, that the project’s efficacy, or effectiveness, in accomplishing its purpose were relatively low.

Final Evaluation, Summer 2002Overall, nonetheless The amount of care and the information provided had

increased in 5 years. The project seemed to have contributed to increasing the

care available for PHA, such as strengthening the functions—i.e., the capacity—of local health centers.

The more treatment became available for PHA, the more positive about living they became, the more active in organizing themselves, and even more avidly involved in activities promoting awareness of HIV/AIDS in the community. PHA responses to surveys indicated high level of satisfaction with participation in the project. None of this would have been possible without the attitude change of health staff because of their training, and the resulting willingness to cooperate with PHA.

Contentious issues A major concern was lack of fidelity between original and

later program design and outcomes. Chen’s definition: “the fidelity evaluation approach assesses whether a program has been or is being implemented according to expectations.” (p.54) Although the program strayed from its original objective through the incorporation of health manpower development, it was seen as successful in context.

The JICA team grudgingly acknowledged the Thai team’s local expertise and chose to heed their advice for altering the program. While this could be seen as loss of fidelity, another perspective is that the local stakeholders knew what was best for the program. Staying true to program mission is not always the best way to form an evaluation and interpret program success. Chen indicates that assessing a program’s impact on stakeholder communities may be as important as maintaining fidelity.

Contentious issuesScientific validity has to be juxtaposed against stakeholder

validity in deciding what the best route for a program to take. In the case of the Thai program, more emphasis was placed on healthcare sector restructuring and manpower development as prior steps.

It is also necessary, as Holden points out, “to understand the political context of an evaluation.” (p.12) In light of the ongoing decentralization and reform of the Thai health system, it became all the more important to address cultural and participation issues. The problem remains of the timing of program changes. Could a hedging approach have helped?

Participatory Impact Pathways Analysis An approach for developing common understanding and consensus among participants and stakeholders as to how the project or program will achieve its goals

Components of the causal logical model(s)/rationale(s)/theories of change to be tested are identified by key stakeholders.

These include changes in knowledge, attitudes, skills, and practice that should result from use of project outputs: Indicators and progress measures that all can agree upon.

Participatory settings provide opportunities to bring policy-makers and “users” together

Including stakeholders increases understanding: Results are more accurate and reliable when group and/or individual responses are immediately cross-verified, and triangulated with information from process monitoring.

Difficulty in achieving agreement on what merits assessment and needs to be monitored, or even on what project should be.

Lessons Learned from applications of PIPA Key stakeholder involvement is critical from the onset to

build consensus for concerted action, but multi-stakeholder partnerships are messy, often conflictive (as with this case).

Participatory, reflexive processes increase commitment and create social capital—largely limited in this case to the Thai side.

Build on existing governance structures and organizational mechanisms, but move beyond community-level or national frameworks to socially disaggregated processes

Drive the process with plausible prospects for short and medium term successes, and engage champions over the long term

Thai and Japanese sides could have made the mutuality of benefits to both sides a much clearer focus of discussion

Case Stakeholders’ Perspectives: Thai Perspective

Insisted on the addition of health manpower development.

Thailand’s health sector reform at the time influenced this perspective

Changing national mores and values

Japanese Perspective Questioned

project redesign, redefined project’s focus

Believed that project efficiency and effectiveness were low

Deflected from donor agency’s (JICA’s) preference for immediate treatment priority

Final Project Evaluation First part of the evaluation considered all efforts made

by the local Phayao Provincial Health Office under Dr. Phanee.

Emphasis on attitudinal changes of people about HIV/AIDS, conditions in the community, and the health services delivery system.

Findings: Local authority and responsibility had increased due to more autonomy and staff training.

Less reliance on external support of foreign agencies meant more sustainability/ownership, greater local capacity.

Thai Perspective on the Project Decentralization of Thailand’s health system was

accelerating by 2002, when the evaluation was conducted. Movement toward more autonomous, and network-based, healthcare systems.

Training local staff and manpower or human resource development of health staff became a high priority for the entire country.

Side benefits included improving local capacity to deal with all health issues.

Japanese Perspective on Evaluation• Measured the program success using a more

standard logic model of inputs, program activities, outputs and its achievement.

• Japanese team insisted that more emphasis should have been on improving the efficacy and quality of care for HIV/AIDS patients, along with disease prevention.

• Claimed that a standardized clinical model for HIV/AIDS care and prevention could have been developed and then its process analyzed before adding health manpower training/development.

Differences in Evaluation Japanese team rated the project low on effectiveness and

efficiency, precisely because the project design was altered in 1999 with the Community Assessment Training program. Weak causal link between manpower training and achievement of project purpose, from the Japanese standpoint.

Other evaluation criteria were made predominant, including project relevance (both a medical and social concern in Thailand), sustainability, and impact.

Joint meetings between the Thai and Japanese teams helped to revise evaluations of the effectiveness and efficiency upward, reflecting JICA’s changing perspective back at Headquarters in Japan.

Final Evaluation, Summer 2002—Can project lay claim for this apparent success?

Reported Traffic Fatalities in Connecticut: 1951-59

Sources: (Campbell & Ross, 1968, p. 42)

Reported Traffic Fatalities in Connecticut and 4 Comparison States: 1951-1959 (per 100K population) suggests that there was regression toward the mean in CT. Could there have been regression toward the mean in the Thai case longitudinal statistics?

JICA Evaluation’s Conclusions The JICA Program was acknowledged to be a success

overall, and in specific respects, but with some reservations.

There was real adaptation of a foreign-sponsored and designed project to local realities.

The Japanese contended a direct method of involvement focused on direct care was necessary, whereas Dr. Phannee advocated a broad community- and culture-based approach. In effect, she wanted to increase the cultural sensitivity/cultural competency of health workers

The final evaluation rated the effectiveness and efficiency of the project low, while the project’s sustainability and the level of “ownership” by Thai officials were acknowledged to be high.

Accountability, Evaluation & Partnerships Simple notions of accountability are threatened by the

growing use of partnerships. Such arrangements lead to causal attribution challenges and to shared accountability for the attainment of desired outcomes, which differs markedly from more traditional arrangements in which accountability is hierarchical. Complex causation and complex accountability make it all the more important that there be a separation of policy determination, including elaboration of change/action models, from program prescription, and that the focus shift to actual outcomes. How would greater emphasis on policy guidance and actual outcomes, rather than a prescriptive JICA program model, helped the Thai-Japanese collaboration in directing and evaluating the program in this case? Were the Thai and Japanese working collaboratively in any sense? Should they have done so? Is causal attribution difficult in this case?

Advocacy Additionally, conflict is more likely in partnered p

programs where there is power asymmetry. As Ringsing and Leeuwis argue, “Power plays and [conflict over] control happen at all levels in organizations or projects, because ‘all actors exercise some kind of “power”, leverage or room for manoeuvre, even those in highly subordinate positions’ (Long, 2001: 17). Deliberate efforts to monitor and evaluate projects are a conducive context for the intermingling of – and tension between – control and learning, since it is in such arenas that past activities are interpreted and assessed and where future courses of action can be legitimately proposed.

Evaluating for outcomes—Chen Outcome evaluation (Chen) has four phases: “identification of

goals . . . of outcome measures and data sources . . . of needed background information, and pre- and post-intervention collection of data.” (p. 185). In the first, goals are chosen by representatives from various stakeholder groups—for the Thai project, people from JICA, from the government in Thailand at both national and provincial levels, and local principals. In the second, evaluators must decide what indicators and types of data that they will use to measure impact: e.g., one of the measures chosen was the quality of life of people with HIV/AIDS. The third requires the collection of socio-demographic data. The JICA and Thai teams used measures of HIV/AIDS prevalence based on gender, age, and location to better measure the impact of their interventions. The last phase requires longitudinal information. JICA performed an ex-post evaluation that ran from 2003-2005, even though the original project ended in 2002 (the first evaluation was performed just a few months before the project was scheduled to finish). Chen discusses the need for standardized measurement in evaluating pre/post outcomes. In the Thai case, however, evaluation criteria changed along with program mission.

Evaluation and threats to validityInternal and External Validity Are you sure that it is the program that caused the effect?

Examples of concerns: secular trends, program history (attrition, turnover), testing effects, regression to the mean, etc.

Are assessments based on truly comparable elements? Mixed evaluation can help ascertain causal relationships

Evaluating for multiple purposes at the same time may be necessary, and uniquely revealing

Program evaluation for partnered programs often means assessment along several lines of inquiry—and it is therefore also more inclusive than is ordinarily the case

Process and formative kinds of evaluation occur at the organizational and program levels (internal validity), while impacts are assessed more comprehensively, encompassing more actors (external validity)

Context is important—program environment, broader strains of causation that include other influences

Assessing both intervention & evaluative comparison in the case.

Mediating (intervening, interventional) variables and moderating (external enabling or constraining) variables further complicate the causal chains anticipated by program designers. In HIV/AIDS education programs, partners or spouses of beneficiaries may either help or hinder program efforts. How? Do these realities militate against over-prescriptive, ex-ante programming?

ProgramOutcome

Mediating variables

Moderatingvariables

External Determinants

(such as spouse orpartner buy-in)

Internal or intervention

determinants, e.g. insistence

on culturaltraining forclinical staff

Interventions and their

determinants (e.g., JICA’s

over-prescriptiveapproach)

Example of the HIV/AIDS in Thailand case study

Mediating Variables Moderating Variables

Implementation (interventiondeterminantsprogram outcomes)

Community Assessment Training intended to educate and change the attitudes of health workers toward victims of HIV/AIDS, engaging the latter in program delivery

Usually less than +: e.g., lack of partner support, social and economic variables such as persistence of poverty, prejudice

Net impact on individual subject(s) of the intervention

aggregate net impacts=outcomes

Action/Change Model

Intervening Variables, Confounding FactorsIntervening variables can take various forms Direct and indirect effects of one’s intervention, other

interventions and/or messages or influences; interactions among variables

Reciprocal relations—mutual causation (e.g., mixed messages from one’s group versus family, faith)

Variables can be observed or latent. How context dependent is the intervention? The more

complex it is, the more dependent on context are both the program implementation and program evaluation. Many interventions sensitive to context and can be implemented only if changed to suit context.

Evaluating complex, partnered programsEvaluation in partnerede contexts—questions pertinent to case:

Does it matter to the functioning and success of a program that it involves different sectors, organizations, stakeholders, and standards? In the Thai case, there was use of multi-level networks along with standardization of care and culturally sensitive services.

What level of consultation are needed to achieve program aims? Can we evaluate the development of partnered efforts and

partnership capacity along with program outcomes and program capacity?

To what extent have program managers and evaluators consulted with each other and with key constituencies in establishing goals and designing programs? Are externally-funded/sponsored programs inherently problematic in these regards?

Resources for Partnership Assessment

http://www.cdc.gov/prc/about-prc-program/partnership-trust-tools.htm

http://www.cdc.gov/dhdsp/state_program/evaluation_guides/evaluating_partnerships.htm

Resources

http://www.cdc.gov/dhdsp/state_program/evaluation_guides/evaluating_partnerships.htm CDC Division for Heart Disease and Stroke Prevention, “Fundamentals of Evaluating Partnerships: Evaluation Guide” (2008)

http://www.cdc.gov/prc/about-prc-program/partnership-trust-tools.htm CDC Prevention Research Center’s Partnership Trust Tool

http://www.cacsh.org/ Center for Advancement of Collaborative Strategies in Health, Partnership Self-Assessment Tool

http://www.joe.org/joe/1999april/tt1.php Journal of Extension article: “Assessing Your Collaboration: A Self-Evaluation Tool” by L.M. Borden and D.F. Perkins (1999)