NOTES
1 .It is important to note that, although some components identified in systems theory bear the same labels as components found in the logic model, the definitions of each arc quite distinct.
2. Chapter 3 includes a more intensive discussion of the concepts of these program stages.
* TWO
■A CONCEPTUAL FRAMEWORK OF PROGRAM THEORY FOR PRACTITIONERS
Hi"1 he benefits that program theory provides to evaluation are well JL documented in the evaluation literature. For example, Bickman (1987a) discussed the usefulness of program theory for improving the generalizability of evaluation results, contributing to social science theory, uncovering unintended effects, and achieving consensus in evaluation planning. Weiss (1998) noted as an advantage of program theory the provision of early indications of program effectiveness. She also found program theory helpful for explaining the occurrence of program effects, which enhances the relevance of evaluation. Nevertheless, Weiss (1997) indicated that program theory's lack of clarity of the concept creates an obstacle for advanced theory-based evaluations In Chapter 2, the book seeks to explain, for evaluation practitioners, the concepts and conceptual framework of program theory in user-friendly terms. Understanding these concepts should pave the way to using them effectively in evaluation.
For users of the book, a vital function is served by the introduction, in this chapter, of the conceptual framework of program theory. The framework helps readers grasp how the evaluation taxonomy can fruitfully guide the choice of evaluation approach or method (Chapter 3). Knowledge of the framework also elucidates the how-to of applying the various approaches and methods (Chapters 4-10).
15
JNTRODUCTION
THE PROGRAM THEORY CONCEPTUAL FRAMEWORK
One popular definition of program theory arises from causal relations. Bickmart (1987a), for example, defined program theory as "a plausible and sensible model of how a program is supposed to work" (p. 5). Similarly, Weiss (1995, p. 66) viewed program theory as the picture of how and why programs work (or don't). Another popular way to understand program theory is from the context of the logic model. For example, Wholey (1987) said program theory incorporates "program resources, program activities, and intended program outcomes, and specifies a chain of causal assumptions linking program resources, activities, intermediate outcomes and ultimate goals" (p. 78). A broader definition of program theory subsuming the existing definitions was given by Chen (1990), who described program theory as "a specification of what must be done to achieve the desirable goals, what other important impacts may also be anticipated, and how these goals and impacts would/be generated" (p. 43). This chapter of the book elaborates on this latter definition of program theory.
The design and implementation of an intervention program are usually based on a set of explicit or implicit assumptions by stakeholders about what action is required to solve a social problem and why the problem will respond to this action. The analysis of the explicit and implicit assumptions underlying a program is called program .theory. Chen's definition of program theory
"suggests its simultaneously prescriptive and descriptive nature, a status requiring program theory to be action-oriented. Thus, program theory goes beyond typical scientific theories—those from the social and behavioral sciences, for
instance—that focus solely on providing causal explanations of phenomena.
Program theory can be viewed, then, as a configuration of the prescriptive and
descriptive assumptions held by stakeholders and thus underlying the programs
stakeholders create.
Descriptive Assumptions
Within program theory, descriptive assumptions concern the causal processes begetting whatever social problem a program tries to address. As an illustration, consider a treatment program for spouse abusers. According to program designers' descriptive assumptions, spouse abuse typically results, at least in part, from the abuser's lack of skill in dealing with anger or frustration
A Com
filial Framework of Program Theory for Practitioners
17
and lack of knowledge of the law's stance on domestic violence. In light of these descriptive assumptions, the treatment program might be designed to employ counseling to develop anger management skills. It might also stress the legal consequences of committing domestic violence. The causal process underlying this treatment program's effectiveness, then, would be the instillation of fear of consequences, to encourage practice of the skills taught, to then reduce the abuse.
Assumptions about causal processes through which treatment or intervention is supposed to work are crucial for any program, because Us effectiveness depends on their validity. If invalid assumptions dictate the strategies of a program, it is unlikely to succeed. For example, among those enrolled in the hypothetical spouse abuse treatment program, if the major motive of the abuse is belief in the patriarchal structure of families, rather than uncontrolled anger or ignorance of consequences, then the program's emphasis on anger management is unwarranted. The set of descriptive assumptions made about causal processes underlying intervention and its outcomes constitutes the causative theory (Chen, 1990) of programs. Outside the field of program evaluation, however, this phrase may not communicate well—and remember that stakeholders come from other fields. The set of descriptive assumptions can also be termed the "change model," for purposes of effective communication, and throughout this book change model is substituted for causative theory or descriptive theory. The change model is emphasized in much of the theory-driven or theory-based evaluation literature (e.g., Donaldson, 2003; Weiss, 1998). As will be discussed in Chapters 3 and 4, the change model concept is very useful for developing a program rationale.
Prescriptive Assumptions
Turning now from descriptive assumptions to prescriptive assumptions, the latter are equally significant, according to program theory, in an intervention program. The prescriptive assumptions of program theory prescribe those components and activities that the program designers and other key stakeholders see as necessary to a program's success. Program designers' prescriptive assumptions thus direct the design of any intervention program. They determine the means of implementing and supporting the intervention so that the processes described in the change model can occur. Because prescriptive assumptions dictate what implemented components and activities will be required to activate
INTRODUCTION
,i Concept
-fpliial Framework of Program Theory for
Practitioners
19
the change model, they are collectively referred to as the normative theory, or prescriptive theory, of programs (Chen, 1990). But again, stakeholders (and evaluation practitioners, too) may appreciate the directness of an alternative term, action model, which is used in the remainder of this book. As will be discussed in Chapters 3 and 4, the action model concept is very useful for developing a program plan.
Program evaluators look to the action model for the requisites of a program, as well as for the feasibility of these requisites in the field. Ifi the action model are found the bases for answering questions such as the following: What are the crucial elements of the intervention? What kind of organization is needed to deliver the services? Who is best qualified to deliver them? How will implemented be trained? What is the target group? How will the target group be reached?
Again, as an example, take the spouse abuse treatment program, Suppose its designers decide that the target group should be abusers convicted by a court; this decision is based on an assumption that most spouse abusers endsup in court, and on the court's agreement to use the treatment program as part of an abuser's sentence. The arrangement would certainly guarantee the program a steady source of clients. It would also necessitate establishment of an administrative linkage between the court and the program's implementing organization, based on an assumption that clear channels of communication will keep the court apprised of any client's failure to attend treatment. Suppose the program designers choose group counseling as the treatment for the abusers, headed by a trained and experienced professional facilitator. This decision could stem from the program directors' favorable experiences with group therapy in other situations. Perhaps the designers decide that group counseling should be provided weekly for 10 weeks because they believe that 10 counseling sessions is a sufficient "dose" for most people. From these assumptions comes the need for the program to hire two professional counselors available for 10 consecutive weeks.
The action model deals with nuts-and-bolts issues, which are not a major topic in most modern social science theory, perhaps due to the social sciences' emphasis on developing generalizable propositions, statements, and laws. "How-to" program issues tend to be trivialized by contemporary social science theory. Plus, the action model has no proposition-like format resembling that defined by and familiar to modern social scientists. However, it is interesting to note that many classic social science texts discuss both descriptive and
prescriptive theories. Both Max Weber (1947) and Emile Durkheim (1965) intensively discussed not just explanations of organizational and societal phenomena but also steps for improving organizations and societies.
The action model translates the abstract ideas that theoretically justify a program into the systematic plan necessary to organize its day-to-day activities. Implementation of the action model puts a program in motion. And just as with the change model, if the action model is based on invalid assumptions and is thus poorly constructed or unrealistic, the program is not likely to meet with success. Another example shows how important an accurate action model is to a program. The government of a developing country found that many farmers could not afford to buy fertilizer or modern equipment to increase productivity. It moved to set up low-interest loans for the farmers. Designers of this financial program postulated a particular change model: Lack of access to capital limits farmers' ability to improve productivity, and fanners would apply for low-interest loans, if available, to buy machinery and fertilizer to boost their land's productivity and their earnings. The designers' programmatic model stipulated use of the government's own banks to process applications and conduct subsequent transactions. The underlying assumption was that, as part of the government system, these banks would require simply an administrative order to diligently and responsibly implement the program; in addition, operational costs would be much less than if commercial banks became involved.
A couple of years after the program had been launched, few farmers had received loans and benefited from the program. Why? Because certain assumptions of the action model were wrong. Local staff of the government bank did not see the new program as all in a day's work. To them, the program meant another burden in addition to their already heavy workload, with no increase in rewards. Consequently, the staff members' behavior concerning implementation of the program was not quite what decision makers had assumed it would be. Not only were they unenthusiastic about the program, but they also pulled up older rules and regulations to actively discourage farmers from applying for, or to disqualify them from receiving, the loans. This maintained their accustomed workload—and made the new program fail.
In this chapter, Chen's (1990) initial conceptual framework of program theory is broadened and altered to increase its relevance within evaluation practice. The components of a change and action model are discussed as follows.
INTRODUCTION
COMPONENTS OF THE CHANGE MODEL |
The components of a change model are its goals and outcomes, its determi- | nants, and the interventions or treatments it is to implement. These change * model components and their interrelationships are introduced here.
Goals and Outcomes
Goals reflect the desire to fulfill unmet needs, such as with poor health, inadequate education, or poverty. Program goals are established in light of certain major assumptions about them, such as their likelihood of being well understood and supported by staff and other stakeholders; their power to motivate commitment of resources and effort; and/or their accurate reflection of stakeholders' aims in valid, measurable outcomes. A program's existence is justified through the meeting of its goals, which are usually articulated in very general, highly laudatory language in an effort to win broad support for the program. In contrast, outcomes are the concrete, measurable aspects of these goals. For example, one goal of welfare reform is to reduce dependency on welfare. An outcome linked to this goal might be increased numbers of welfare recipients obtaining jobs, alleviating need for government support. "Reducing dependency on welfare" is a notion with many ramifications; it is imprecise. But the outcome "obtaining jobs" gives "specific meaning to the program's orientation.
Outcomes themselves may have components, and some outcomes may have both short-term and long-term manifestations. For example, in an HIV prevention program, the outcome over the short term may be increased use of condoms by a high-risk population. The outcome of the same program in the long term may be a lower number of HIV transmissions. Furthermore, a program's outcomes may include intended and unintended developments. If program stakeholders and evaluators suspect that unintended outcomes exist (whether desirable or undesirable), then the evaluation should include the identification of all unintended outcomes.
Determinants
To reach goals, programs require a focus, which will clarify the lines their design should follow. More specifically, each program must identify a leverage mechanism or cause of a problem upon which it can develop a treatment or
\ Conceptual Framework of Program Theory for Practitioners 21
intervention to meet a need. The assumption is that, once the program activates the identified leverage mechanism, or alleviation of the cause of a problem, its goals will soon be achieved. That leverage mechanism is variously called the mediating variable, the intervening variable, or the determinant, and in this book, the latter term is used. In the field of health promotion, theories suggest a variety of determinants that program designers and key stakeholders can deploy in a program (Bartholomew, Parcel, Kok, & Gottlieb, 2001). For example, the health belief model (e.g., Strecher & Rosenstock, 1997) outlines these determinants influencing an individual's course of action (or inaction) for a health problem: perceived susceptibility to the problem, perceived seriousness of the problem's consequences, perceived benefits of a specific action, and perceived barriers to taking action. Similarly, social learning theory (Bandura, 1977) cites self-efficacy—or the conviction that one can, in fact, carry out the behavior that elicits the outcome—as the most critical determinant of behavioral change. The PRECEDE-PROCEED model (Green & Kreuter, 1991) identifies predisposing factors, reinforcing factors, and enabling factors as important determinants for health behavioral change. The determinants identified by scientific theories are intensively studied and applied in scientific research.
Of course, few programs designed and conducted by stakeholders are designed for strict conformity to social science theory. Naturally, what is identified as the determinant often relates to the program designers' understanding of what actually causes the problem they want to alleviate and on which exact cause or causes they want a program to focus. There have been program designers, for example, who believed that urban school students' poor test performance stemmed from a lack of parental involvement, making parents the appropriate focus for programs meant to improve scores. These program designers saw in parental involvement the determinant to help students perform better; for them, it followed that, if the program activated parental involvement, student scores would improve. With a determinant identified, they could move on to figuring out how parents could be encouraged to participate and trained to help children study. Again, a program's identified determinant will provide its focus.
Social problems often have roots in multiple causes, but an intervention program usually focuses on one, or perhaps a few, determinants that program designers see as the major cause of the problem—or the most feasible to address or the one best suited to their expertise. It would be difficult for a
22
INTRODUCTION
[ Conceptual Framework of Program Theory for Practitioners
23
program to deal simultaneously with all potential determinants, given typical constraints on resources and time. The unmanageability of multiple determinants aside, it remains important to specify clearly on what determinant a program has selected to focus and to justify that selection; Consider the case of juvenile delinquency in a community. High rates of such delinquency may be the result of peer pressure, failure in school, a lack of positive role models, a lack of discipline, a subculture of violence, or a dearth of economic opportunity. A program to lower rates of juvenile delinquency must state plainly, to stakeholders and the community, the cause or causes it assumes to be most relevant and the determinant or determinants upon which it will focus. Selection of the determinant or determinants could be shaped by social science theory and research, the success of other programs, and/or the program designers' own experiences and ideas.
Intervention or Treatment
Intervention or treatment comprises any activity or activities of a program that aims directly at changing a determinant. Intervention/treatment is, in other words, the agent of change within the program. The vital assumption made in the intervention/treatment domain is that by implementing these activities, the program changes the determinant and ultimately reaches its goals! For example, a treatment program for juvenile delinquency chooses to focus on a community's lack of accessible positive role models for youth. The intervention or treatment provided by the program is to team each youth with a volunteer, an accomplished professional or businessperson from the area, to serve as a role model. Volunteers are expected to spend 2 hours each week with the participant, providing guidance and encouragement related to school, home, and neighborhood. Once a month, the pair is asked to attend a community event or visit with a private or public organization. As the pair's relationship deepens, the program designers assume, the status of the volunteer... and his or her personal interest in the youth will motivate the youth to identify with the volunteer and emulate his or her agenda of productive and beneficent activities. This will lower the odds of future involvement in delinquency. In many cases, an intervention or treatment has a number of elements. For example, alcohol abuse treatment is likely to include detoxification, individual and group counseling, and family therapy.
Some intervention programs can attain program goals without mediating by a determinant. Food relief programs in a disaster or warring region are a
good example. A food relief program is regarded as successful as long as foods are distributed to and consumed by refugees. However, the great majority of intervention programs aim at changing knowledge, beliefs, behaviors, and/or skills. These kinds of programs usually require the intervention to change some determinants in order to affect goals or outcomes.
The terms treatment and intervention have been used interchangeably in the program evaluation literature. However, for health-related programs, at least, there is a subtle difference between the two concepts. In health-related programs, treatment is equal to caring for and, ideally, curing people who currently have some illness. Intervention more often refers to an effort to alleviate an existing problem, to ward off a potential problem, or to improve some aspect of quality of life. An intervention might sometimes comprise treatment. The evaluation principles and strategies discussed in this book can be applied to either treatment or intervention programs. For simplicity's sake, in the remainder of the book, the term intervention will be used, covering both meanings.
COMPONENTS OF THE ACTION MODEL
An action model is a systematic plan for arranging staff, resources, settings, and support organizations in order to reach a target population and deliver intervention services. This programmatic model specifies the major activities a program needs to carry out: ensuring an environment for the program that is supportive (or at least not hostile), recruiting and enrolling appropriate target group members to receive the intervention, hiring and training program staff, structuring modes of service delivery, designing an organization to coordinate efforts, and so on. It is vital to recognize that the impact made by a program's change model results jointly from the intervention's effect and the particulars of the program's implementation. The success of a job training program, for example, is not totally determined by its curriculum but also by the quality of its teachers, the motivation and attitude of its participants, the job search strategies employed, and the vigor of the local economy. The following discussion touches on all major elements—that is, the complete form—of the action model; it provides an exhaustive list, which may be much more than the evaluator requires in actual practice. (A rule of thumb is that large-scale programs may need all six elements, whereas small-scale programs may be just as effective with only a few of them.) Nevertheless, familiarity with the
24_ INTRODUCTION
complete action model enables the evaluation practitioner to discuss more than | one version of program theory. Access to the complete action model also helps in determining which components are important in a unique set of circumstances and in understanding how to simplify or otherwise modify the model to fit particular evaluation needs. The elements of the action mode! are the implementing organization, program impiementers, associate organizations/
community partners, context/environment, target population,.and intervention.......
and service delivery protocols. From this list of elements, program evaluators can draw ideas about areas of potential focus within evaluations they are designing.
Intervention and Service Delivery Protocols
The change model for a program reflects general and abstract ideas about 1 intervention that must be translated into the set of concrete, organized, imrjle-mentable activities constituting its programmatic model. Basically, there are two requirements for this translation: an intervention protocol and a service & delivery protocol. The intervention protocol is a curriculum or prospectus stating the exact nature, content, and activities of an intervention—in other words, the details of its orienting perspective and its operating procedures. To j. begin to ascertain the intervention protocol of a family counseling program, for example, answers to the following general questions are needed: What is the nature of the counseling? What is the content of the counseling? What is ^ the schedule for the counseling? Specific answers to these might be generated by asking questions such as the following: Is the counseling based on behavior therapy? On reality therapy? On another kind of therapy? Will counselors [ proceed by following standardized documents? How many counseling sessions ; are planned, and how long will each last? ']
In contrast, the service delivery protocol refers to the particular steps to be ! taken in order to deliver the intervention in the field. The service delivery protocol has four concerns: client processing procedures, or how clients move \ from intake to screening to assessment to service delivery; division of labor in ; service delivery, or who is responsible for doing what; settings, which may ; be formal (e.g., at a program's office) and/or informal (e.g., in a client's home); and communication channels (face-to-face, telephone, mail, etc.). As an example, the service delivery protocol of a program addressing child abuse t would provide answers to the following questions: Where will counseling
4 Conceptual Framework of Program Theory for Practitioners_____25
take place—in a counselor's office or in clients' homes? Will each parent be counseled separately, or will they meet with the counselor together? At what point, if any, will child and parents be counseled together? In general, one place to look for the level of quality of a program is in its establishment (or lack of establishment) of an appropriate intervention protocol and service delivery protocol.
Implementing Organizations:
Assess, Enhance, and Ensure Its Capabilities
A program relies on an organization or organizations to allocate resources; coordinate activities; and recruit, train, and supervise impiementers and other staff. How well a program is implemented may be related to how well the organization is structured. Initially, it is important to ensure that the implementing organization has the capacity to implement the program, and strategies exist that can be helpful in determining this. For example, if a funding agency gets to choose the implementing organization from among several qualified candidates, that agency may be well equipped to determine which organization is most capable of implementing the program. In reality, however, such a pool of capable organizations ready for action may be missing. This is especially true for community-based organizations. Usually, an implementing organization's capacity to conduct the program must be built up. Capacity building involves activities such as training, technology transfer, and providing— financially and otherwise—for the hiring of experts or consultants to help plan and conduct the implementation.
Program Impiementers: Recruit, Train, and Maintain Both Competency and Commitment
Program impiementers are the people responsible for delivering services to clients: counselors, case managers, outreach workers, schoolteachers, health experts, and social workers. The impiementers' qualifications and competency, commitment, enthusiasm, and other attributes can directly affect the quality of service delivery. The impiementers' competency and commitment also have a direct effect on the quality of the intervention delivered to clients, and thus the effectiveness of the program in large part depends on them. Under the action model, it is important for a program to have a plan for ensuring
26
INTRODUCTION
I Cu""'/"""' ^rcimeu-ork of Program Theory for Practitioners
27
competency and commitment among program implemented, using strategies such as training, communication, and performance monitoring/feedback.
Associate Organizations/Community Partners: Establish Collaborations
Programs often may benefit from, or even require, cooperation or collabora-tion between their implementing organizations and other organizations. If linkage or partnership with these useful groups is not properly established, implementation of such programs may be hindered. In the example of the spouse abuse treatment program, program implemented need to work closely with the court to develop procedures requiring convicted abusers to participate in treatment as part of their sentences. This program would meet with serious difficulty if it lacked a working relationship with the court or failed to win the support of judges. Under the action model, it is important to create feasible strategies for establishing and fostering relationships with associate organizations and community partners. As will be detailed in Chapter 5, this element is most important when an evaluator is asked to take a holistic approach to help program designers and other stakeholders plan and develop a program.
Ecological Context: Seek Its Support
Ecological context is the portion of the environment that directly interacts with the program. Some programs have a special need for contextual support, meaning the involvement of a supportive environment in the program's work. (Indeed, most programs can be facilitated to a degree by an environment that supports the intervention processes.) A program to rehabilitate at-risk juveniles, for instance, is more likely to work when it obtains the support and participation of juveniles' families and friends. Both micro-level contextual support and macro-level contextual support can be crucial to a program's success. Micro-level contextual support comprises social, psychological, and material supports that clients need in order to allow their continued participation in intervention programs. For example, under current welfare reform laws, in order to receive benefits, mothers must attend job training or find work. But these reforms present two immediate problems: Is transportation available to get the women to the workplace? And who will care for the children while they work? A welfare-to-work program is hardly manageable
without tackling these issues. Furthermore, clients may be more likely to participate seriously in programs when they receive encouragement and support from their immediate social units (typically family, peer group, and neighborhood). When program designers or implemented realize that micro-level contextual support could play an important role in an intervention, it is up to them to try to build this support into a program's structure. For example, designers 0f an alcohol abuse program might organize a support group for clients that includes family members and peers who encourage and support them during and/or after intervention. In addition to micro-level contextual support, program designers should consider the macro-level context of a program; that is, its community norms, cultures, and political and economic processes. These, too, have the ability to facilitate a program's success. A residential program for the mentally ill can anticipate real difficulties if the local community has a generally hostile attitude toward its clients. But if an adequate campaign for community support of such patients is one component of the residential program's implementation, these difficulties may be alleviated. In any case in which stakeholders believe macro-level contextual support to be crucial to their program's success, the generation of this kind of support should be included as an element of their program.
Great effort goes into ensuring the capabilities of implementing organizations, establishing collaborations with associate organizations, and winning contextual support, if these are truly done well. Finding resources with which to make the effort can be a challenge. There is a worthwhile payoff, however. If a program does succeed in these activities, it is considered an ecological, or multilevel, intervention program: that is, a program with goals not just for individual clients but also for the surrounding community. Ecological programs may be likelier to attain their goals than are programs concentrating simply on client issues. This element signals a need to take a holistic approach to conduct program evaluation.
Target Population: Identify, Recruit, Screen, Serve
The target population or group is the people that the program is intended to serve. Concerning target population, three assumptions that often figure in evaluation are the presence of validly established eligibility criteria, the feasibility of reaching eligible people and effectively serving them, and the willingness of potential clients to become committed to or cooperative with (or at
28
INTRODUCTION
least agreeable to joining) the program. Faced with resource constraints, a program usually cannot provide services to everyone in a target population. Therefore, it needs a clear and concrete boundary for eligibility. Criteria must also be established by which the program determines which specific populations it will recruit. For example, the target population of one Head Start program is preschool children from disadvantaged families residing in a particular community. Similarly, an HIV prevention program in one community chooses to serve addicts who inject drugs rather than trying to target the entire high-risk population. A program is usually regarded as ineffective if it finds itself serving the wrong population or failing to reach enough members of the right population. A nursing care program intended to serve low-income elderly people, for example, has failed if its services benefit many comparatively well-to-do people. Similarly, a job training program that is well funded and well run will have failed if it produces only a handful of "graduates."
Whether or not clients are prepared to accept intervention also can affect program outcomes. Especially for labor-intensive types of programs, client screening and assessment are vital. Identification of actual needs is vital, and information from assessment can suggest whether a client needs services in addition to the central intervention. For example, when assessment reveals the need, clients can be referred by program staff for housing assistance, mental health care, education, employment, or other social services. A labor-intensive program must be certain of its clients' readiness for intervention; client readiness being the extent to which an individual's mental and physical state permits his or her acceptance of an intervention. If clients are not mentally and physically ready for it, intervention is unlikely to work. Mental readiness of a client is the degree of his or her willingness to recognize a problem or deficiency, or the degree of motivation to accept an intervention. For example, a person who insists alcohol is not a problem for him or her will probably not succeed in an alcohol abuse counseling program. Clients also exhibit varying degrees of physical readiness for interventions. Health status affects delivery of some interventions. For example, counseling clients about HIV prevention can be difficult when they suffer from severe mental health problems or have no food or shelter. In such a case, the successful intervention program is likely to provide case management or referral services to meet basic needs prior to beginning intervention. Similarly, a client still under the influence of alcohol is no more physically ready than mentally ready for intervention. Trying to deliver alcohol counseling services is futile until the client has completed a detox program; alcohol abuse intervention starts once the client is sober.
j Conceptual Framework of Program Theory for Practitioners
29
Action Model
Figure 2.1       Conceptual Framework of Program Theory (Basic Form)
RELATIONSHIPS AMONG COMPONENTS OF THE CONCEPTUAL FRAMEWORK OF PROGRAM THEORY
It is important to understand relationships among program components. The relationships among components can be pictured as in Figure 2.1.
In general, program components need to be organized or connected in a meaningful way in order to achieve the goals. Figure 2.1 shows how an action model is implemented in order for a change model to activate the causal process. The double-banded arrows between two components within the action model represent a sequential order between these two components. That means that the completion of some component provides the basis for completing the next component. For example, in the figure, the double-banded arrow from "implementing organization" to "implementers" indicates that it is usually a requirement to have a capable implementing organization in place in order to adequately recruit and train implementers. With a spouse abuse intervention program—or virtually any program—this means that there must be an organization responsible for implementing the program before counselors or clients . can be recruited. In other words, the relationships among components of the action model represent a kind of "task order" relationship: Some components must be in place and complete before others can be brought in line. The only exception
30
INTRODUCTION
4 Conceptual Framework of PrognimJTwoiyfor Pracliliorwrs_
31
is the two-way double-banded arrow between the box of the implementing organizations and the box of associate organizations and community partners. This means very often the associate organizations and community partners collaborate with the implementing organizations in planning program activities at the same time.
On the other hand, the solid arrows within a change model in Figure 2.1 depict causal relationships in the change ..model,. A causal relationship exists between elements when changing one creates change in the other(s). A solid arrow leading from an intervention to a determinant represents the model's assumption of a causal relationship between the two. In the spouse abuse program, the model assumes that group counseling has the power to create anger management skills in abusers and to teach them about the legal punishments associated with spouse abuse.
The conceptual framework should make clear that the action model must be implemented appropriately in order to activate the transformation process in the change model. In order for a program to be effective, its action model must be sound and its change model plausible; its implementation is then also likely to be effective. For example, in order for an HIV prevention outreach program to succeed, it needs to coordinate activities, reach the target group, and provide the group with adequate exposure to the prevention message; it must also determine which activities will strengthen the target group's knowledge of risk prevention, which should manifest itself in decreased high-risk sexual behavior. This conceptual framework of program theory should be useful to evaluators charged with designing an evaluation that produces accurate information about the dynamics leading to program success or program failure.
If evaluators and stakeholders want mainly to highlight the relationships among the components of program theory, Figure 2.1 is sufficient. However, Figure 2.1 does not address the relationships among program, environment, and feedback discussed in Chapter 1. For evaluators and stakeholders interested in elaborating these further relationships, a comprehensive diagram, such as Figure 2.2, is necessary.
In Figure 2.2, the large square around the program represents its boundary. Everything within the large square is part of the program; all that is outside the square is "environment," providing the program with necessary resources and support (in other words, its inputs), or, perhaps, working against implementation of the program. Figure 2.2 shows that, generally, a program starts with the acquisition of resources from the environment and the development of an action model. Fueled by the acquired resources, the action model can be
Resources
Figure 2.2      Conceptual Framework of Program Theory (Comprehensive Form)
implemented in order to activate the change model. It is the operation of the change model that leads to the attainment of program goals. Solid arrows joining an action model to a change model indicate that, strictly speaking, whatever effect the program has on the outcomes is not due to the implementation of intervention alone but to a joint effect of the implementation of intervention and the implementation of other factors in the action model. Evaluation feedbacks are represented in dotted arrows, The evaluation feedback in the figure comprises information about how the action model was implemented in the field, such as whether the program reached the right target population,
Similarly, the dotted arrow from the implementation to action model indicates that evaluation feedbacks from the implementation can be used to improve the planning or the development of the action model. The dotted arrows from the change model to the implementation and action model indicates that the information from the causal process of the change model can be used to improve or modify the implementation process or the planning of the action model. The conceptual framework provides two distinct general evaluation feedbacks: the internal and the external. The dotted lines in Figure 2.2 represent evaluation feedback and feature two sets of "feedback loops." Each
INTRODUCTION
set of evaluation feedback loops indicates one path that program evaluation can follow to obtain information vital to program improvement. Each path accommodates distinct audiences and purposes. Therefore, the evaluation approaches and strategies involved in various evaluation feedback loops can be quite dissimilar. The first set, constrained inside the program boundary, is for an internal audience of program implementers, administrators, and others who deal with programmatic concerns and .service delivery matters on a daily basis. This audience wants from the program evaluator timely information on whether a program is operating smoothly in the expected manner. If there are difficulties, the internal audience wants to understand, if possible, the sources of the problems as well as the likely remedies. This aspect of evaluation is called internal use evaluation or development-oriented evaluation. Strategies and techniques of internal-use evaluation must be flexible and creative, and they must be accomplishable quickly. If a program is not on the right track, its course must be corrected before too much time and energy are wasted.
The remaining set of feedback loops in Figure 2.2 passes to the environment and then back again to the program. This is the external feedback loop, incorporating both scrutiny by the environment and improvements from the program itself. Conducting external feedback evaluation requires more resources and more time than conducting internal feedback evaluation. The audience for external feedback is funding agencies; decision makers; interested groups; the public at large; and the stakeholders who work inside the program, such as program directors and implementers. The external feedback loop represents a mechanism that delivers to the environment information about the merits of a program, what changes the program may need, and the appropriate general direction for the program in the future. There are two types of evaluation relating to the external feedback loop. One is intended to serve accountability needs and is called assessment-oriented evaluation. The other is designed to serve both accountability and program improvement needs and is called enlightenment-oriented evaluation.
These different types of evaluation will be discussed in detail in the remainder of the book.
APPLIED PROGRAM THEORY: AN EXAMPLE
A good example for the application of program theory for program evaluation is found in an evaluation of an antismoking program (Chen, Quane, & Garland, 1988). Program designers devised a comic book with an antismoking
4 Conceptual Framework of Program Theory for Practitioner^
33
Figure 2.3      Program Theory of an Antismoking Campaign SOURCE: Adapted from Chen et a], (1988).
story as an intervention to change students' knowledge, attitudes, and behavior concerning smoking. Program designers expressed a desire for an outcome evaluation of the program that would provide information needed to make improvements to the program. The program theory supporting the program was stakeholder theory, stemming from the program designers' own ideas and experiences. Evaluators conducted intensive interviews to clarify the stakeholder theory.' The stakeholder theory is illustrated in Figure 2.3.
Change Model
The program designers' main idea for the program came from their observation that teenagers are fond of reading and collecting comic books. Accordingly, they thought a comic book that conveyed an antismoking message would create an opportunity for students to absorb selected facts and change their attitudes and behavior concerning smoking. More specifically, the program designers' change model contained two determinants in a sequential order: the students' enjoyment of reading comics and students' familiarity with
34
INTRODUCTION
AC
oiiceptual Framework of Program Theory for
Practitioners
35
the characters (heroes and villains) and story. The designers hypothesized that these determinants would lead to stronger antismoking beliefs and behaviors.
Action Model
The program designers had in mind a story, characters, and even scripts, and they collaborated with a community-based organization to implement the project. They proposed hiring a comic book artist to draw the pictures and a project coordinator and staff to run the program. They named a target population— young people attending middle school—and sought support from principals, teachers, and parents in encouraging students to participate. They planned to distribute the comic book in health classes.
After the evaluation was conducted, results showed the program to be well-implemented based on the proposed action model. Results for the change model were mixed. Although students read and kept the comic book, possessing it as expected, these determinants alone were not sufficient to translate into attainment of the intervention goals. The evaluation showed where the program had misstepped in the change model. The information was useful for program stakeholders to design a better program in the future.
RELATIONSHIPS BETWEEN
"PROGRAM THEORY" AND "LOGIC MODEL"
The terms program theory and logic model have been used i nterchangeably in the literature, but, in fact, the two serve separate purposes. A logic model is a graphical representation of the relationship between a program's day-to-day activities and its outcomes. It does resemble program theory in its employment of diagrams to aid the analysis of programs, and yet conceptualization of programs within program theory and within the logic model is utterly distinct. Program theory is a systematic configuration of prescriptive and descriptive assumptions underlying a program, whereas the logic model stresses milestones like components. Wholey (1979) rendered the logic model in two primary parts: the program components, and the goals and effects of the program. Program components are activities that can, whether conceptually or administratively, be grouped together.
Building on Wholey's work, subsequent versions of the logic model have tended to add parts to the logic model. One popular twist on the model is the version developed by the United Way of America (1996). With it, evaluators
Resources dedicated What the program to or consumed by    does with the inputs
the program . Money
• Staff and staff time . volunteers and
volunteer time . Facilities . Equipment and
supplies
to fulfill its mission
Examples
• Feed and shelter homeless families
• Provide job [raining
• Educate the public about signs of child abuse
• Counsel pregnant women
• Create mentoring relationships for youth
Constraints on the program
Rtarnples . Laws
. Regulations
• Funders' requirements
The direct products    Benefits for of program activities  participants during and after program activities
Examples
• Number ot classes taught
• Number of counseling sessions conducted
• Number of educational materials distributed
• Number of hours of sen/ice delivered
• Number of participants served
Examples
• New knowledge
• Increased skills
• Changed attitudes or values
. Modified behavior
• improved condition
• Altered status
Figure 2.4       The United Way's Logic Model SOURCE: Reprinted with permission of the United Way.
of United Way programs consistently examine inputs, activities, outputs, and outcomes. In this particular logic model, inputs are defined as resources dedicated to or consumed by the program: money, supplies, staff, and even ideas. Activities in this logic model comprise actual services or work that the program provides to fulfill its mission. Examples include recruiting and training staff, counseling clients, providing referral services, and educating the public. In the United Way model, outputs are defined as the direct products of program activities: number of clients served, number of classes taught, amount of goods distributed, and so on. Finally, this logic model defines outcomes as the actual benefits resulting from program activities. Examples are improved health, new knowledge, better skills, and higher income. Relationships of these elements to each other are illustrated in Figure 2.4.
It is possible to view the logic model as a simple version of program theory. The logic model's parts essentially comprise generic categories of