Centre Publications

The Promise of Evaluation: What Evaluation Offers Policymakers and Practitioners

by Clem McCartney
Published by the University of Ulster, Coleraine 1992, reprinted 1994
ISBN 1 87120 613 8
Paperback 19 pp £1.00

Out of Print

This material is copyright of the Centre for the Study of Conflict and the author(s) and is included on the CAIN web site with the permission of the publisher. Reproduction or redistribution for commercial purposes is not permitted.

The Centre for the Study of Conflict welcomes this study by Dr Clem McCartney on the subject of evaluation. The need for accountability has led to an increasing demand that the consequences of social action be examined with care and with a level of sophistication. This paper is an important contribution to thinking in this area. It is a thoughtful and closely analysed review of the components of the evaluative process by an experienced practitioner who has reflected on his own practice. I am sure that it will be found valuable by all those involved in the worlds of community relations, and social and community development within Northern Ireland, and further afield.

We are grateful for help of various forms from a range of funders and individuals in the production of this work, including Paul Sweeney and NIVT, Tony McCusker and Julie Mapstone, CCRU and PPRU, Eric Adams and the Barrow and Geraldine S. Cadbury Trust.

Before publishing a research report, the Centre for the Study of Conflict submits it to members of a panel of external referees. The current membership of the External Advisory Board comprises:

Dr Halla Beloff, Department of Psychology, University of Edinburgh;
Dr Paul Brennan, UER Des Pays Anglophones, University of Paris III;
Professor Ronnie Buchanan, Institute of Irish Studies, Queen's University Belfast;
Professor Kevin Boyle, Centre for the Study of International Human Rights Law, University of Essex;
Professor John Fulton, School of Education, Queen's University Belfast;
Dr Richard Jenkins, Department of Social Anthropology, University College Swansea;
Dr Peter Lemish, Department of Education, University of Haifa;
Professor Ron McAllister, College of Arts and Sciences, Boston, USA;
Dr Dominic Murray, Department of Education, University College Cork;
Professor James O'Connell, School of Peace Studies, University of Bradford;
Professor John Rex, Centre for Research in Ethnic Studies, University of Warwick;
Professor Peter Stringer, Centre for Social Research, Queen's University Belfast;
Professor Joseph Thompson, Department of Politics, University of Villanova, Pennsylvania.

Seamus Dunn
Director, Centre for the Study of Conflict
March 1992.

Return to publication contents

This paper reflects the interest of the Centre for the Study of Conflict in the practice of evaluation. It is prepared for those who are interested in using the results of evaluation studies: policy makers, funders, programme managers, and practitioners. There is often confusion about what an evaluation study tells the reader and how they can use it confidently in future planning. This is because there are many types of evaluation studies, and many types of evaluation reports, though superficially they may look similar. This paper identifies the different types of evaluation work and indicates the value and limitations of each, and to whom it should be particularly relevant. It should be of help at the stage of commissioning evaluation studies in clarifying what type of study should be undertaken. It is offered also in the hope that it will help in recognising the type of report which has been offered and appreciating what answers it should give and what answers it is not able to provide.

In recent years, evaluation has come close to achieving the status of a talisman in the hands of social policy makers and practitioners. It is hoped that it can provide the answers to many, if not all, the difficult decisions which have to be taken about social issues. Its range is from decisions about renewing grants as small as £50 for voluntary community groups to assessment of the continuation of major statutory services and programmes with budgets of billions of pounds. A thorough and expensive evaluation of a comparative small project may still be justified as a source of guidance for similar programmes on a wider scale.

There is no doubt that evaluation does have an important function to play in the policy process; in fact it can fulfil a number of functions. But it is important to be realistic about what to expect from any specific evaluation study. All evaluations are not the same. They do not address the same questions, and therefore they do not offer the same insights. The danger arises when evaluation is viewed as a uniform process, so that all evaluations are expected to offer similar types of insights. Evaluation then is unable to fulfil its promise, and may be criticised unfairly.

The purpose of this short paper is to consider the expectations which are held about evaluation and compare them with what different types of evaluation can offer. It is hoped that this will allow all those concerned to accept a common set of expectations and that more clarity will be the result. For any programme or service, the funders, policy makers, practitioners, service agencies, and the recipients of the programme are all interested in how it has been introduced and implemented. Many of their questions and their expectations of an evaluation study will be conflicting, but they may not always be aware of what type of evaluation will best answer their questions. Equally they need to know what kinds of questions a specific type of evaluation will not answer adequately, so that they will not be disappointed by the results or try to shape them to give insights which the study cannot sustain.

There are particular difficulties in evaluating some programmes, and they highlight the issues in evaluation very graphically. One such area is community conflict and the examples in this paper are mostly taken from that type of work.

Evaluation, in the sense of learning from experience, has been part of human endeavour since the beginning of time. But evaluation as a systematic and conscious process is of recent origin. Evaluation is concerned with activities and programmes, and their impacts and achievements, whether intended or unforeseen. There are a number of aspects to any good evaluation. It starts with questions to be answered. It collects information or data as the basis for answering the questions. It then analyses the Information in some way depending on the type of questions being addressed. Finally it presents conclusions or suggested conclusions, which, in the hands of planners and practitioners, are a tool for developing future work. Clearly some conclusions will be of greater benefit in some circumstances than in others, depending on the nature of the policy and practice issues which need to be resolved.

This paper follows this pattern in order to distinguish different types of evaluation. It considers each of the main groups of questions with which evaluation is concerned, and tries to show what kind of information it can give, what the value of those findings are, and who is likely to be Interested in them. It emphasises the importance of considering what the findings mean and what are their limitations, as opposed to the easy assumption that the findings give a direct prescription for future action. Other terms such as "monitoring" and "assessment" are often used interchangeably with "evaluation", but it causes less confusion if these terms are restricted to specific activities which are part of the general field of evaluation and these terms will be referred to again when the relevant aspects of evaluation are being discussed.

This paper therefore categorises elements of evaluation according to their possible functions or contributions to the development of policy and practice. For the sake of clarity, a number of important issues are only touched on briefly, as they are not central to the present discussion. The relationship between evaluators and those being evaluated has both practical and ethical implications, but that issue has been dealt with elsewhere (e.g. Weiss 1991). Other typologies such as one based on methods of investigation are not discussed in detail, but the final part of the paper relates the main kinds of methodology to the categorisation used here. The practical problems of information collection and analysis have been adequately discussed already (e.g. Ball 1988). The presumed dichotomy between quantitative and qualitative methods is only touched on briefly. Here the emphasis on functions of evaluation supersedes this controversy, because all methods can contribute to finding answers, and conclusions which are supported by a range of methods will be the most reliable. The danger lies in presuming that too much can be learnt from one method.

Return to publication contents

DESCRIPTIVE EVALUATION:
WHAT HAPPENED?

This is perhaps the most basic question and the first step in all evaluation work. It is important to know what actually happened. This involves recording numbers: how many people were involved, how much money was spent, for example. It is also concerned with collecting "softer" information about the nature of what happened: the type of programme being delivered, how it was implemented, who received the service or how people participated in it. Often these investigations are referred to as "monitoring", the process of collecting agreed information and comparing actual performance against expectations. However the term is often used more loosely to refer to evaluation in general.

Descriptive evaluation is important to all those concerned with the development of programmes and practice, because the answers to all the other questions depend to some extent on knowing what actually happened. The case study is a common example of a descriptive account, and is useful to practitioners as a source of ideas for programme content and activities. It is particularly important in experimental work where there is doubt whether the proposed programme is feasible. For example, this kind of documentation of the Schools Cultural Studies Project showed that controversial issues could be discussed in the classroom (Jenkins 1980), and this should no longer need to be proved. Funders of programmes find descriptive accounts useful in demonstrating if ‘the resources have been used as agreed, and it can thus serve an expenditure monitoring function.

While descriptive evaluation seems superficially straight-forward, there are a number of problems in practice. It is difficult to demonstrate the spirit of what happened as opposed to the surface procedures. It is possible to describe the commitment and enthusiasm which went into the programme, but it is a more difficult task than compiling an inventory of events and activities. A farther problem is separating observation and collection of data from analysis. The decision to collect specific information and to ignore other possible findings is itself based on some initial analysis. Equally the descriptive data often suggests deceptively obvious conclusions. There is a tendency to make judgements about the implementation of the programme or specific activities solely on the basis of a descriptive account. However it is important to view the description and analysis as discrete activities so that one can distinguish fact fromjudgement. This approach can indicate what happened, but, in itself it offers no basis for making any assessment. It can indicate if the programme was carried out as intended, but it does not give any insight into whether It was worth doing. It may show that the people implementing the programme did not do exactly what it was expected they would do. However before criticising, it is important to investigate why the discrepancy occurred.

For example a children’s playscheme might get a grant designed to support inter-community contact, even though the group had not claimed in their application that it promoted inter-community contact. In these circumstances the description of the project may show that the project did not fulfil the objectives of the grant, but the cause must be related to the decision of the administrators of the grant and the reasons they chose this project, rather than to the playscheme’s activities. It is situations such as this that make evaluators hesitant to carry out a descriptive evaluation without including other types of evaluation, because they are concerned that the findings will be misunderstood and used to support explanations or views that can not be justified on the basis of the data available.

Return to publication contents

CHANGE MEASUREMENT:
WHAT CHANGES HAVE OCCURRED?

Trying to measure change involves two sets of questions: what was the situation like before? What was the situation like afterwards? The second question will be considered first.

This element is concerned with understanding what the situation was like after the programme. This is again a matter of collecting basic information, and there are many sources for such a study: in the example of interventions in community conflict they could include people who took part in the programme, friends and relatives of the participants, the wider local community, and behavioural patterns as an expression of attitudes. It is important to identify aspects of the situation which may have been affected by the programme, but the investigation should not be too narrow, as one may not be able to identify in advance those aspects of the situation which have been affected by the programme.

The programme also may have had an impact on the organisation and individuals implementing it, and they should be included in the investigation. A programme may have had a radical impact on the organisation’s view of its role (International Voluntary Service/Harmony Community Trust 1987), even if it had little impact on the participants.

None of this information is very meaningful in itself, and it will not normally be collected in isolation. In isolation it does not take forward the process of evaluation. There are two problems. Firstly how does one determine if the situation observed is desirable. There may be different views on this, especially in the field of community conflict. There may not even be a consensus that a lowering of levels of violence is positive, as one side may feel this weakens their position and others may feel that the problems still exist with no hope of resolution even if the overt violence has been reduced. Northern Ireland Catholics who support violence would use this line of reasoning, but a pacifist group has also put the dilemma clearly in relation to demonstrations against Orange marches through a Catholic housing estate: "if we do not demonstrate we do nothing to remove an important symbol of disrespect. Ifwe do demonstrate. Orangemen and others can see us as engaged in a childish power struggle" (Drumcree Faith and Justice Group. 1991, p 13.) In this context it is difficult to evaluate the impact of their actions.

Even more problematic is a situation where there has been a lowering of the sense of group identity. This may demonstrate a process of acculturisation which the participants may not want to happen, especially if it is predominantly happening on one side, and Is the beginning of a process of assimilation of the group with a weaker sense of identity by the group with a more confident or stronger sense of their identity. In Northern Ireland this fear of cross community contact is expressed most often by Protestants (Gallagher and Dunn, 1991).

Secondly one can make no evaluation of the information as an indicator of change because one needs a standard to compare the current situation against. This will sometimes mean investigating a comparable situation or control which has not been subject to the programme under review, but the difficulties with selecting a genuine comparison have been well documented. For example one psychological study (Trew and McWhirter 1985) of a contact programme for teenagers selected as a control group teenagers, many of whom became involved in other contact activities during the course of the study. The other aspect of assessing change is to understand the situation before the programme, in order to make a comparison with the situation afterwards.

Similar information needs to be collected as in the previous question, and ideally it should be done before the programme has started. Sometimes aspects of the situation which need to be examined only become apparent in the course of the evaluation, and In this case there is no alternative to a retrospective investigation. It is easier to do this with existing statistics, but they may not have been collated in a form which is appropriate to the study. Much other information will not be available or, if it is recalled by those involved, it may not be reliable, especially if they are reporting on attitudes. This reinforces the importance of planning an evaluation before the programme is begun and as part of the planning process.

This type of study is very attractive, even seductive, to policy makers because of its apparent scientific basis, and in the past this has been one of the main types of evaluation activity. But there is one important reservation which undermines its apparent authoritative findings and which makes evaluators Increasingly unwilling to use this approach in isolation. It is important to stress that the identification of change does not in itself Indicate that the programme had an impact. It shows that change took place between two points in time, during which the programme took place. But many other activities and processes could have been wholly or partly responsible for the change, and these must be considered before drawing conclusions.

Return to publication contents

CONTEXTUAL ASSESSMENT:
WHAT ELSE WAS HAPPENING ALONGSIDE THE PROGRAMME?

The context within which the programme was taking place is a very important but diffuse element in any evaluation. The context can include the wider locality in which the programme was taking place, or from which the participants come. It may be changes In the social and political situation. Particularly important is the organisational context in which the project took place.

This information makes a two-fold contribution. Firstly it indicates other important developments which may have had an impact on the situation, and then one is in a position to begin to look for evidence of their relationship to any changes identified. Even comparatively minor events may be relevant, including the weather during an activity. Many groups have commented on the Impact, both positive and negative, of some violent incident which happened at the same time as their project (McCartney, 1990).

The understanding of the organisational context offers a different kind of insight. Attitudes of key people towards the programme will have an important bearing on the way the programme is implemented. It would be wrong to assume that the organisation as a whole or those directly responsible are committed to the programme. Sometimes they have personal motives which have nothing to do with the aims of the programme. Sometimes they may feel that they have no choice but to do it. They may be committed to a different programme which looks sufficiently similar to Introduce under the guise of the avowed aims. A good example in the past was groups who were able to finance schemes to offer deprived children holidays by using funds intended for cross-community contact holidays. There is an extra problem in relation to work on community conflict because there are a wide variety of assumptions underlying this work, and many people do not clearly identify what are their own assumptions.

It is therefore important to add this contextual element by collecting the available information on the initial development of the project in reports, correspondence, minutes of meetings and recollections of those concerned.

Return to publication contents

IMPACT ASSESSMENT:
WHAT WAS ACHIEVED? WHAT WAS THE OUTCOME OF THE PROGRAMME?

This element of evaluation is less a process of data collection, but it is dependent on the data collected under the questions already discussed. If there is very little solid information available any analysis will be very unreliable. While data collection Is the foundation of this aspect of evaluation, the main procedure is analysis. It is concerned with making causal connections, and looking for patterns and relationships in the information. Apparent contradictions are a fruitful starting point for analysis. This can employ statistical analysis, if figures are available, but it can also involve more subjective assessment incorporating the judgments and assessments both of the participants and implementors of the programme, and of the evaluators themselves.

It would be wrong to assume automatically that either of these sources of information is more reliable than the other. It is easy to place more weight on the ostensible objectivity of statistics, but the figures themselves may hide subjective judgments and the limitations of the statistics must always be recognised.

On the other hand, it is becoming more common to recognise the ability of more qualitative methods to indicate the subtleties and complexities of social change process, but again one must be cautious, as the selective elements in this process can distort the findings. Each approach makes a contribution and the most strong conclusions are those which balance both types of approach and can create a new synthesis in reconciling the different findings.

The term "assessment" is often used to refer to all kinds of evaluation, but it is more appropriate to use it for this type of analysis of the quality and value of what is being done. It should be stressed that evaluation can seldom give a categorical statement of the impact on a programme. Ultimately, evaluation produces findings which guide future action and will be confirmed or challenged by the outcome of future activities.

Return to publication contents

POLICY PROCESS ANALYSIS:
WAS THE PROGRAMME IMPLEMENTED AS PLANNED? WHY DID CHANGES OCCUR?

An important purpose of evaluation is to learn from experience. In this sense a programme which is not very successful may have more lessons to offer than one which progresses smoothly. Some of the most instructive evaluation reports are of projects which had significant problems. The focus here is again on analysis, as contextual assessment will provide most of the basic data. The desired outcomes of all interested parties need to be identified. Participants themselves will have different reasons for becoming involved and their interests will also influence the shape which the programme finally takes.

Having identified the expected outcomes of the different constituencies, one can compare each of them against the outcomes one is able to identify. If there is a discrepancy in that some outcomes have not been achieved and others have happened without being intended or perhaps even desired, then the process of analysis will be directed to explain the discrepancies. There are three main factors. The way the policy has been developed in the organisation gives useful insights into policy processes including programme development. Secondly, there maybe external factors which have disrupted programmes. And lastly there may be practice issues, in that the way the work was carried out may not have been appropriate. As a result the study can guide practice and indicate procedures and behaviours which are effective and those which may be counter-productive to one’s overall aim.

Implementors of programmes therefore have a special interest in this type of evaluation as a guide to improving their work and developing their skills. It is also important, in association with impact assessment, as an indication of ways in which the programme was unable to meet its targets and therefore it can help to explain why the impact may have differed from that Intended.

The main concern with this area of evaluation is because it is very revealing of limitations in practice. To be most useful it requires a willingness to reveal one’s work including mistakes. If the findings are then used as an impact assessment exercise by funders it can lead to criticism and the withdrawal of funding at the point at which lessons have been learned for future action.

Return to publication contents

POLITICAL JUDGEMENTS:
IS THE CHANGE WORTHWHILE?

This question is in one sense not a question about evaluation at all. The evaluator can only measure the outcomes against some standard, either a social consensus on the desired outcome, or against the stated aims of the programme. The judgement as to whether the direction is the right one to take must be decided on grounds of policy and the underlying philosophy of the policy makers and society as a whole through open debate. However there is one contribution to the debate which the evaluator is well placed to make, and that is to comment on the effect of the outcomes. These may be long term, but the evaluator can seek information on the consequences of the outcomes and extrapolate from there into the future. In this way evaluation should play an important part in not only indicating the way practice and policy implementation takes place, but also in contributing to the debate about how society is developing and the preferred ways of resolving social problems.

Return to publication contents

OTHER TYPOLOGIES:
FUNCTIONAL AND METHODOLOGICAL

This paper has developed a typology of elements of evaluation using their function in clarifying issues as the key feature. Six elements or functions have been identified, descriptive, change measurement, contextual analysis, impact assessment, policy process analysis, and political judgement. Alternative classifications have been based on the research methods used. They are not discussed in detail here, but they include the traditional scientific paradigm, participative, ethnographic, process analysis, and phenomenological methods. These approaches are often seen as alternatives and there is some debate as to the most appropriate approaches. However if one examines the functions of evaluation, as is done here, it quickly becomes clear that different methodological approaches can play a part in clarifying issues, but each will have strengths which are particularly relevant to specific functions.

Thus the scientific paradigm, which tests hypotheses, is a method which will have an important part to play in change assessment. Participative methods, which involve the people with an interest in the programme, will be very helpful in the descriptive evaluation and policy process analysis where those involved can help to identify key processes going on. It is also an important tool in impact assessment where we have seen it is important to develop an agreed statement of desired outcomes. It is important to consider not only how the evaluation can be helped by the participation of those with an interest in the programme, but also their right to be consulted about an evaluation which affects them. Ethnographic study, using the anthropological approach, can illuminate a descriptive account and provides a means to Incorporate qualitative material. Process analysis, was developed for the examination of policy formulation and implementation and is accordingly useful in policy process analysis. The phenomenological approach, which has developed ways to find underlying meanings behind phenomena and thereby make sense of disparate information, is useful across the whole range of functional elements especially where qualitative material is analysed and conclusions are drawn.

Return to publication contents

CONCLUSION

This paper has not tried to weigh different elements of evaluation against each other. They all have different and important contributions to make, and It is important to know which is suitable in different circumstances. One would not use a push bike to cross the Atlantic, nor would one use a jumbo jet to go to the corner shop. The more elements which are included in an evaluation the more comprehensive and useful should be the findings. To use the transport analogy in a different way, the more one pays for a car the greater the range of equipment one can expect. So we should not pick a vehicle just because it happens to be in the show room.

In commissioning evaluation research, similar considerations apply. We should not chose a method just because it is convenient. Nor should we select researchers before we have decided what we want them to do. The first step Is to ask ourselves the kinds of questions we want answered, and here the sub-headings used in this paper can be of help. For example, do we want to know what happened in the programme or do we want to know what changes have occurred? Secondlywe need to knowwhat resources we have for the evaluation study. In many cases we will have to make compromises, but if we have to settle for less than we would want to do, it is important that we know what questions our scaled-down evaluation will not answer.

Having decided what questions we think can be answered within our budgetary constraints, we are ready to commission a study, or carry it out personally. On the basis of our decisions about what we want from the study, we will now be in a position to identify the qualities we want from our researchers. What skills should they have? Do they need to be familiar with the subject of the study? For some kinds of study this may not be necessary, but, for example, in descriptive evaluation it is important that the researcher understands the nature of the work being done.

Ideally those with an interest in the programme will have been involved in this process and there will be agreement on what questions about policy and practice we are expecting the study to answer.

If the planning of the research is considered in this way the results will be more likely to match the expectations of those commissioning the study. Equally if we read research reports in a similar way, and consider what questions it is legitimate for the report to try to answer, we will be more likely to make realistic assumptions about what the report can tell us. And in this light, we can discuss the report on an agreed basis, and make decisions about the practice and policy implications of the findings

Return to publication contents

REFERENCES

Ball, M. (1988) Evaluation in the Voluntary Sector London: The Forbes Trust.

Drumcree Faith and Justice Group (1991) The Story of a Journey: A response to "Breaking Down the Emnity" Portadown: Drumcree Faith and Justice Group.

Gallagher, A.M. and Dunn, S. (1991) "CommunIty Relations in Northern Ireland: Attitudes to contact and integration" in Stringer, P. and Robinson, G. Social Attitudes in Northern Ireland Belfast: Blackstaff Press.

International Voluntary Service/Harmony Community Trust (1987) Checking out the Stalemate Belfast: IVS/HCT.

Jenkins, D. (1980) Chocolate, Cream Soldiers: A final evaluation report on the Rowntree Schools Cultural Studies Project Coleraine: New University of Ulster Education Centre.

McCartney C. (1990) Making Ripples: An Evaluation of the Inter-Community Contact Grant Scheme of the Northern Ireland Voluntary Trust, Coleraine, Centre for the Study of Conflict.

Trew, K., McWhirter, L, Maguire, A. and Hinds, J. (1985) Children’s Summer Programme in Greensboro: Evaluation 1984-85 Belfast: Department of Psychology, Queen’s University.

Weiss, C. (1991) "Evaluation and Education.-at Quarter Centrury" in Ninetieth Yearbook of the National Society for the Study of Education, Chicago: National Study for the Study of Education.

Return to publication contents