The section gives a brief introduction to some of the issues in evaluation. It considers the purpose of the evaluation, some types of evaluation, the "measurables", and the methods which might be applied. The section is not intended to be a comprehensive guide to evaluation, but tries to give a flavour for the issues involved. A reading list is given at the end for those who wish to study evaluation further. Sample quotes and statistics raised throughout the research are also included.

Why should we undertake evaluation

There are a number of different reasons why evaluations are carried out. Sometimes groups and projects need to reflect on their raison d’etre, review their objectives, think through the activities they engage in and review their practice. This is an important part of project development and is a process which can be facilitated by supportive and reflective evaluation. Sometimes this is called formative evaluation because it helps groups and projects in the formative stages of their work.

Funders of groups and projects often require evaluation for a different purpose. Funders often need to know how their money is being spent, what kinds of things have been achieved with it and whether the group or project is achieving good value for their money. These kinds of evaluations tend to focus upon the outputs of a projects work and are often called summative evaluations because they sum up the projects achievements.

The needs of projects in evaluation and the needs of funders are not necessarily mutually exclusive. Most funders realise that a group or project which takes time to reflect upon its work, through formative evaluation, is likely to be a better project which meets the funders requirements for effectiveness and value for money. And many groups and projects realise that summative evaluation is important in being able to demonstrate to funders and others that their work can be efficient, effective and good value for money as well as achieving its objectives.

Sometimes it is possible to combine the needs of funders and the needs of projects into one evaluation which not only sums up the projects achievements, but which also helps the project reflect on its direction for the future. Sometimes it is better for groups or projects to have formative evaluation and support separated from a funders evaluation.

What should an evaluation look at?

In deciding what elements of a project to focus upon in an evaluation or in deciding what should be measured it is important to consider which features are most critical or valuable to describe. It is also important to think about what access will be needed both to people and to information, as the availability of both is critical to what the evaluation can cover. Ideally an evaluation plan should be drawn up and agreed by all those who are party to the evaluation, including those who will have to be interviewed or consulted.

It is difficult to be prescriptive about what should be in an evaluation and, anyway, because each evaluation is unique, it would be wrong to prescribe a general set of issues to be covered. As a guide, however, some of the broad issues worth considering for evaluation are:

• What did you set out to achieve
  
• To what extent have you achieved the objectives that you set out with
  
• Have these been useful objectives to have and are those still the things you are trying to achieve,
  
• What has changed in terms of demand or the general environment which might affect the relevance of your objectives
  
• What has impeded and what has impelled the achievements of your objectives;
  
• How well are your processes working, for example, your management, committees, personnel policies, decision making structures, day to day running etc,
  
• Are the activities you engage in the best ones to achieve your objectives
  
• Are your activities in line with best practice
  
• What impact is your work having on your target audience/group/members

A brief word about qualitative and quantitative methods

In the past it has sometimes been supposed that quantitative methods, collecting facts and figures, were appropriate only to summative evaluation and that softer, qualitative, methods such as life histories, story telling and self reflection were only appropriate to formative evaluation. In fact all these methods are valuable and a good evaluation, whether it be summative or formative, should draw as appropriate from both qualitative and quantitative approaches.

Quantitative methods are appropriate in measuring resource inputs, outputs and costs. For example, resource inputs might be staff and capital, costs are the costs of these resources in monetary terms and resource outputs might be the volumes of service (Kendall and Knapp, 1999). Quantitative analysis is also appropriate where there are good sources of data about the project. These data might describe project participants or clients, or they might be the quantitative results of studies on changes in participants attitudes or behaviour.

Qualitative methods are particularly appropriate for an examination of process and quality (Patton 1987) Typical methods would be in-depth interviews, case studies , focus groups, life histories, story telling etc. Qualitative methods flesh out hard quantitative data and give a real feel for the achievements of a project or group.

Who Should Evaluate?

Many people and organisations offer evaluation services. These range from management consultants to academics, researchers and professional evaluators. Each offers a different type of service and it is important to match the evaluator to the type of evaluation that you want. Some evaluators are much better than others at strategic assessment, some have particular skills in helping groups and projects to reflect on their achievements.

Funders will often prefer an evaluator with a proven track record in assessing efficiency, effectiveness and value for money. For groups and projects themselves it is often important to get someone that you can you trust and can feel comfortable with.

Community Evaluation Northern Ireland (CENI) offer a service which helps to match clients with appropriate evaluators. Their address is given at the end of this section.

Finally you should consider the role of self evaluation. CENI has produced a report on the uses of self evaluation and the methods involved. This is worth having a look at if you are interested in some form of self evaluation.

And finally:

Evaluation is always going to be with us. It can be an uncomfortable process for those being evaluated, but it can be a useful learning process if it is done well. Consider all methods and all possible sources of help. And if you are choosing an evaluator for yourself, make sure you get someone who will tell you the truth no matter how painful that is!

Some Views of particpants in the CRPR project

This section contains a selection of the views of participants in the CRPR project. 24 groups stated that the observation of attitudinal and behavioural change offers the best means to assessing project impact upon participants. However as discussed above the exact methods used for observing change have not been specified by respondents. If the practitioner has no formal training in the observation of psychological change how can they accurately assess project impact? 13 CR organisations have stated that one of the main methods utilised when employing the above techniques involved the use of written and verbal evaluation.

Seventeen groups stated that staff commitment to a projects stated aims and objectives was not only essential to the success of the project but also a reliable impact indicator, low staff turnover may highlight the existing staffs satisfaction with progress. It has also become evident throughout the research that project managers felt that treating staff and participants as equal contributors to a project was an essential element of good practice.

Areas suggested for further investigation included discovering which are the most cost effective ways of assessing project impact and how practitioners go about avoiding resource intensive impact assessment. The fact that 18 groups have stated that rising re-attendance figures are an adequate means of assessing project impact has to be examined further, is this really a reliable way of gauging progress?

Relevant Literature

Blackley. S, Goddard. M, and Seymour. H, (1997) Project Monitoring and Evaluation: A Practical Guide, Leeds: ICOM.

Finne. H, Levin. M and Nilssen. T, (1995), "Trailing Research: A Model for Useful Program Evaluation", Evaluation, Vol 1(1) 11-31.

Fitzgibbon. C T, and Morris. L L, (1987), How to Design a Program Evaluation, SAGE, Newbury Park, CA.

Flick. U, (1998) An Introduction to Qualitative Research, London: SAGE

Kendall. J, and Knapp. M, (1999), Measuring the Outcomes of Voluntary Organisation Activities, London: London School of Economics.

King. J A, Morris. L L, and Fitz-Gibbon. C T, (1987) How To Assess Programme Implementation, SAGE, CA.

Lobosco. A F and Newman. D L, (1992), "Stakeholder Information Needs: Implications for Evaluation Practice and Policy", Evaluation Review, Vol 16 (5) 443 - 465.

Love. Arnold J, (1996) Internal Evaluation: Building Organisations from Within, SAGE, CA.

McCartney. C, (1994) The Promise of Evaluation, What Evaluation Offers Policymakers and Practitioners, Coleraine: University of Ulster. (ISBN 1 87120 613 8) 19 pages £1.50

Marshall. C and Rossman. G B, (1989), Designing Qualitative Research SAGE, Newbury Park, CA.

Morris. L L, Fitz-Gibbon. C T, and Freeman, (1987), How To Communicate Evaluation Findings, Sage Publications, CA.

Quinn Patton. Michael, (1987), How to Use Qualitative Methods in Evaluation, New York: Sage Publications.

Robson. C, (1998), Real World Research, Oxford: Blackwell.

Smith, A. and Murray, D. (1997) The Chance of a Lifetime: An Evaluation of Project Children, Coleraine: University of Ulster. (ISBN 1 87120 649 9) 68 pages £4.00

Stecher. B M, and Davis. WA, (1987), How To Focus An Evaluation, SAGE, Newbury Park, CA.

Suchman. E A, (1967) Evaluative Research: Principles in Public Service and Action Programs, New York: Russell Sage.