Contents

Ideal experimental design conditions rarely exist in public policy evaluation. It is not uncommon for evaluators to be faced with a situation where a collection of programmes is being implemented in pursuance of a public policy, couched in nebulous terms, which is more politically defensible than pragmatic. The individual programmes, which collectively equate to "the policy", may have had staggered starting times with no baseline measurements and only a tenuous sense of the causal link between policy activities and impact. This paper describes a policy evaluation in such a scenario. Concept mapping is used as an evaluation technique to assess community relations programmes in Northern Ireland. From this a taxonomy of policy responses is devised and some conclusions drawn on the usefulness of concept mapping as an evaluation technique.

Introduction

One of the more common problems facing public policy evaluators is being invited to undertake an evaluation when a programme has been in operation for some period of time. Not only does this make it difficult or impossible to obtain base line measurements, but monitoring mechanisms which could have been set up to assist a review of the programme are frequently overlooked, or are at best inadequate in the haste to 'get something going' and demonstrate some evidence of policy implementation. Such a scenario is all too apparent when time-frames for the allocation of money to a new programme area are short, finance sources are multifarious (Europe, central government, local authorities, charities) and evidence of activity is required quickly, using staff with limited training and experience who are expected to deliver services instantaneously. If one adds to this the fact that the policy to be evaluated is often couched in the most nebulous objectives, set more to indulge politicians than as an operational framework for officials, then the complexity of the task facing the evaluator gains some perspective.

This is not to shift the burden of "blame" to others. Frequently policy innovation demands broadly defined parameters to allow for flexibility of response by officials in a new or revised policy area. The prescription of policies in quantitatively-expressed objectives with provision for an experimental design (true control group with pre- and post-test measurements), whilst administratively convenient for evaluators, may be inappropriate for the type of programme under review. Our understanding of the impact model, referred to by Rossi and Freeman (1993) as the causal, intervention and action hypotheses, or what links policy activities to outcomes, may at best, be tenuous or at worst erroneous. In such circumstances, which characterise many areas of public policy, the intuitive judgement of both politicians or policy formulators, initially, and officials thereafter, must play a large part in the nature of policy responses for the programmes on offer.

A dilemma therefore exists. New or revised public policies, endorsed by politicians whose careers dictate an urgency for results, are implemented by executive officials with no more than a feel for the cause and effect relationship implied by the policy. Their task produces a variety of policy responses; different types of programmes are offered to attain the broadly defined policy objectives. At some point, normally after the policy has been in operation. the evaluator is invited into this policy quagmire and asked to make a coherent judgement on the cumulative impact of the various approaches on the policy target group.

Although this introduction has been described in abstract terms, the practical reality of the scenario outlined is a familiar one for policy evaluators. This paper describes an evaluation response to such circumstances through a case study of community relations in Northern Ireland. We begin by describing the background to the programme, its objectives and the variety of policy responses by officials. Using concept mapping (multi-dimensional scaling and hierarchical clustering) as an evaluation technique, we attempt to structure the multiplicity of programmes offered under the programme in an effort to assess their relative contribution to the attainment of the programme objectives and as an aid to future planning. Finally. we devise a taxonomy of policy responses from the data and offer some conclusions on the usefulness of concept mapping as an evaluation technique.

Background

Northern Ireland has, since 1969, been the scene of much grotesque sectarian violence resulting in the death of almost 3,200 people. Recent and much welcomed political developments include cease-fires announced by both the Irish Republican Army (IRA) in August 1994 followed soon after by a reciprocal gesture from the Loyalist Military Command.

Aside from these well publicised political developments, a comprehensive program of equality (of opportunity) and equity (of treatment) initiatives has been devised by United Kingdom government to tackle the root causes of underlying divisions and tensions between the two communities. Such an approach recognises that equality and equity issues must be addressed in parallel with efforts at the macro level to achieve progress on the political, security and economic fronts.

This policy is operationalised via a number of initiatives which include targeting areas or sections of the community suffering the highest levels of disadvantage and deprivation with priority public funding, referred to as the Targeting Social Need Initiative. Underpinning this approach is the assertion that community differentials, or greater levels of disad vantage among Catholics (unemployment, education, skills), contribute to divisions in the population. These differential experiences sustain feelings of disadvantage, discrimination and alienation, which in turn influence Catholic attitudes to political and security issues. The high level of Catholic unemployment and job discrimination, a source of much inequality, has also been addressed through legislative changes requiring firms to monitor the composition of their workforce and take affirmative action, where necessary (Fair Employment [Northern Ireland Act 1989). A cross-community contact scheme, administered by the Department of Education for Northern Ireland, was introduced to establish and develop contact between Catholic and Protestant schools, youth and community groups. This paralleled education reforms in schools where two cross-curricular themes, education for mutual understanding (EMU) and cultural heritage, became intrinsic to teaching a range of school subjects under a recently introduced common curriculum. A cultural traditions programme was also established to support arts, museums and Irish language groups in a way which encouraged respect for the richness and diversity of shared cultural heritage.

Alongside these initiatives the government established, in 1987, the Central Community Relations Unit (CCRU), reporting directly to the Head of the Northern Ireland Civil Service on all aspects of relations between the two traditions. The Unit was charged with "formulating, reviewing and challenging policy throughout the government system with the aim of improving community relations" (Central Community Relations Unit, 1992: 2). It was also responsible for developing new ideas which would improve relations, and supporting ongoing efforts aimed at prejudice reduction. A new independent voluntary body, the Northern Ireland Community Relations Council (CRC), was also set up in 1990 to promote better community relations and the recognition of cultural diversity in Northern Ireland. Finally, the 26 local authorities in Northern Ireland were invited, by CCRU, to become involved in a community relations programme in their areas. In short, improving community relations became a significant feature in the government's drive towards the attainment of "equality of opportunity and equity of treatment'.

All of the above initiatives were conceived to attain the following government-declared objectives:

- to develop cross-community contact and co-operation;

- to promote greater mutual understanding;

- to increase respect for different cultural traditions.

This brief description of the various activities offered under the parameters of community relations policy does not do justice to the evaluation morass faced by the researchers. Not only was there a great deal of ambiguity about what actually constituted community relations work, not to mention how its "success" might be operationalised and measured, but the variety of funded programmes and agencies (public and voluntary) involved was daunting. Some evaluation work on individual projects had been carried out as one-off studies but since the policy was still at an early stage of development, most had been formative evaluations, more concerned with process than outcome or impact (Knox, 1994). Equally, because some of the projects involved relatively small government grants, little if any formal evaluations had taken place. The task facing the evaluators was to make sense out of this diverse offering of policy responses to an important public policy, in operation since 1987, within the context of Northern Ireland. What was required was not simply a retrospective analysis of what had been achieved but, importantly, a prospective review which would inform future public funding of community relations programmes. In short, a taxonomy of programme types was called for which gave some guidance as to their relative contribution in "improving mutual understanding between Catholics and Protestants". Such a taxonomy had to be informed by the experiences of those involved in the delivery of policy programmes thus far. Concept mapping provided the mechanism to distil the views of both programme executives and policy officials, and a means of ordering and ranking the various policy approaches which had evolved in an iterative way under the broad parameters of "community relations".

Methodology

Concept mapping has been described as a process in which a group of people devise a pictorial or graphical representation of their thinking, normally as a result of brainstorming. depicting how their collective ideas relate to one another, and which ideas are more relevant or important for policy planning or evaluation purposes (Trochim. 1989). Typically the process involves the generation of statements or responses from an evaluation focus question, measuring their interrelationship and using multivariate analyses (multidimensional scaling and cluster analysis) to conceptualise the statements in the form of a map. The technique has been used in studies such as planning services for the elderly (Trochim, 1989), an investigation into "caring" within a nursing context (Valentine, 1989). developing a framework for understanding feminism (Linton, 1989) and conducting a needs analysis for mental health services (Wiener et al, 1994).

The application of concept mapping to the evaluation of community relations in Northern Ireland involved the following 3 steps:

First, a group of 30 participants were invited to take part in an exercise which "took stock" of community relations projects thus far, with a view to providing some input into future planning and funding of set-vices in this policy area. Since the range of community relations programmes varied considerably, as described above, a broadly heterogeneous group was easy to identify. Those approached to participate, given the nature of the evaluation were, in the main, active community relations project workers, although many had quite different titles. The aim in the selection of participants, therefore, was to straddle as many of the programme types as possible and target those involved in front-line community relations work. At a group session participants were asked to focus on the following question:

What types of community relations programmes best contribute to improving mutual understanding between Catholics and Protestants ?

This question emerged from discussion with group members about the aims of the policy evaluation and a consensus over what was understood by the question posed. Some debate arose as to whether the word "policies" should be inserted for "programmes" but was rejected as being too ambiguous given other initiatives in this area. After agreement, a brainstorming session facilitated by the researchers generated 116 responses. The response statements were edited to a list of 90 (see appendix 1) by eliminating overlap, duplication and those patently outside the parameters of the question posed.

The second stage of the process involved a different group of 15 people who were asked to sort out individually 90 index cards, each containing a statement generated from stage one, into piles which had some coherence for future planning in community relations. This group had been selected on the basis of their professional involvement in community relations at a policy level and comprised senior civil servants, senior local government officials, representatives from the voluntary sector, policy-specific "experts" and academic researchers in the area. The restrictions imposed on the exercise included having more than one pile in the final sort but less than 90 (the total number of statements) and each card could only be placed in one group. All 15 participants were then asked to rate the importance of each of the 90 response statements for future community relations policy on a Likert-type scale ranging from 1 (unimportant) to 5 (very important). The link between the first and second stage of this process was, in the former, to obtain from policy activists a cumulative view, based on their varied experiences, of what represented effective community relations work and, in the latter, to assess the feasibility of the ideas generated in policy terms. Innovative ideas may come to nothing in planning the future of services if, for example, they are capital-intensive (e.g. building an infra-structure of "neutral" venues). Moreover, the two groups retlected a retrospective look (stage 1) at what had been achieved to date in community relations terms, with a realistic assessment of future planning prospects (stage 2) based upon suggestions of what "had worked" so far.

The third stage involved analyses of the data generated by both groups. The analyses comprised 3 parts as follows:

(a) Each of the 15 sorted piles of index cards was recorded in a binary square symmetric similarity matrix which had 90 rows and columns (same as the number of response statements). The particular sort-combinations for each of the 15 participants were recorded in the similarity matrix as either a "1", if the intersecting items were sorted together. or a "( )" if they were not. The individual matrix entries were then combined to form a group similarity matrix containing all the sort combinations for the 15 respondents. The final table could conceivably have scores ranging from 0, where no participants sorted a combination of the two intersecting items together, through to 15 where all respondents grouped the two statements together, represented by the appropriate row and column cell. In fact the responses tended to be in the lower end of this range. The group similarity index was then used as the data to conduct a two-dimensional, non metric multidimensional scaling analysis (Kruskal and Wish, 1978; Davison, 1983) from which two orthogonal dimensions of the response statements were extracted. This a multivariate technique which produces a map. or in this case a 2 dimensional graph, representing each of the 90 response statements. The proximity or distance between points, each representing a statement, on the graph/map was indicative of how likely the statements were to have been sorted together along 2 dimensions or concepts. Those points furthest apart were sorted less frequently and vice-versa.

(b) The second part of the statistical analyses involved partitioning the multidimensional scaling map into groups or clusters of response statements, still along the 2 extracted dimensions, reflective of similar concepts. This was undertaken by performing a hierarchical cluster analysis (Everitt, 1993) on the values (X & Y) arising from the multidimensional scaling co-ordinates for each of the 90 points on the 2-dimensional concept map. Deciding on the appropriate number of organising clusters required some judgement on the part of the evaluators, since there can be as many clusters as statements. Although information was available to assist the choice (e.g. by examining the dendrogram and agglomeration schedule to assess the point at which the squared Euclidean distance between the statement points sharply increased) it was much more useful to look for some coherence from the combined statements, within limits, which made up each cluster.

The end result of the analyses up to this point was a grouping of the response statements. generated by policy activists and interrelated by policy planners, into similarity clusters which. in turn, were organised along 2 dimensions.

(c) The final and much less complex stage of the analyses involved calculating the average importance ratings, from Likert scales, for each response statement and then the average importance of items comprising each of the selected clusters. From this information the clusters could be ranked or prioritised in terms of their contribution to improving mutual understanding between Catholics and Protestants.

The results

The 2 dimensional concept map showing the six cluster solution is illustrated in figure 1. Making sense of each cluster and, in turn, naming the 2 dimensions required a detailed examination of what each cluster comprised. As an example we look at cluster I here. This cluster contained the following response statements:

- single identity work is the key (statement 63);

- programmes must be underpinned by community development work (statement 84);

- community relations should be a by-product of a programme (statement 73);

- community economic development as a common interest (statement 26);

- evolve from a community development programme (statement 4);

- help catalyze network of community groups (statement 74);

- more research on the community relations - community development model (statement 89);

- community based focus groups to test new community relations ideas (statement 46).

- rooted in communities, assisted by grant-aid (statement 3);

- voluntary groups act as a buffer between community and institutional bodies (statement 8);

- an accessible and non-bureaucratic grant scheme for voluntary groups (statement 17).

This amalgam of response statements seemed to suggest an approach to improving mutual understanding between the 2 communities through combined community development community relations projects. Therein, areas of mutual interest in communities such as economic investment, infrastructure (housing, roads, the environment) and social conditions are recognised as problems which do not have religious boundaries. Working jointly on these common problems improves community relations, as an important by-product of the process, and builds confidence across religious groups.

The same procedure was carried out for all six clusters which allowed each to be named from the combination of response statements contained therein. When all six named clusters are then reconsidered within the concept map (figure 1) it is possible to label the 2 dimensions on the X-Y graph. The 2 dimensions were interpreted as an institutional-based or "top-down" approach to community relations involving the key reconciliation bodies (X-axis) and mutual interest projects, an example of which was described above on cluster 1, as the second dimension (Y-axis).

The ranking of clusters as derived from their average importance scores is depicted in figure 2 and forms the basis for a taxonomy of community relations programmes. The taxonomy differentiates between those programmes which are most effective in improving mutual understanding between Protestants and Catholics to those making least contribution to the programme objective. These are discussed briefly in order of importance derived from the analysis.

A taxonomy of community relations approaches

1. Key reconciliation bodies (cluster 3)

This cluster refers to programmes undertaken by bodies (public, voluntary and independent) set up with a specific community relations brief. Examples include the Community Relations Council, the District Council community relations programme and Corrymeela. The success of programmes under their remit is, in part, attributed to their more stable financial circumstances but also their strategic planning and long-term community relations goals. Lead-in times are long and investments costs high in this policy area which seeks to alter attitudes and behaviour towards "the other" community. Moreover, a core of professional staff, trained and experienced in community relations and conflict resolution work has now emerged in these organisations. This contrasts with the well-meaning and committed volunteer in voluntary organisations, many of whom have inadequate training and resources to implement projects but are nonetheless enthusiastic to "do something". The key reconciliation bodies are also more likely to target groups or geographical areas for community relations work. They can more effectively deploy their resources and are better able to assess the impact of programmes. Again, this contrasts with some voluntary organisations who see Northern Ireland in its entirety as their target group and judge their own success by community relations activity levels, regardless of its relative effectiveness.

2. Publicly funded grant schemes (cluster 4)

This cluster typically includes programmes which are undertaken by a variety of public and voluntary bodies through the assistance of government and European grant-aid (Community Relations Council Inter-Government Grant Scheme; Northern Ireland Voluntary Trust Inter-Community Contact Scheme, European Physical. Social and Environment Programme. The scope of the programmes is enormous. Two points of importance do appear to emerge from the clustered response statements. First, the contribution which this funding has made to infrastructural capital projects is readily acknowledged. Although funds have not be so generous as to allow major investment, the value of purpose-built "neutral venues" or cross-community facilities appears to have made a significant impression on community relations policy activists and officials in their bid to operationalise mutual understanding between the two communities. The second issue with regard to grant schemes is that the flexibility, scope and size of grant-aid available has afforded innovation, experimentation and the dissemination of wider good practice than would otherwise be the case. This is seen as important in a policy area where cause and effect relationships are not, as yet, well established. Notwithstanding some of the difficulties outlined in relation to voluntary bodies, small scale, sometimes one-off grants can be made to voluntary groups on a trial-and-error basis. This practice gained approval from the research respondents as making a contribution to an evolving policy area.

3. Community development - community relations (cluster 1)

This cluster represents projects undertaken by organisations and agencies originally established with a community development brief and incorporating a community relations agenda (Co-operation North, The YMCA and other Christian or church groups are examples). The cross-cutting nature of many community development issues, already referred to, created an obvious foundation for both communities to promote greater mutual understanding. Community development work is also seen as a useful, and in some cases necessary, prerequisite for community relations particularly in single identity communities, so common given the sectarian geography of Northern Ireland. It is important that single identity groups feel comfortable working together on community issues before embracing more difficult cross-community problems. The ranking of this type of approach is best explained by reservations about where community development ends and community relations stan. Some organisations have behaved opportunistically when funds for community relations were available and tried to incorporate a cross-community agenda. The issue, however, is the extent to which these bodies have become genuine in their efforts to pursue community relations as part of their ongoing role or remain no more than token participants. Policy activists and officials, from their responses, seem to acknowledge that this approach is a useful mechanism for the furtherance of community relations goals but the ranking would imply that not all programmes under this heading have made the transition from community development to community relations.

4. Cultural traditions (cluster 2)

This cluster is best described by programmes involving the use of language, history. arts, culture, sports and drama as a means of promoting mutual respect and understanding of diverse cultures. As with most things in Northern Ireland, however, these are not innocuous activities. Protestants and Catholics have their own versions of history, play different sports and espouse two distinct languages. The validity of promoting both traditions in an effort to demystify, foster and sustain cross-cultural awareness within and across communities is unquestionable. The key issue here, however, much like the community development -community relations continuum described above, is how far and how quickly can the debate shift from an emphasis on a "within" community focus to cross-community activities. Crudely described, it is easy to get single identity support for an Irish language group; extending that support to the Protestant community is problematic. Again, the ranking of this cluster would suggest an endorsement of the programme aims in this area but, at the same time, acknowledge the practical problems associated with their delivery.

5. Education, training and personal development (cluster 5)

A variety of programmes appear in this cluster and it is perhaps surprising, given its description, that it should be ranked as low in the taxonomy. This is more a reflection of the types of programme funded under this category and their perceived effectiveness than a rejection of the approach. Some community relations education and training schemes have been seen as too narrowly focused or elitist in their mode of delivery. Such a model of education promotes greater understanding of issues germane to the conflict, taught to postgraduate students, or those interested in the intellectual underpinnings of cross-community issues. Whilst laudable in itself, both can be criticised as having a rather small and select target market. The rationale for this approach is that programmes are aimed at leaders and community activists who are likely to be influential and whose newly acquired knowledge will permeate their work and "trickle down" to community level. Clearly the concept mappers thought that such a link had not, as yet, been well established.

Training and personal development programmes have also received bad press. Again this is less a judgement on the value of training per se, as a commentary on what has been provided by some funded programmes thus far. Training on offer appears to range from programmes which see the practitioner in need of personal development, those which attempt to teach activists reconciliation/conflict resolution skills, through to programmes which aim to improve facilitation and enabling skills of community relations workers. Because the role of the practitioner is an evolving, eclectic and unstandardised one, trainers have felt overwhelmed by the variety of needs, diversity in backgrounds of those now involved in community relations work and the very different organisational contexts within which they work and practise. The low importance ascribed to education, training and personal development programmes is merely a reflection of the unfortunate experiences of those involved in community relations work at the coal-face level.

6. Reactive community relations (cluster 6)

This cluster was characterised by programmes offered through organisations which had been established in response to specific paramilitary atrocities, and in support of a public mood towards peace and reconciliation (The Peace Train Organisation, Enniskillen Together. Community of the Peace People). Whilst it is accepted that much good public-spirited work has and is taking place in these bodies, some of which is dangerous (dealing with paramilitaries), voluntary and unacknowledged, their reactive role, by definition, has, given the cluster ranking, contributed little to long-term community relations goals. They may well be symbolically important, having captured the imagination and support of the public (who in Northern Ireland and beyond has not heard of the Peace People ?) but their capacity to make a sustained and focused contribution is questioned in the research analysis. Programmes offered under this category are provided by organisations in search of a role in a changing post-ceasefire climate, some of which may well become anachronisms. Many of these organisations have acted as a conduit for the revulsion of the public to anti-social or heinous activities of terrorists. Their contribution has been immediate and effective but, nonetheless transient and ephemeral.

Conclusions

The taxonomy outlined provides a framework for future planning and funding within community relations policy based upon the combined views and experiences of those invoked in both policy formulation and implementation to date. At a general level, however, a number of points can be made about the usefulness of concept mapping as an evaluation technique. First, concept mapping is appropriate in circumstances where the policy focus is abstract and devoid of operational parameters. What does the term community relations really mean ? How can we achieve greater mutual understanding between communities ? The fact that this is a relatively new public policy means that few, if any, valid and reliable quantitative indicators of "success" exist. Concept mapping allows for the extraction and utilisation of what is meant by these terms in practice and uses this as the basis for both evaluation and planning.

Second, the inclusion of policy activists in the process of constructing the map provides a degree of street credibility which results from the more abstract experimental designs may lack. Moreover, it creates ownership amongst the respondents whose ideas, after all, constitute the emergent map - the evaluators merely provide the means for structuring and ranking their views.

Finally, evaluations take place in less- than-ideal circumstances. In this case the more difficult, but not the only, problems were a new and ill-defined policy, no obvious performance indicators and multifarious programmes operating ostensibly under the guise of a community relations policy. If one adds to that the fact that most of the programmes had been in operation over different timescales with inconsistent monitoring arrangements, then the evaluator's task of assessing the contribution to improving mutual understanding between the two communities becomes more apparent. Such circumstances are not unique to this evaluation. The less-than-ideal circumstances are the norm and this requires more innovative approaches to policy evaluation, of which concept mapping is an important one. The approach also allows for a combination of other methods. Resources (time, money) permitting this evaluation could have incorporated participant observation of a cross-section of community relations programmes and qualitative interviews with other key actors, not included amongst the participants in designing the concept map. Concept mapping is not without its limitations at various stages of the process, for example, the selection of participants, the focus statement for brainstorming, the choice of the number of clusters and the interpretation of both the clusters and dimensions, can present difficulties. It does, however, represent one method of evaluating public policy where true experimental design conditions, with valid and reliable indicators and a proven causal link between output and outcomes, do not exist.

Central Community Relations Unit (1992), Community Relations in Northern Ireland. Belfast: Policy Planning and Research Unit.

Davison, M. (1983), Multidimensional Scaling. New York: J. Wiley.

Everitt, B. (1993), 3rd ed. Cluster Analysis. London: E. Arnold.

Knox, C. (1994), "Conflict Resolution at the Micro Level: community relations in Northern Ireland", The Journal of Conflict Resolution, Vol. 38, 4, 595-619.

Kruskal, J.B. and M. Wish (1978), Multidimensional Scaling. London: Sage

Linton, R. (1989), "Conceptualizing Feminism: clarifying social science concepts", Evaluation and Program Planning, Vol. 12, 1, 25-29.

Rossi, P.H. and H.E, Freeman (1993), 5th ed. Evaluation: a systematic approach. London; Sage.

Trochim, W.M.K. (1989), "An Introduction to Concept Mapping for Planning and Evaluation", Evaluation and Program Planning, Vol. 12, 1, 1-16.

Valentine, K. (1989), "Contributions to the Theory of Care", Evaluation and Program Planning, Vol. 12, 1, 17-23.

Wiener, R., D. Wiley, T. Huelsman and A. Hilgemann (1994), "Needs Assessment: combining qualitative interviews and concept mapping methodology", Evaluation Review, Vol. 18, 2, 227-240.