Report

The Special Support Programme for Peace and Reconciliation in Northern Ireland and the Border Counties of Ireland is a distinctive application of the European Union Structural Funds.

The Programme arose as a key element of the EU response to the Peace process in Northern Ireland, and was developed after an extensive process of consultation undertaken both by a Task Force of senior Commission officials and by the two Member States involved.

The Programme will operate over the period 1995 - 1999. However, funding has been allocated only for the period 1995 1997. Extension of the Programme beyond 1997 and the level of funding committed over that period will depend in large part on a report to be prepared by the European Commission. An important input into that report will be the Mid Term Review of the Programme, which is to be undertaken in late 1996 and early 1997.

1. Introduction

The purpose of this report is to lay the foundations for the Mid Term Review of the Programme by identifying the key evaluation requirements and the means by which they can best be fulfilled.

This Report brings together the findings of parallel reviews of the Programme in Northern Ireland and in the Border Counties.

This Executive Summary provides a brief summary of the issues considered in the Report and of the conclusions which we reach. The Executive Summary follows the structure of the Report itself.

2. The Overall Impact of the Programme

Section 2 of our Report is concerned with issues relating to the overall impact of the Programme.

2.1 Aim & Objectives

The strategic aim of the Programme is

'to reinforce progress towards a peaceful and stable society by increasing economic development and employment, promoting urban and rural regeneration, developing cross border co-operation, and extending social inclusion.'

The Programme has two strategic objectives

• 'to promote the social inclusion of those at the margins of economic and social life', and
• to exploit the opportunities and address the needs arising from the peace process in order to boost economic growth and stimulate social and economic regeneration'.

2.2 Prior Appraisal

The prior appraisal of the Programme noted that

'The Programme will only be successful in achieving its aim if it not only promotes reconciliation, social inclusion and economic development but is widely seen to be doing so and its strategic purpose [reinforcing progress towards a peaceful and stable society] is widely understood.'

2.3 Expected Impacts of the Programme

The Programme does not indicate the expected form of it impacts or the quantities which might be associated with those impacts.

Section 2 derives the following statements of the desired forms of impacts of the programme from a careful analysis of the Programme document and other documentation

• changes in inter-community values and behaviour
• benefits to socially-excluded groups
• benefits to the economically disadvantaged
• realisation of economic opportunities arising from the Peace process, and
• realisation of cross-border development opportunities.
  

These 'impact statements' are used throughout the Report as the core of the logical structure to the evaluation approach which we develop.

2.4 Baseline Profiles

Appendix 1 provides brief baseline profiles of the following issues

• reconciliation and social inclusion in Northern Ireland
• the economic context in Northern Ireland
• the social and economic context of the Border Counties, and
• the cross border development context

2.5 Distribution of the Benefits & Implementation or Delivery Mechanisms

Other elements of the overall impact of the Programme include

• the actual distribution of the benefits of the Programme, spatially and to a number of defined socially excluded groups, and
• the creation of new implementation or delivery mechanisms.

The evaluation of the Programme will need to identify the degree to which the Programme has, in fact, distributed expenditure to the targeted communities and the benefits which arise from this skewing and from the creation of new delivery mechanisms. Section 2 identifies a number of issues relevant to both aspects of the evaluation.

2.6 Appropriateness of Indicators

Good baseline indicators for the Programme would be

• based on up to date information
• capable of being updated regularly without excessive cost or delay
• changed as a result of Programme actions, and
• not likely to be changed as a result of other factors.
  

Few, if any, macro-indicators meet these criteria. In particular, Census data (including analyses developed from Census data such as the Robson analysis of deprivation in Northern Ireland) does not meet these criteria and, while of value in the targeting of spatial concentrations of deprivations, is not likely to be of use in the monitoring and evaluation of the Programme.

The Department of Finance and Personnel has undertaken a bespoke survey of issues relevant to the evaluation of the Programme. The Omnibus Survey was undertaken prior to the operational launch of the Programme. Replication of the Omnibus Survey in time for the Mid Term Review will be an important element of the evaluation of the Programme. We recommend that a similar survey, suitably adjusted, should also be undertaken in the Border Counties to establish an attitudinal baseline for the Programme in the Border Counties.

However, while the Omnibus Survey data is free from most of the deficiencies of Census data, the detected shifts in attitudes, values and behaviour will be influenced greatly by factors other than the direct impact of the Programme. The Omnibus Survey data cannot, therefore, be seen as a means of directly measuring the impact of the Programme on intercommunity values and behaviour.

In relation to these issues, the Omnibus Survey will still provide important information on changes in attitudes, values and behaviour in the general population in Northern Ireland, against which changes detected in those directly participating in the Programme can be assessed.

The Omnibus Survey will also provide important evaluation information in relation to knowledge and awareness of the Programme and its operation and perception of the Programme's impacts on the various priorities. This is also important information, especially as the importance of the perception of the impact of the Programme was identified in the prior appraisal as an important element of the overall evaluation

2.7 Reference Indicators

Although there are few good baseline indicators at the highest - or macro-level of the Programme, it is useful to record the current position of key indicators in Northern Ireland and in the Border Counties. A selection of such information is provided in Appendix 2.

2.8 Expected Employment Impact

The economic impact of the Programme will be assessed principally by means of a survey of the employment which it creates.

Appendix 3 provides an indication of the potential impact of the Programme on employment in Northern Ireland and in the Border Counties. Using the ratios of employment to expenditure employed in the Northern Ireland Single Programme, it is estimated that some 6080 job years of employment might be generated by the proposed expenditure. The Appendix also provides a number of other estimates of possible employment impacts, at the upper level of the range up to 18,750 job years of employment could be generated if a range of supply side impacts is achieved. Most of the anticipated jobs impact occurs in Northern Ireland, rather than the Border Counties, reflecting the proposed pattern of expenditure.

2.9 Issues Arising from the Sub-programme Review

Arising from the detailed review of the Sub-programmes undertaken in the Northern Ireland and Border Counties studies, Section 2 identifies a number of issues important to the overall evaluation of the Programme, these include

• the diverse nature of the indicators used in the Programme document, even within Sub-programmes
• the value of the clear definition of the socially excluded groups
• the need for a clear definition of the targeted geographical areas of disadvantage. The Programme document includes a wide range of references to spatial targeting, but there is no consistent definition of the areas of concern. This is an important failing which needs to be corrected
• in relation to a number of proposed indicators, the absence of a clear idea of how the data required will be collected
• the absence in almost all cases of a clear idea about the expected scale of impact of the supported activities on the indicators identified, and
• the absence of a clear idea of the current, or baseline, status of most indicators.
  

While the Sub-programme review identifies a number of possible improvements in the indicators in the Programme document, we do not favour ad hoc variation of individual indicators to meet the overall evaluation requirements of the Programme. Such ad hoc variation would not be effective and would tend to focus monitoring and evaluation on the Measure level impacts, rather than meso-level impacts at the level of Sub-programmes and the overall aim and objectives of the Programme. Instead, we propose a series of 'core indicators' which are intended to meet the strategic monitoring and evaluation requirements of the Programme.

3. Proposed Monitoring & Evaluation Framework

Section 3 sets out our proposed approach to the monitoring and evaluation of the Programme. The analysis in the Section distinguishes between the highest- or macro-, intermediate- or meso- and micro-level evaluation requirements. As our focus is on the strategic evaluation of the Programme. our recommendations relate principally to the macro- and meso-level evaluation requirements.

3.1 Macro-Level Monitoring & Evaluation Requirements

As we noted above, the Omnibus Survey is central to the macro-level evaluation of the Programme in Northern Ireland.

Role of the Omnibus Survey

The Omnibus Survey can directly provide impact information in relation to knowledge and awareness of the Programme and perceptions of its impacts. However, the Omnibus Survey can provide only contextual information in relation to the impact of the Programme in relation to social distance and social inclusion. In relation to these factors we propose a 'bottom up approach'. Careful design of the bottom up questioning and of the replication of the Omnibus Survey will facilitate the task of disentangling the effects of the Programme and of the wider, and greater, effect of the general degree (or lack of) progress of the Peace process.

It is important, in our view that a social attitude baseline survey similar to the Omnibus Survey in structure, but adapted to local circumstances, is undertaken in the Border Counties.

Economic Impact of the Programme

The economic impact of the Programme will be assessed primarily by means of a survey of the employment which is created.

The Programme has a number of specifically economic components. Section 3 identifies a number of macro-economic variables which may be influenced by the economic components of the Programme.

Other Macro-Level Requirements

The remaining macro-level monitoring and evaluation requirements of the Programme are as follows

• assessment of the distribution of the benefits of the Programme, in particular the extent to which resources were directed towards targeted communities, both spatially and in terms of defined priority groups
• assessment of the contribution of the Programme to cross border development, and
• review of the efficiency, economy and effectiveness of the new implementation or delivery mechanisms which have been developed for the Programme.
  

3.2 Meso-Level Monitoring & Evaluation Requirements

Given the difficulty of finding suitable macro-level indicators for the monitoring and evaluation of the Programme, effective monitoring and evaluation arrangements at the meso-level are essential.

The Report recommends that a set of 'core indicators' should be put in place related to each of the 5 impact statements for the Programme, which were developed in Section 2.

Section 3 develops a small number of core indicators for the Programme under each of the impact statements. These are not the same as Sub-programme or Measure indicators. The core indicators have been developed deliberately to cut across the Programme structure so that each core indicator can be effected by a number of Sub-programmes.

Section 3 shows how the core indicators can be combined with an analysis of the distributional impacts of the Programme to develop a sophisticated matrix of Programme impacts which will show the actual participation of socially excluded groups, the economically disadvantaged and disadvantaged areas in Programme activities.

3.3 Micro-Level Monitoring & Evaluation Requirements

In relation to micro-level monitoring and evaluation it is important to distinguish between the requirements at Programme level and those at the Measure level. Our concern is with the former set of issues but we recognise that there are a number of issues which will require monitoring and evaluation at the Measure level and which go beyond the issues we discuss.

In relation to projects in Northern Ireland and to cross border projects with a reconciliation aim, we recommend that the Identity Structure Analysis approach which has been applied to the evaluation of the reconciliation aspects of the Physical and Social Environment Sub-programme of the Northern Ireland Single Programme should be applied to the appropriate Sub-programmes of the Programme.

The results of individual Identity Structure Analyses can be aggregated to provide meso-level as well as micro-level data and to assist in the disentangling of the influence of the Programme on the overall movement recorded in the Omnibus Survey.

We recommend that in Northern Ireland the Central Community Relations Unit of the Department of Finance and Personnel should take responsibility for this work and have a co-ordinating role in data gathering and analysis for the Programme.

4. Conclusions

Section 4 provides a brief restatement of the main issues arising and conclusions reached in the Report.

The Programme is complex and is aimed at addressing problems which are also complex and enduring and at realising new opportunities. Inevitably, the evaluation of such a Programme will also be complex.

The proposed monitoring and evaluation system which we recommend is, in our view, the minimum framework necessary for effective monitoring and evaluation of the Programme and we believe that it can be implemented without imposing excessive burdens on Departments, Intermediary Funding Bodies, District Partnerships or Programme participants.