Article Type : Research Article
Authors : Nucera T
Keywords : Ex post evaluation; Railway upgrading; Short and long term effects; Socio-territorial sustainability
The ex post evaluation of the effects of a
regional rail network enhancement project is described through the explanatory
model of the methodology used: the econometric estimation model of measurable
effects is integrated into a qualitative assessment model, based on the process
of reconstructing the theory of change underlying the intervention programme.
The main observable effects in the context are based on the improvement of the
quality of life, the socio-economic growth, the contribution to decarbonisation,
the improvement of social welfare. Besides, the determinants that play the
important role of hinge in the chain of causation of the effects of the project
are then explained, as the followings: relation with the context; selection
process; design of the project; forecasting capabilities; project governance;
managerial capacity. Through the attribution of the contribution weight to the
final performance of the project, to each determinant, it is finally possible
to show and describe the chain of interconnected causations and effects
determining the performance of the project in the medium-long term horizon.
The model of socio-territorial
sustainability adopted is represented by theories of social practice under the
use of the model concurrent factors that are the competences, the materials and
the meaning. The performative dynamics are represented under two points of
view, that of the user and that of the manager of the railway service, where
the social practice of the user is described like "going by train"
while the social practice seen from the manager’s perspective, is described as
"making the train go".
The evaluation highlights the findings of
the applied methodological model, also adopting the model of consistency of the
design performance with the selected criteria of the European Better
Regulation, in case that the investment project is financed by cohesion
policies and programmes.
The paper presents an ex post evaluation model
highlighting the effects of the implementation of a regional rail network
enhancement project, through the explanation of the methodology used: the
econometric estimation model of measurable effects, applied with an ex post
cost-benefit analysis carried out during the functional management phase of the
intervention, integrated into a qualitative assessment model, based on the
process of reconstructing the theory of change underlying the intervention
programme.
Section 2 describes the project design on which the ex
post evaluation is simulated, referring to the speeding up and securing of a
regional rail crossing the rural and internal areas of the Calabrian Apennines
preside, finalized to the upgrading of the actual level of service,
characterized by some critical issues, mainly due to the state of maintenaince
of the infrastructure and the economic and financial sustainability. The
preminent issue is inhancing the accessibility to the rural areas, often
characterized by steep orographic conformations and planoaltimetric tortuosity
of the access roads. The project, accompanied by an ex ante cost-benefit
analysis, has been financed with resources coming from the National Program of
Resumption and Resilience (NPRR), and is actually in progress of realization
for the requalification and speeding up.
Section 3 illustrates the methodological model and the
evaluation approach to the proposed ex post evaluation, based on an integrated
assessment, both quantitative and qualitative, that combines the econometric
analyses (evaluation of the allocative efficiency of resources) with theory
based impact evaluation (TBIE) “White (2009)”, aimed at knowledge production,
through the use of techniques typical of the constructivist approach, to
reconstruct the chain of causation and the mechanisms explanatory to the
project unmeasurable effects.
Section 4 contains the model of socio-territorial
sustainability which the project is inspired by, correlated to theories of
social practice whose principles and inspiring criteria are placed in the field
of social science studies “Giddens (1984)”. The evolutionary change is represented
by the comparison of the two pre and post intervention configurations,
represented by the concurrent factors connotative of the model used that are
the competences, the materials and the meaning. The performative dynamics are
represented under two points of view, that of the user and that of the manager
of the railway service, where the social practice of the user is described like
"to go in train" while the social practice seen from the manager
perspective, is described as "making the train go". The analysis
highlights some issues emerging from the process of adaptation to change by the
the different categories of stakeholders involved, leading to the need of
promoting territorial policies aimed at social welfare and inclusion, with the
aim of supporting different target of users and enhancing more accurate project
designs.
The Conclusions represent the reliability of the model
adopted, with the integrated approach, by the use of econometric analyses
combined with quality assessment exercise carried out directly on the field, in
applying of the constructivist approach. The applied model, at last, estimates
the weight of each relevant determinant of the project performance, showing the
causal chain and the final long term impacts.
Territorial and
socio-economic context
The territorial area in which the project is located
is situated in the central-northern part of the Calabria Region, on the slopes
of the Sila plateau. The following figure (Figure 1) represents the territorial
area of influence of the railway line. In particular, the municipalities
crossed by the railway with at least one station and/ or stop are twenty-five,
of which eighteen belonging to the province of Cosenza and seven belonging to
the province of Catanzaro, including the municipalities of Catanzaro and
Cosenza. The line in operation, about 98.7 km long, narrow gauge, connects the
two provincial capitals through the municipalities of the inner mountain area,
reaching altitudes that exceed 800 meters above sea level. The service offer,
on the typical school working day, returns 18 pairs of daily rides on the route
Rogliano - Cosenza with an average journey time of about 40 minutes, and 15
pairs of daily rides, on the route Soveria Mannelli - Catanzaro, with an
average journey time of about 60 minutes.
In order to return the most updated picture of the
socio-economic conditions of the territorial area in exam, reference can be
made to a significant sample of municipalities lying on the area of influence
of the railway, most of which have a station/stop on the same line (ten out of
fourteen), belonging to the national strategy for inner areas (SNAI), among the
four SNAI areas selected in Calabria in the 2014-2020 programming cycle, in
particular in the area called Area Reventino-Savuto (Figure 2).
The Strategy of the Area Reventino-Savuto, approved in
2018, includes the statistical data provided by the Open kit of the National
Agency for Cohesion, updated to 2016-2017, and provides some knowledge of the
territorial and socio-economic context. The Area consists of fourteen
municipalities, divided equally in the two provinces of Cosenza and Catanzaro,
located in a largely mountainous territory, with an average altitude of 752 m
between 540 and 937 m. The inner Area Reventino-Savuto, on 1 January 2016, can
count on 21.535 inhabitants, which make up 2.2% of the population of the inner
areas of Calabria and about 1.1% of total residents in the Calabria Region. The
area is characterized by significant ageing of the population: the population
over 65, 23.3% in 2011 increased to 25% in 2017, surpassing both the regional
and national average of inner areas, respectively 21.6% and 23%, at the time.
The critical issues in the provision of basic
services, for example in education and training, are mainly related to the presence
of a high fragmentation of school complexes, not justified by the internal
distances of the territory. On the other hand, the steady and progressive
demographic decline shows that primary and secondary education services are
sparsely populated, with an average number of pupils far below the regional
average, for example for primary school 41 pupils compared with the average
value of 86 in the regional inner areas.
The consequence of the scarcity of pupils is the
incorporation of the same in multi-classes that represent as much as 25.4% of
the primary school classes in the Area. Despite the situation of marginality
and criticality of the offer for all basic services, a production system is
present with 1720 companies (data Infocamere 2016), although not all in
production, of which 46% in the tertiary sector, 25% in agriculture, and 24.1%
in industry. The manufacturing sector is the main one, in particular, in the
wood-furniture sector, with a concentration and tradition of production of
wooden chairs, especially in the municipality of Serrastretta; in publishing,
with the presence of a publishing house of national fame (Rubettino), located
in the municipality of Soveria Mannelli; in textile production, with an
artisanal wool mill that reinterprets in a modern way yarns and designs, then
raw materials and design.
The redevelopment project, with the speeding up and safing of the railway route, as well as improving the accessibility of places especially for the use by commuters of systematic mobility, will be getting relevant impacts, constituting not only a garrison of social coexistence and environmental sustainability, but also a vector of enhancement of the landscape and cultural resources of the territory crossed and of their touristic potential.
Project objectives
The general objective of the project is to raise the level of safety and speed up the line in operation through the following main interventions:
The general objectives are aimed at achieving
appreciable savings in the timing of inter-municipal travel and to the capital
urban area, main provider of services, making the rail service more attractive
and predicting the potential diversion of mobility flows from individual to
collective transport, with the potential achievement of the environmental
impacts due to the consequent reduction of air pollution from road transport.
Other specific objectives achievable relate to increasing safety levels and the
quality of rail transport services, improving the operating management with the
reduction of the service supply and infrastructure maintenance costs, reducing
the enviromental pollution with the purchase of new sustainable rolling stock.
The following figure represents the interurban
infrastructural layout, with the planned design variants between the inner
areas and the urban area capital of region (Figure 3).
In particular, the project involves the construction
of three off-site variants, which will allow the shortening of the route of
about 5100 m (5.1 km), with the simultaneous suppression of three stations in
the inner localities, whose services to current and potential users, are
expected to be integrated with access/egress shuttles to stations closer to
territorial areas involved. The localized variants on the original track
consist in the adjustments of the curves with reduced geometry to allow easier
curvatures and the construction of some viaducts and/or natural tunnels for
crossing the embankments.
The variations made to the route can allow the
improvement in the current journey times of about 10 minutes, contributing to
the overall efficiency of the line together with the other contextual project
interventions.
The intervention has been subjected to an ex ante
cost-benefit analysis, carried out on the project alternatives whose results
show, with the use of specific financial and economic parametric indices (FNPV,
FIRR, ENPV, EIRR) the utility and economic-social convenience of the selected
project compared to the design alternatives. In particular, the intervention
cost, using the updated parametric costs related to the railway works (material
and immaterials) to be built, is estimated 210 million euros about, including
the sums available to the administration.
Assessment drawing
The methodology developed for the ex post evaluation
uses an integrated approach, quantitative and qualitative based, through the
application of the econometric evaluation of an ex post cost-benefit analysis
(CBA) on measurable effects, and the impact assessment based on theory, through
the reconstruction of causation chain on unmeasurable effects; the evaluation
takes into account the complexity of the large investment project and the
medium-long-term time horizon that characterizes its exercise and effects in
the context.
In particular, the Cost-Benefit analysis applies an
appropriate analytical tool for impact assessment because it can provide
quantification and monetization of some of the short and long-term effects
produced by the project, at the same time providing a framework to identify the
crucial aspects of its results and performance. The evaluation based on theory
is instead characterized by the observation of each phase of implementation of
the logic of intervention, aimed at identifying causal relationships and
mechanisms of change, for the identification of the how and why the
intervention works, focusing on the understanding of the determinants and
causation chains of the process leading to the appreciating of the difficult
monetization effects. It is placed within the framework of the Theory of Change
which concerns the opening of the so-called “black box” in the search for an
understanding of the generative mechanisms of change and the reconstruction of
the underlying theory by the evaluator “Weiss (1995)”. These approaches fit
into the framework of generative causation “Stern et al. (2012)” [1], which
bases inference on mechanisms that explain the effects, and identifies the
application approaches in theory-based evaluation (TBE) and realistic evaluation
“Pawson and Tilley (1997)” [2,3].
The tools and the main qualitative techniques of investigation, used in application of the impact assessment based on theory, consist in conducting interviews with stakeholders, focus groups, surveys on samples of commuters, research on institutional authorities and service providers archives, search for press articles, suitably combined and interpreted according to the type of contribution to be made to the evaluation process. The questions that guide the evaluation process are articulated below:
What change can be
observed as a result of project implementation?
To what extent can the
changes observed be attributed to the project?
Are there any unexpected
effects?
What mechanisms explain
the impacts? What are the key factors of the causal chain that explain the
mechanisms?
What kind of short and
long-term effects can be identified?
What is the minimum time needed for a long-term effect to manifest and stabilize?
The purpose of the ex post evaluation is to assess the
degree of effectiveness, efficiency and relevance of the results achieved
compared to those expected, using all the available resources, taking into
account also the unexpected and unplanned effects, to understand the internal
and external dynamics of the program [4].
The evaluation analysis develops substantially through the following three dimensions:
The "WHAT": this dimension refers to the types of long-term effects that can be observed following the implementation of the project. The classification of effects can be represented by the following categories:
The "WHEN":
this dimension refers to the point in the life cycle of the project where the
effects materialise for the first time (short-term horizon) and stabilise
(long-term effects). It discusses the moment of the post-intervention time
cycle in which the evaluation is carried out and the role it can play in
relation to the implementation of the project.
The "HOW": this dimension implies the development of a reasoning on the factors, both internal and external to the project, that have determined the observed causal chain of effects and have influenced the performance of the observed project. For this purpose it is possible to identify six stylised determinants of project results: relation to context; selection process; project design; forecasting capacity; project governance; managerial capacity. The interrelations of these determinants and their influence on the effects of the project is crucial to understand the final performance.
Methodological model and
evaluation approach
The methodology adopted consists of four main phases:
The first step is mapping the effects. In principle,
transport infrastructures can achieve a variety of short- and long-term
effects, which must be appropriately identified together with the investigation
of temporal dynamics and causal chains. This phase responds to the dimension of
analysis of the "what" and also observes the "when", the
temporal dimension of the effects.
The second phase aims to measure the relevant effects.
This is done through the combination of econometric analysis (quantitative)
supported by retrospective CBA and qualitative assessment. In particular,
effects that are not quantifiable through monetary assessments will be
qualified and evaluated through qualitative analyses resulting from the
application of the specific methodologies adopted.
The third phase concerns the understanding of the
effects or better the dimension of the "how", aiming to reconstruct
the logical framework of the intervention by identifying the elements
responsible for the causal chain of the effects detected. The final evaluation
integrates qualitative and quantitative evidence to focuses the most relevant
effects derived from the implementation of the project, highlighting the
project impacts. The figure (Figure 4) represents the schematic conceptual
framework of the adopted methodology.
The description of the four methodological phases is shown below:
The
first phase, concerning the mapping of long-term effects, identifies the
potential impacts on social welfare by public investment projects on transport
systems, and, for the project being evaluated, the following have been selected
[5]:
Effects on economic growth: the first category of effects concerns economic growth, which is declined through productive effects. In the specific case of the transport project, most of the effects are related to the reductions in production costs, the increase in accessibility and attractiveness of the territory in which the transport project is implemented. There are several groups of actors on which the economic effects can impact (users, producers, managers/promoters of the infrastructure project). Direct indicators that can measure effects on economic growth are:
Effects on quality of life and social well-being: in line with the theories of “Dasgupta (2001)” and “Stiglitz et al. (2009)”, the concepts of quality of life and social well-being refer to factors that affect social development and satisfaction as well as the perception of users and society in general [6,7]. The main direct effects on the quality of life, in this project case, are:
quality
of service (speeding up, comfortable seats, wifi, etc.);
safety
prevention (safety);
security
protection (security);
noise.
Effects on environmental sustainability: the concept of environmental sustainability refers to the possibility of ensuring the needs of the current generation without compromising the environmental conditions of future generations. Transport infrastructures can influence environmental sustainability in two opposite ways: on the one hand, the expansion of transport infrastructure is negatively linked to environmental sustainability as it could generate greater environmental pollution and reduction of natural resources, and, on the other, more efficient and technologically modernised solutions, provided by improved infrastructure, can help to achieve environmental pollution reduction targets. The main effects of the transport project are related to the contribution to the mitigation of the following factors:
Effects on distributional impacts: distributive impacts can relate to social cohesion and territorial cohesion. The lowering transport costs can mitigate social inequality of the weakest social groups, while the improving of accessibility can inhance the territorial cohesion of the peripheral areas, contributing to the bridging of territorial accessibility gaps.
Measurement of the project effects
With regard to the measurement of effects, the cost benefit analysis (CBA) is the methodological approach adopted, most desirable from the point of view of “quantitative” analysis, for the following reasons:
is the most desirable
quantitative method to investigate the detail elements required to isolate the
impact of a single project;
is a reliable tool for
presenting project benefits and externalities in monetary terms;
measures the impacts in terms of welfare changes, the CBA being based on the welfare economy. This makes it possible not only to rank projects, but also to formulate conclusions about their social desirability.
The methodological approach is completed with the
qualitative evaluation of those effects that cannot be measured with the CBA,
enriching the evaluative research with the consideration of the variety of
long-term productive effects of the contribution to the impact on the social
welfare.
The methodology used for the application of the CBA is
set out in the Community Guide adopted in respect of investment projects
financed by the European Cohesion Policies , assuming that the projects
selected for ex post evaluations have been functional for at least five years,
with the implications of reducing any bias by an optimistic approach, given the
assessability of actual observed data, without making a comparison between the
two CBA ex ante and ex post, rather analyzing the contributions of the project
with respect to medium-long term impacts.
General
principles in methodology application
The ex post evaluation is carried out five years after
the completion of the works of renewing and modernizing rail infrastructure, to
assess its medium-long-term impacts, therefore, adding the average period of
completion of the work to the first five years of operation, will be quantified
analytically the historical cash flows that the project has accumulated,
including the actual project implementation and operating costs, and revenues
recorded during the first five years of operation. In the remaining years,
included in the time window following the current year, within the time horizon
(30 years), instead, operating cash flows forecast data will be used relative
to the management phase already started, that is the costs of the service, the
costs of the infrastructural management in relation to usury, the railway
service revenues. The approach used for the ex post CBA is that of the
differential analysis between the design scenario and the non-intervention
scenario and other possible alternative design scenarios; in particular the
non-intervention scenario is the counterfactual scenario without the
intervention on the railway line, where the cash flows of the railway
management are considered retrospectively and prospectively, which, in addition
to tariff revenues, include maintenance costs, functional renewal (Business As
Usual), or even planned investments in extraordinary maintenance and
operational improvement, for the functional operation of the service (Scenario
Do minimum). The counterfactual scenario is characterised by significant costs
to be incurred in terms of extraordinary maintenance and hydrogeological risk
mitigation, higher than those that occur in the intervention scenario
characterized by improvements in infrastructure and increased safety and
resilience to environmental risks.
The implementation of the railway requalification
project involves the reduction of operating costs compared to the
non-intervention scenario, both for the infrastructure operating costs and for
the service management costs, also the number of the human resources employed
is reduced for effect of the increase of the service level of renewed
infrastructure; the modernization of the transport system also allows a
significant reduction of costs of extraordinary maintenance. Based on these
circumstances, the savings obtained in the differential operating costs
translate into an increase in differential revenues in the calculation of the
financial and economic performance of the project. The increase of the tariff revenues
derives also from the forecasted increment of railway transport demand
consequent to the upgrading intervention.
The main benefits generated by the implementation of the project are the following:
If the sum of the positive and negative components
results in a favourable final balance, measured in the amount of avoided
externalities in millions of euro/year, this translates into net benefits for
the community. The process of discounting cost and benefit items leads to the
definition of the economic net present value (ENPV), the economic internal rate
of return (EIRR) and the benefit/cost ratio. The results show that the
intervention scenario is characterised by a positive socio-economic feasibility
assessment if the economic indices are characterised by the positive value of
ENPV, the value of the TRIE higher than the social discount rate used for the
prospective analysis, and the benefit/cost ratio higher than one.
The objectives pursued by the qualitative analysis are identified in the following:
describe the project with
a critical focus on its identification;
gather evidence on the non-quantifiable effects and factors influencing design performance.
The following qualitative evaluation focuses on the effects produced by the realization of the railway project, not measured by the cost-benefit analysis, and any unexpected effects. The approach, unlike quantitative methods, produces an explanatory narrative of how things should logically function to produce the desired change. A precondition to the realization of the process of reconstruction of the theory of change is that the evaluator works collaboratively and as a facilitator, among a wide range of stakeholders, in a typically constructivist approach. The evaluator, once the reconstructed “theory of change” ensures that the intervention logic is acceptable to stakeholders, appropriates the map of the program resulting from the stakeholder involvement process and, using the most appropriate data collection techniques, monitors and analyses the progress of the intervention programme and the deployment of its short-medium term effects, under the integration of the evaluation evidence.
Understanding of the project effects
Once the effects of the project have been identified
and measured, and the causation chain of the different categories of short and
long-term effects has been investigated, the next step of the methodological
approach involves the understanding of the elements and mechanisms, both
internal and external to the project, that have determined the succession of
the causes and effects determining the performance of the project. The
interactions of these determinants may reinforce or mitigate one effect over
another. In addition, each determinant can contribute, whether positively or
negatively, to generating, accelerating or curbing certain short or long-term
effects (Table 1).
The final evaluation consists in the integration of the
findings of the analyses carried out, both quantitative and qualitative,
applying a qualitative grading to focus the contribution of the most relevant
effects derived from the implementation of the project, giving evidence to the
expression and description of the impacts, both negative and positive, of the
medium and long term. The evaluation, for projects financed by the Structural
Funds, shall complement the analysis with the application of the criteria
indicated by the European regulation (Better Regulation 2015) with regard to
relevance, coherence, effectiveness, efficiency, added value of the EU
Structural Funds of the project under ex post evaluation, establishing
consistency reports.
Following the measurement of the levels of contribution, for each medium-long term effect (qualitative and quantitative) on design performance, a ranking of all effects with the associated weight, will be carried out, within the four macro-categories already described (economic growth, quality of life and wellbeing, envinromental sustainability, distributive impacts) (Table 2).
Understanding
of the effects and final assessment: key factors
The final step of the methodological approach involves
the understanding of the key factors and mechanisms, both internal and external
to the project, that have determined the succession of the causes and effects
that lead to the final performance of the project. The final phase of the
pathway goes on to show and describe the chain of interconnected causations and
determinant effects of the performance of implemented project.
The behavioural model of the project being evaluated
that leads to the final performance assessment, is represented in the figure
(Figure 5).
The socio-territorial
sustainability of the project
The model of socio-territorial sustainability to which
the intervention is inspired is represented by theories of social practise
whose guiding principles and criteria lie in the field of social science
studies and intersect with the field of study of environmental sociology.
The basic field of study in the social sciences, consistent with the theory of structuring, is neither the experience of the individual, nor the existence of social totality, but the social practices distributed in space and time, “Giddens” (1984). "The practice is a routinized type of behavior, a path that can be populated by a multitude of single and often unique actions", “Reckwitz” (2002). The most recent studies on the dynamics of social practice (“Shove et al.” (2012)) state that the trajectories of sociotechnical change are produced by changes that occur within configurations that connect the symbolic-cultural dimension (images), the know-how (skills) and technologies (the material dimension). A significant quality of social practice is that it is exercised in a collective way by people who are actively involved in combining the constituent elements of it. The constituent elements are defined below:
The analysis of the dynamics of social practice contains two simple assumptions: the first is that social practices consist of elements that integrate when practices are brought into being; the second is that practices are generated, stabilise and disappear when connections between the constituent elements form and consolidate or when they break (Figure 6).
The system of collective mobility represents a dynamic social practice that can be configured within the framework of social practices and their characterizing components. The following figure (Figure 7) represents the graphic scheme of practices, respectively adopted from the point of view of rail users and service managers, in pre and post intervention.The evolution of the exercise of the two social practices leads to the potential strengthening of the preexisting connections between the connotative elements of the social practice, in a dynamic process supported by the insertion into the territorial integrated iron-rubber mobility system of accompanied material and immaterial strengthening actions. The modernization of the technological information and communication system and the eco-sustainable renewal of the rolling stock which, in the potential production of an increase in the level of service offered to improve the accessibility of the territorial system and social and environmental sustainability, performatively regenerate and recreate the management practices and use of the renewed collective transport mode.
Given the remoteness of the inner areas crossed by
rail service, it becomes essential to guarantee the usability, with reference
to the railway service but also to other flexible and shared forms of local
public transport that could be activated in these places, for example with
on-call booking or with other forms of community participation. The attention
of policies and planners should be directed towards adequately achieving social
inclusion and welfare, for example already foreseeing in the design of
intervention the appropriate solutions that can be translated into support
actions to knowledge of traditional services and innovative ones, planning
assistance for their way of use, which can be shared in real time (ex.
telephone technical assistance, telephone and digital contacts exposed to
information installations, digital portals with facilitated access, etc.) and
such as to favour the weakest social groups, guaranteeing their inclusion in
local welfare system.
The methodology proposed upon the impact assessment of
the project case for strengthening the regional railway service demonstrates
the validity of the model adopted, through the combination of both quantitative
and qualitative analyses, to evaluate the medium-long term effects produced in
the post-implementation phase, the modernized public service being fully
operational. The model applied constitutes an evaluation practice to support
the ex post assessment of investment programs in the transport sector, where
the application of cost-benefit analyzes is more appropriate to infrastructure
projects according to the econometric approaches in use, as integrated by
qualitative analyzes with the active participation of stakeholders according to
the constructivist approach of interpreting the key factors determining the
succession of causes and effects of the project performance and its final
impacts.
The relationship with the socio-territorial
sustainability of the project model also appears to be essential, around which
revolves the activation of the performative dynamics characterizing the renewed
social practices moving towards adaptation to the new skills required by the
project technological and digital modernization. In this sense, the evaluation
reflection leads to the need to promote territorial policies aimed at welfare
and social innovation in the direction of the adequate planning of support
actions for the weakest segments of the population, integrated with more
accurate project designs, to allow equal opportunities for use of the renewed
public services, from a participatory and shared perspective.