final evaluation of leonardo da vinci i programme · leonardo da vinci 1 programme coincided with a...

Disclaimer

The views and comments expressed in this text are the responsibility of Deloitte & Touche and do not necessarily reflect the opinion of the European Commission.

Acknowledgements

The production of a report of this scope would not have been possible without the effort, contribution and support of a large number of organisations, experts and individuals.

This report was written by Isabelle Collins, Richard Doherty, Christopher Allinson, Sarah Batterbury, and Wouter Van Den Berghe. Responsibility for its content is solely with the authors.

The authors wish to acknowledge the commitment and professionalism of the expert team who provided area-based reports, case studies, and valuable inputs into the final report.

♦ Hanne Shapiro ♦ Paddy Carpenter ♦ Lionel Chaty ♦ Peter Debreczeni ♦ Pedro das Neves ♦ Prof Theo Papatheodossiou ♦ Anna Gammaldi ♦ Gabriele Marchl ♦ Anne Marie De Jonghe

The authors also acknowledge the contribution of Nathalie Belnet (Project Assistant) during the early stages of the evaluation.

Valuable insights and direction were provided by representatives of the European Commission - DG Education & Culture and DG Employment & Social Affairs, the Expert Group on the External Evaluation, the Committee of the Leonardo da Vinci Programme, Member State Officials, National Co-ordination Units, and CEDEFOP. The authors would like to thank these individuals for giving freely of their time and making the evaluation possible.

The production of this report would also not have been possible without the effort of the many anonymous respondents from the Pilot projects, Mobility projects and Exchange participants themselves. The authors would like to thank all of the many individuals across Europe who responded to questionnaires and gave up their time to contribute in this way to the evaluation.

Deloitte & Touche

Final Evaluation of Leonardo da Vinci 2

1 Executive Summary ...............................................................................................4

1.1 Opening comments ............................................................................................4

1.2 Background and introduction.............................................................................4

1.3 Policy Context....................................................................................................5

1.4 Complementarity................................................................................................6

1.5 Implementation ..................................................................................................6

1.6 Outputs...............................................................................................................7

1.7 Impact ................................................................................................................7

1.8 Conclusions and recommendations....................................................................8

1.9 Overall Considerations.....................................................................................12

2 Background and Introduction ..............................................................................13

2.1 Genesis of Leonardo da Vinci 1.......................................................................13

2.2 Scope of this evaluation ...................................................................................13

2.3 The consultant ..................................................................................................15

2.4 Methodological Approach ...............................................................................15

3 Vocational Education & Training Policy Context of Leonardo da Vinci I .........24

3.1 Introduction......................................................................................................24

3.2 The origins of the policy framework ...............................................................25

3.3 Chronology of vocational training and relevant education action programmes 1986-1993 ................................................................................................................26

3.4 Clarification of the legal aspects of vocational training policy .......................27

3.5 Vocational training - EU fora ..........................................................................28

3.6 Other Policy Influences....................................................................................28

3.7 Other EU Policy Influences .............................................................................30

3.8 Launch of Leonardo da Vinci ..........................................................................31

4 Complementarity with other EU instruments ......................................................35

4.1 Relationship with previous programmes .........................................................35

4.2 Relationship with other Community interventions ..........................................36

4.3 Duplication or synergy/value added ................................................................43

4.4 Take-up of lessons by direct funding programmes..........................................45

5 Implementation ....................................................................................................46

5.1 Structures and Procedures................................................................................46

5.2 Project selection and management...................................................................52

5.3 Problem Areas..................................................................................................58

Deloitte & Touche


5.4 Costs, benefits and value for money ................................................................61

5.5 The Interim Evaluation ....................................................................................62

6 Outputs.................................................................................................................63

6.1 What were the outputs .....................................................................................63

6.2 Outputs recorded centrally...............................................................................63

Outputs recorded at national level ...........................................................................72

6.4 Quality of outputs ............................................................................................73

6.5 Products - Dissemination .................................................................................73

7 Impact ..................................................................................................................78

7.1 Approach..........................................................................................................78

7.2 Impact of Leonardo at the EU and national levels...........................................79

7.3 Impact of Pilot Projects....................................................................................79

7.4 Impact of the Mobility Projects .......................................................................85

7.5 Impact for the Participants ...............................................................................86

7.6 Impact in Central and Eastern Europe .............................................................88

7.7 Impact in respect of Equal Opportunities ........................................................90

7.8 Impact of specialist strands..............................................................................91

7.9 Impact with respect to the Social Partners .......................................................92

7.10 Impact with respect to sectors......................................................................93

8 Conclusions and Recommendations ....................................................................96

8.1 The Programme’s objectives revisited.............................................................96

8.2 Types of action.................................................................................................97

8.3 Complementarity..............................................................................................99

8.4 Implementation ..............................................................................................100

8.5 Outputs and Impact ........................................................................................104

8.6 A summary.....................................................................................................107

8.7 Overall............................................................................................................107

8.8 Specific Objectives of the evaluation ............................................................108

Deloitte & Touche


1 Executive Summary

1.1 Opening comments This evaluation was carried out to assess whether the Leonardo da Vinci Programme contributed effectively to the achievement of its objectives, chiefly development of EU vocational education and training policy. In particular the evaluation aims to contribute to the final report on the programme, which the Commission will present to the European Parliament, the Council, the Economic and Social Committee, and the Advisory Committee on Vocational Training later in 2000.

The evaluation is based on the national reports from the Member States, area–based expert reports; published material from the Commission and from CEDEFOP; and internal Commission documents. The report is also based on interviews with key stakeholders and questionnaires sent to pilot and mobility projects and exchange and placement participants. It therefore represents a process of extensive consultation and has been shaped and guided by individuals in the expert group, the Leonardo Committee and our clients in DG Education and Culture.

This document presents an evaluation of the programme’s overall implementation and results, but is not intended to represent a financial audit of the workings of the programme.

This Executive Summary contains: 1. Background and introduction to the evaluation

2. An overview of the policy context of Leonardo da Vinci

3. An overview of the complementarity of Leonardo with other EU Instruments

4. An analysis of the key implementation issues which shaped the impact and delivery of the Leonardo programme

5. A discussion of the key outputs of the programme

6. An analysis of the impact of Leonardo – with particular emphasis being directed to experiences of the programme for participants and the impact for social partners, SMEs and in Eastern and central Europe and at sectoral level.

7. Conclusions and recommendations This format is also followed in the main report.

1.2 Background and introduction The Leonardo da Vinci programme was adopted for 5 years in December 1994 and had a key objective in supporting the development of policies and innovation action in the Member States by promoting projects in the context of trans-national partnerships which involve different organisations with an interest in training. Leonardo had a budget of 620 million ECU and was open to the 15 Member States and countries from the rest of the European Economic Area (Iceland, Liechtenstein and Norway), plus Cyprus, the Czech Republic, Estonia, Hungary, Lithuania, Latvia, Romania, Poland, Bulgaria, Slovenia, and the Slovak Republic. Leonardo da Vinci 1 was established with 19 different objectives in 1995. The programme's objectives were very broadly drawn and the result was a programme that

Deloitte & Touche


reflected the wide differences in training policy, systems, priorities and resource levels that characterise the European training scene. This is even more marked in the case of the pre-accession countries. Accordingly, drawing conclusions in the form of a European synthesis presents difficulties, and some of the report's detailed discussions are more relevant to individual, or a small group of, participant countries.

The specific objectives of the evaluation were to evaluate the Leonardo da Vinci 1 programme and provide advice and counselling to the European Commission on five key issues:

• the outputs and impacts (results and outcomes) of Leonardo da Vinci 1

• the effectiveness of the programme in terms of achievement of its own general, specific and operational objectives;

• the efficiency of Leonardo da Vinci 1 in terms of management and delivery structure;

• Mechanisms for the improvement of the effectiveness and efficiency of Leonardo da Vinci 1 in future programmes;

• Mechanisms for the development of complementarity and synergy with other vocational education and training instruments and policies both at the EU and Member State levels.

1.3 Policy Context The Leonardo programme has been formulated in a dynamic policy environment, consisting of a variety of actions with potential for synergy. The life span of the Leonardo da Vinci 1 programme coincided with a period of evolution and maturation of the policies and principles in respect of lifelong learning and its relationship with the maintenance and growth of employment in the EU. The policy context existing at the launch of Leonardo da Vinci 1 in 1995 was rather less focused than it was by 1999.

In the run up to the elaboration of Leonardo a debate emerged about the legal base for European-level vocational training policy. This was resolved in the Maastricht Treaty in 1992. A Community vocational training policy was introduced, explicitly designed to support and supplement the vocational training activities of the Member States. There was, however, no provision for harmonisation at the EU level.

Alongside the development of the legal policy framework concerning vocational training, a number of European level fora concerned with vocational training policy and practice have been operating largely independently of each other. The profile of vocational training in the EU policy arena was also raised by white papers, activities such as the European year of Lifelong Learning and of course the broader developments in employment and Information Society policy.

Leonardo was launched in 1995 in this context. It was developed from the rationalisation of a number of former action programmes: COMETT, PETRA, LINGUA, EUROTECNET, FORCE, and HELIOS. The 19 different objectives with which Leonardo da Vinci 1 began were a cumulative listing of aspirations and principles inherited from this group of preceding programmes. This reflected the absence of a well-planned and thoughtful approach to a complementary European-

Deloitte & Touche


level contribution to policy-making at a time of change, and the lack of a clear hierarchy of objectives in Leonardo made implementation problematic.

1.4 Complementarity At a conceptual level the Leonardo programme was complementary to other EU programmes but this was almost as a result of accident rather than careful planning. The different EU programmes tend to operate in parallel rather than in an integrated fashion. Leonardo’s distinctiveness vis-à-vis other European programmes can be found in its provision of opportunities for active testing of innovative ideas through transnational projects, and it making possible of large-scale mobility programmes for trainees. The achievement of complementarity between EU interventions could be enhanced if they were conceived as part of a continuum, or a hierarchy of interventions, which are complementary to one another in the policy process. Such an approach would enhance the value of each programme’s contribution by placing it in an integrated context. It would also illuminate the potential for cross–programme knowledge sharing and dissemination of good practice. Continuity between programmes would entail the establishment and utilisation of information channels between programmes and related policies. This is essential for optimising knowledge management strategies within the overall vocational educational and training area.

1.5 Implementation The Leonardo programme is extremely complex in its formulation and in its implementation. Many of these complexities are embedded in the Council Decision establishing the Programme, while others are related to the management systems of the European Commission. A large proportion of the administrative work was externalised to a Technical Assistance Office in Brussels and to National Co-ordination Units in the participating countries. During the course of the programme there were a number of events which culminated in the closure of the TAO and the transfer of its functions to a unit within the Commission - the Cellule Leonardo (CLEO). The underlying principles and processes did not change fundamentally with the change from the TAO to CLEO.

The complexity of the programme’s structure caused problems for potential promoters in identifying the appropriate procedures and strands for their projects. Documentation was also hard to follow and not user-friendly. The requirement for projects to form partnerships may have biased project selection in favour of projects with pre-existing networks. The financial circuits for centrally administered projects were immensely long. This clearly discriminated against SMEs and non-profit organisations. The co-funding regime was also extremely onerous. Serious questions arose about the capacity of the Commission to process the number of payments involved within a reasonable time-scale.

Project selection took a period of almost one year between the publication of the call and notification of the successful projects. The programme was not able to respond to changes which took place for the projects in this period. Projects also complain about the lack of feedback received during and after the process. Backdating contracts was

Deloitte & Touche


also not conducive to project implementation. In addition, no mechanisms were established to take forward the lessons from the programme at European level. There was no overall strategy to use the outputs of the programme.

1.6 Outputs This evaluation uses the term outputs to describe the activities that took place in Leonardo, and the products associated with the pilot projects. The emphasis on detailed quantitative reporting on activities at the European level appears to have resulted in downplaying systematic appraisal of the quality of the Leonardo processes, of the personal and experiences, or of the products. Our research has indicated that there was significant satisfaction with the opportunities within the programme for establishing organisational, networks and European information exchange and with the personal development opportunities for the participants.

Leonardo suffered major difficulties with the collection and dissemination of outputs, however. The Commission’s ambitions in this area have been frustrated by the programme’s emphasis on managerialism and unnecessary bureaucracy and by the collapse of the TAO.

Leonardo therefore lacked a Europe-wide approach to product dissemination, a catalogue of products, the provision of good practice material and guidance, marketing events, and opportunities to build on the potential links with the European Employment Strategy.

1.7 Impact The impact of Leonardo extends beyond what can be quantified and measured. It is equally conditioned by the receptivity of the implementation environment as much as by the design and quality of the programme. Impact is also heavily contextually determined and reflects the complex interaction between programme setting and design in specific places and contexts.

The projects were largely satisfied that their expectations had been met, although there was widespread dissatisfaction with the delays. In many instances this led to comments that projects would not be willing to participate in Leonardo in the future.

Trans-nationality and partnerships were welcomed but there is evidence that the learning curve for the operation of partnerships was fairly steep for the projects. Evidence of organisational learning may be grouped around:

• The acquisition of new, substantive, work related skills,

• Learning about inter-organisational relations,

• Administrative learning centred on managing complex administrative procedures and multiple teams from different countries and organisations.

Exchange and placement participants also report widespread positive impact in self development as the result of their involvement in Leonardo. Both participants and projects have been increasingly sensitised to the added value that can be obtained by being open to learning from other parts of Europe. Leonardo has therefore been an educational experience contributing to the process of Europeanisation of civil society and raising the level of European consciousness.

Deloitte & Touche


However, it is too early to assess the impact that Leonardo may have had in policy terms and on the training systems at national and European level. Many projects have not yet finished and no real assessment has been carried out of the products that have been delivered. There is no evidence that the Leonardo Committee or the Commission have engaged in serious analysis or debate of the practical and policy lessons that undoubtedly have emerged through the projects.

The failure to do this is partly a consequence of inadequate resources and an inappropriate focus on administration and control with the Commission (to the detriment of content-related activities). It is also symptomatic of the relative under development of the European policy system in this area, and is a consequence of the programme’s lack of clarity about its focus. Was it a demand given funding machine allowing practitioners to experiment on interesting themes, or was it a co-ordinated exercise in developing policy through focused funding of agreed priority actions? The former is much truer than the latter.

No serious attention has yet been given to ensuring that the results will be processed and considered at European level. Some evidence exists that Member States have tried to bring national results and lessons together but without direct attributable impact on policy and or practice at least at this stage.

1.8 Conclusions and recommendations We have grouped the principal conclusions and recommendations for the report around the three main themes – complementarity, implementation, outputs and impact.

1.8.1 Complementarity Leonardo is embedded in a range of potentially complementary EU policies in vocational education and training and social integration. The EU needs to operationalise and optimise the links. Providing the opportunities for the exchange of best practice and lessons between programmes and departments would entail the provision of resources and the installation of mechanisms for the flow of information between programmes.

Although Leonardo did not necessarily achieve great synergy between ministries at national level, it did begin a process in some Member States which may yield fruit in the future.

Recommendations: Complementarity

• High-level political commitment to strategic integration of the whole range of current EU training activities.

• Commit to a policy continuum approach involving research, field testing, and mainstream delivery, to replace current uncoordinated EU interventions.

• Dissemination of best practices across other EU programmes should also be undertaken by the establishment of an effective knowledge management strategy

Deloitte & Touche


1.8.2 Implementation The implementation of Leonardo has arguably been the most problematic area of the programme. The programme suffered from a number of structural and design difficulties which impeded effective programme implementation. Both programme design factors and the delivery procedures and mechanisms shaped the implementation outcomes.

Programme Design Recommendations

• A needs analysis should be undertaken to ensure that the programme is accessible to the target group.

• Simplification of all procedures is urgently required. Efficient payment systems must be developed.

• Administrative practices should be tested early in the programme, changed as required in the light of experience

• The actions for mobility projects need to be put into a context where they can contribute to policy ideas.

• Dissemination of project results and best practices is essential.

• Ex-ante targeting needs to be increased.

• Formative utilisation evaluations should be introduced.

• Information systems should be defined from the start of the programme to meet the potential evaluation and exploitation needs, as well as the administrative processes.

• A proper monitoring system which is focused on outputs and outcomes and their use needs to be developed.

The programme was successful in terms of provision of an important tutelage function and an information brokerage service for the projects. However, the opportunity for strategic European planning by the Leonardo Committee was lost, and there was no other effective policy channel for the review of results and the capitalisation thereof.

Leonardo exhibited some over-rigidity, symptomatic of a lack of both organisational learning and knowledge management. There was a tendency to over-regulate aspects of the programme. Which was also a constraint on the implementation capacity of the programme. The programme objectives were lost sight of and were substituted by an over-emphasis on procedures and the administrative process. Inter-organisational boundaries between the TAO and the Commission and between administering bodies and projects were unclear, as they were not adequately and clearly defined. Decisions about the allocation of resources within the TAO did not always reflect the existing administration requirements - insufficient resources were allocated to project selection and proposal management for example. The financial arrangements for the

Deloitte & Touche


TAO, whereby profit was effectively excluded, were extremely questionable and risky. This was an important factor impacting on the operability of the TAO. Participants were effectively discouraged from seeking to participate in Leonardo again. There was no real interest in their environment - the programme suffered from a very inappropriate management system and there was no strategy for the crisis when it came. The system was stacked against the supposed target groups. Vicious circles were set up, such as budget reductions that interfered with the successful completion of the project work programmes, and setting unrealistic project deadlines as a result of late starts. For pilot and mobility projects (under selection procedure II) project quality was secondary to geographic spread.

Recommendations: programme delivery

• Build on the positive aspects, be willing to revise and modify goals. Recognise the natural evolution of programme life-course.

• Succession planning for the post-Leonardo continuation of projects should be an integral part of project planning and approval.

• A crisis management strategy should be in place.

• The roles of the different bodies should ideally be defined through the setting up of guidelines and agreed conventions.

• A system for quality management and procedures needs to be developed.

• In recognition of the partnership character of the projects, selection should not necessarily be based on the nationality of the lead partner alone.

• For Leonardo to be a true European programme the Member States must operate to agreed European guidelines in the selection procedures.

• Incentive (profit) is necessary for limiting risk, and enhancing motivation and quality in outsourced management activities. This should be considered when awarding future management contracts.

1.8.3 Outputs and Impact Outputs and impact are two aspects of the policy cycle that are intrinsically related and so the two sets of findings are discussed together. Apart from the fact that many projects are not yet finished, and that impact could take years to emerge, impact is difficult to measure because of the absence of necessary data about products and achievements. The need for this should have been identified at the outset of the programme. Impact has been conditioned by the organisational dynamics and by the structural programme characteristics which helped shape the implementation environment and subsequent attainable impact for the programme. A number of positive impacts of Leonardo can be identified: • Project participants created informal networks, initiated learning about partnership

operation and learned about participating in European programmes. Evidence of organisational learning was identified.

Deloitte & Touche


• The statistics strand was very useful: the resources provided allowed real progress to be achieved.

• The programme allowed space for the projects to remain at the forefront of technological advances, to adapt to changing market needs, and to allow the projects to become a platform for innovation. This was more by luck than design, however.

• The trans-nationality of the programme contributed to the Europeanisation of civil society and the growth of European consciousness. A wide network of trans-national contacts was built up allowing the exchange of skills, cultures and learning.

• The programme provided a useful boost for change in Central and Eastern Europe.

• Leonardo was responsible for initiation of a number of projects which otherwise could not have come into existence.

However, the evaluation has also found that the programme outputs and impact were limited as the result of: • No attention to policy lessons by Commission of Leonardo Committee. • Absent monitoring of outputs and products and absent dissemination strategies. • Failure to capitalise on project results. • The delays experienced during the programme which served to exclude some

target groups as beneficiaries and constrain the innovativeness of many projects. • The surveys and analyses strand being isolated from the rest of the programme.

Recommendations: Outputs and Impact

• Data on project services and products should be collected and best practices disseminated.

• Mechanisms for ensuring quality of the products should be put in place.

• Programme design should be transparent to users.

• Important best practices must be captured, innovative trends pointed up and strategies for follow through put in place. This is especially important for projects of significant potential and value.

• Future Leonardo programmes will need to capitalise on organisational learning and participant learning through an effective dissemination and knowledge management strategy. .

• The eastern and central European experiences must be built on where they were positive.

• Appropriate data collection is of critical importance for the subsequent measurement of impact. Baseline data needs to be collected and performance measures established.

• Member States and the Commission should establish a formal mechanism for assessing results and processing them thereafter.

Deloitte & Touche


1.9 Overall Considerations We have set out above the recommendations arising from the conclusions of this evaluation. In general terms we argue that future Leonardo programmes would benefit from a much greater user-focus.

In the area of complementarity, output analysis and dissemination there is a need for considerable change. CEDEFOP, the ETF, and new internet-based initiatives could play a useful role in drawing all of the relevant material together (not just from Leonardo 2). We also propose the convening of high level workshops to disseminate key findings and to roll-out best practice to other related EU programmes. The complementary links between Leonardo and other programmes should be officially co-ordinated. Complementarity entails recognition that EU actions in this domain area are systematically related and need to be integrated at the European level and not just at the micro level by individual participants and projects.

A high-level think tank, with the imprimatur of the Member States, could rise above some of the management and allocation functions that the Leonardo Committee carried out, and set an integrated course for the whole range of training activities that the EU is now engaged in. This would encompass the selection of major themes for policy development, research, demonstration projects, events and so on. The continuum of programme styles could be harnessed to give real power to this work and to allow each area of Community intervention to contribute in an integrated way.

With regard to evaluation, we strongly urge the establishment of proper monitoring and data storage systems. In addition - given the need to tailor the programme to the needs of the target groups and to ensure that evaluation findings are relevant and would advocate the establishment of formative evaluation structures. This would entail the decentralisation of the evaluation process to more broadly include the relevant stakeholders. The Expert Group that followed this evaluation would provide a good forum for this as it comprises a fairly representative cross-section of stakeholders, broadening the group to include other important stakeholders would be helpful, if it were to take on this role.

Deloitte & Touche


2 Background and Introduction

2.1 Genesis of Leonardo da Vinci 1

On December 6, 1994, the Council of Ministers of the European Union adopted the Leonardo da Vinci programme for the implementation of a Community vocational training policy (Official Journal L 34O, 29 December 1994).

This programme, adopted for a period of five years (1995-1999) had a key objective of supporting the development of policies and innovative action in the Member States, by promoting projects in the context of transnational partnerships which involve different organisations with an interest in training.

The programme had a total budget of 620 million ECU for the five years and was open to the 15 Member States, the 3 EFTA States of the European Economic Area (Iceland, Liechtenstein and Norway), and progressively to; Cyprus, the Czech Republic, Estonia, Hungary, Lithuania, Latvia, Romania, Poland, Bulgaria, Slovenia, and the Slovak Republic.

The adoption of the Leonardo da Vinci programme was also aimed at a rationalisation of Community action in the area of vocational training, providing the basis to enhance the value of the acquis. Leonardo da Vinci was to facilitate the taking forward of initiatives successfully developed under COMETT, PETRA, FORCE, LINGUA and EUROTECNET, and add new dimensions.

The programme came at a time when the Commission’s White Paper on "Growth, Competitiveness and Employment" forcefully emphasised the crucial importance of vocational training as a key factor in combating unemployment and strengthening the competitiveness of European enterprises.

The programme aimed at responding to the demand for new skill needs which are generated by the evolution of our societies and linking training firmly to solving the problem of employment in Europe. This was echoed in the White Paper "Teaching and Learning - Towards the Learning Society", approved by the Commission on November 29, 1995.

2.2 Scope of this evaluation The bases for our work were the terms of reference issued by the Commission in May 1999 in conjunction with an open call for tender, adapted and complemented by a series of discussions with the Commission and its expert group.

This evaluation took place after the disturbance to the programme caused by the closure of the Technical Assistance Office and the accompanying allegations. Our mandate was to carry out an evaluation of the programme’s overall implementation and results, but - as is the nature of an evaluation - not to attempt to perform another financial audit of the workings of the programme. Nevertheless, we do consider the role and performance of the TAO over time, and inevitably we also look at the consequences of its closure for the delivery of the last year of the programme. The evaluation report provides a mixture of factual information and evaluative conclusions. This is based on a form of methodological triangulation - in built cross-checks – by employing both qualitative and quantitative instruments designed to

Deloitte & Touche


ensure credibility, rigour and reliability of the findings. It has been produced after a thorough programme of documentary research, reviews of key output documents, discussions with representatives of the European Commission, CEDEFOP, Member States authorities and the National Co-ordination Units (NCUs), as well as a series of visits to case study projects. Our analysis and synthesis stage draws together these key findings and has been critical in the generation of conceptual insights into the functioning and operation of the programme. Our report is structured in such a way as to use all of these inputs in order to reach the conclusions and output that we offer.

2.2.1 General objective of the evaluation The general objective of the evaluation was to assess whether the Leonardo da Vinci Programme contributed effectively to the development of EU vocational education and training policy. In particular it aims to contribute to the final report on the programme, which the Commission will present to the European Parliament, the Council, the Economic and Social Committee, and the ACVT later in 2000.

2.2.2 Specific objectives of the evaluation The specific objectives of the evaluation were to evaluate the Leonardo da Vinci 1 programme and provide advice and counselling to the European Commission on five key issues:

• what are the outputs and impacts (results and outcomes) of Leonardo da Vinci 1 compared to the expectations described in the Decision?;

• the effectiveness of the programme: does it achieve its own general, specific and operational objectives?;

• the efficiency of Leonardo da Vinci 1; how economically did it produce results and impacts; how efficient was the management and delivery structure?;

• how to improve the effectiveness and efficiency of Leonardo da Vinci 1 in future programmes?;

• how to develop complementarity and synergy with other vocational education and training instruments and policies both at the EU and Member State levels.

The programme's objectives were very broadly drawn and the result was a programme that reflected the wide differences in training policy, systems, priorities and resource levels that characterise the European training scene. This is even more marked in the case of the pre-accession countries. Accordingly, drawing conclusions in the form of a European synthesis presents difficulties, and some of the report's detailed discussions are more relevant to one, or a small group of, participant countries.

Deloitte & Touche


2.3 The consultant This study was undertaken by the European Services department of Deloitte & Touche, based in Brussels working in collaboration with an international team of consultants all of whom have extensive experience in the field of education and training but were totally independent of any involvement in the Leonardo da Vinci programme.

2.3.1 The Study team The study team consisted of a core team based in Brussels responsible for the project management, survey work and analysis and an expert team based in the participating states and responsible for the field work.

The core team consisted of Richard Doherty (Project Director), Isabelle Collins (Project Manager), Christopher Allinson (Expert), Sarah Batterbury (Expert), Wouter Van Den Berghe (Expert) and Nathalie Belnet (Project Assistant).

The expert team consisted of:

♦ Hanne Shapiro Nordic Countries ♦ Paddy Carpenter UK and Ireland ♦ Lionel Chaty France ♦ Peter Debreceni Eastern Europe ♦ Pedro das Neves Iberia ♦ Prof Theo Papatheodossiou Greece and Cyprus ♦ Anna Gammaldi Italy ♦ Gabriele Marchl Austria, Germany, Liechtenstein ♦ Anne Marie De Jonghe Benelux

2.4 Methodological Approach The consultants originally planned for this work to be carried out on the basis of a September start. Unfortunately, due to a series of difficulties outside the control of the Unit responsible in the Commission, this was not possible, and the contract was finally signed on 22 December 1999.

In addition to this, a set of thematic evaluations which had been planned to contribute to this overall evaluation were never undertaken. Additional inputs in the form of the reports from the Member States are in many cases still not available. However, the date for presenting the final report was not changeable.

Our methodological approach was designed to respond to the key evaluation questions set out in the call for tenders and refined in further meetings with the Commission (see Section 1.2 above). A multi-method approach was employed in order to enable us to answer questions concerned both with process (implementation, management) and with measurable outputs. These two categories of question necessitate two different methodological instruments – qualitative and quantitative. In addition to methodological triangulation (the use of multiple methods to ensure rigour and reliability of the data), the evaluation also employed researcher and data-source triangulation. These provide additional sources of cross-checks which ensure that the findings are neither biased in favour of the view-point of a single researcher, nor based on information derived from a solitary source. Triangulation is a standard and essential aspect of any kind of qualitative research and is employed to ensure rigour,

Deloitte & Touche


reliability and credibility of research through multiple cross-checks of data sources and in the analysis of these sources. In this evaluation the three forms of triangulation - methodological, researcher and data source - were combined to ensure quality, reliability and credibility in the data.

The methodological approach was structured around four main pillars:

• Methodological design

• Documentary review

• Interviews with key informants at European and Member State levels, and case studies with selected projects

• Analysis and synthesis

These methodological pillars overlapped during the course of the implementation of the evaluation as follows:

Nov Dec Jan Feb Mar Apr May

Methodological design

Documentary review

Interviews & project visits

Analysis & synthesis

The evaluation also adopted an explicit stakeholder focus, ensuring that the voices of all key stakeholders were consulted during the collection of data. A number of specific instruments were designed to hear the voices of all the key stakeholders in the programme.

2.4.1 Methodological Design Methodological design was centred on an iterative process with the client. It reflects the complex nature of the evaluation questions which span both output and process issues. As indicated above, the research was also founded on the principle of triangulation - in-built cross-checks serving as a quality control to ensure scientific rigour for the research. In addition to broad methodological criteria, the process of methodological design also encompassed the design of specific research instruments. For this evaluation a number of different instruments were drawn up including: • An interview proforma and reporting template - to direct and structure the field

visits by the national experts (comprising both semi-structured and structured data collection instruments)

• Semi-structured and unstructured interviews designed as a guide for one-off, bilateral encounters with key informants.

• Questionnaires for pilot projects.

Deloitte & Touche


• Questionnaires for mobility projects.

• For placement and exchange participants.

• A format for case study records.

The questionnaires were designed to capture process-oriented information which illuminated the nature and experiences of the programme implementation, management, and performance, and to gather more descriptive data concerned with outputs, products and measurable impact. Structured questions were included to facilitate comparability of responses between projects in a format amenable to statistical manipulation.

2.4.2 Documentary review / Desk study In order to examine the role of Leonardo within the European policy context we have undertaken a review of documentation from the Commission, mainly DG Education and Culture, and also DG Employment and Social Affairs. This was supplemented with discussions with CEDEFOP and examination of a number of documents from them. Finally we have looked at the data provided by CLEO (the Commission’s Leonardo support office established in March 1999) in order to provide certain statistical analyses of the programme.

The documentary analysis was a comprehensive exercise involving the sifting of a considerable quantity of information. We have not summarised all the sources consulted in this evaluation, as they would be too long to list. Moreover, a considerable number of sources consisted of unpublished documentation provided to us by Commission officials and other parties. This information is not amenable for listing in the form of an annex. We place particular importance on respecting the confidentiality of programme records and documents. Accordingly, to give an idea of the breadth of our documentary analysis, we have clustered the documentary sources into 9 categories which are listed below:

• National reports1

• Unpublished documentation supplied by Commission officials and other interviewees

• Interim evaluation, 1997

• Published reports from other related programmes

• CEDEFOP policy report 2000

• The White paper on Growth, Competitiveness and Employment

• Other data supplied from CLEO.

• Thematic papers from CEDEFOP

• Web based materials

These sources were an invaluable input into the evaluation as they offered different perspectives and inputs into our research. While the documentary sources are not listed above in any particular order, the national reports and informal documentary 1 Not all national reports were available at the time this evaluation was finalised.

Deloitte & Touche


sources provided by the Commission were of the greatest import. Programme records and documents were used to help identify important issues for the evaluation, to identify the important processes shaping the implementation of Leonardo, and to provide a useful source of information - filling gaps that could not be observed or learned through interviews. Both official and unofficial documents generated by Leonardo helped to prompt appropriate questions for the interviews and questionnaire design. They were also an invaluable source of contextual information and cross checks.

2.4.3 Consultation with Stakeholders The evaluation also aimed to ensure that the voices of all key stakeholders were consulted during the collection of data. We placed considerable importance on inclusiveness in the evaluation process. We therefore employed both interviews and questionnaires in order to hear the voices of the different key stakeholders in the programme. The key stakeholders identified are set out in the matrix below.

Stakeholder Data-collection instrument

The European Commission Semi-structured interviews

CLEO Semi-structured interviews

CEDEFOP Semi-structured interviews

Member State officials Semi-structured interviews

NCUs Interview proforma (semi-structured and structured data collection instruments)

Pilot projects interview proforma (semi-structured and structured data collection instruments)

questionnaire

Mobility projects (that had been selected centrally)2

questionnaire

Exchange participants questionnaire

In addition to bilateral exchanges with key actors (European Commission, CLEO, Member State officials, NCUs, CEDEFOP), one of the most important groups to question on their views on the Leonardo da Vinci 1 programme were the projects that had actually participated. In view of the fact that these projects are dispersed throughout the European Union, questionnaires were considered to be the best option to spread the consultation process as widely as possible.

2 Absence of data on mobility projects selected in the Member States limited the sample to those which had been selected centrally.

Deloitte & Touche


2.4.3.1 Questionnaires A number of surveys have already been undertaken among projects that have participated in Leonardo da Vinci 1 . These include surveys undertaken as part of the Member State reports. Several projects have reported to us that this is the third or fourth time they have been approached, but there is no evidence to suggest that this unduly reduced the response rate.

After careful consideration, it was decided that the questionnaire would focus mainly on the following points:

The needs of the projects taking part in the programme.

The dead weight of Leonardo da Vinci 1, i.e. would the projects have taken place in some form even if they had not received financial support from the EU? Would they have attempted to use other sources of EU funding?

The added value of Leonardo da Vinci 1, i.e. are projects aware of other programmes? What is the difference between these programmes and Leonardo da Vinci 1?

The management, content and awareness of the programme, i.e. what is their impression of the awareness of the programme? What are the most valued components of Leonardo da Vinci 1? What opinions are there on the quality of the programme?

A small number of complementary questions concerning the concrete and intangible results of Leonardo da Vinci 1 on the projects questioned.

In addition, projects were encouraged to share any further comments not specifically covered by the questions contained in the questionnaire.

The questionnaire was drawn up after studying previous questionnaires drawn up by ECOTEC as part of the intermediate evaluation. Separate questionnaires were provided for Pilot projects and Placement and Exchange projects. In addition, the Placement projects were sent a questionnaire which they were asked to pass on to participants.

Sampling - The Questionnaires

Stratified sampling was employed in selecting projects and exchange and placement participants for consultation via the questionnaire instruments.

The sample of pilot projects was broken down (by project leader) as shown in the matrix below, spread proportionately across pilot projects, placement and exchange projects and study/analysis projects, and by year of approval.

Deloitte & Touche


Total

B Belgium 46

BU Bulgaria 0

DK Denmark 26

D Germany 140

GR Greece 48

E Spain 157

EE Estonia 2

F France 123

IRL Ireland 35

I Italy 142

L Luxembourg 12

NL Netherlands 34

A Austria 27

P Portugal 33

FIN Finland 29

S Sweden 29

UK United Kingdom 140

IS Iceland 9

FL Liechtenstein 1

N Norway 21

EUR Europe 16

CY Cyprus 3

CZ Czech Republic 8

HU Hungary 6

RO Romania 10

LT Lithuania 0

LV Latvia 1

PL Poland 6

SK Slovakia 5

SL Slovenia 0

TOTAL 1108

Deloitte & Touche


As agreed with the Expert Group on the External Evaluation, no projects were included from the 1999 call, as they were insufficiently advanced to be able to comment on outcomes.

As we were constrained to use data provided by CLEO, we only had contact information for the projects administered centrally. No survey was possible of the projects administered and selected in the participating states. As we discussed earlier, a multi-method approach has been the cornerstone of the evaluation. For this reason, sampling was also carried out in a purposeful fashion, seeking out key themes and illustrative events which shed light upon the specific process aspects of the programme. Participants’ responses were particularly useful in this. Maximum variation was sought across Member State, programme strand, and individual themes which were generated from the data and firmly grounded in the rich data we obtained from the participants’ responses. 2.4.3.2 Contacts with National Co-ordinators The importance of the role of the national co-ordinators of Leonardo da Vinci 1 derives from their role in the marketing and promotion of the programme, their contribution to the selection of projects and their permanent support function. In addition their new, more central role in the decentralised Leonardo 2 programme means that their views are central to the implementation and management of the programme. They are also key stakeholders in the programme and so their views are central to the programme’s evaluation.

Therefore, it was important that the views of the national co-ordinators be collected in a systematic way. It was necessary to design a comprehensive interview template which would allow us to tap their knowledge and experience to the fullest extent. A draft template was drawn up by the team for use in interviews, telephone interviews and group meetings and supplemented by a detailed reporting framework. In order to ensure a degree of flexibility and context sensitivity the experts used their knowledge of the participating states to ensure suitability of questions within the broad parameter specified by Deloitte & Touche.

2.4.3.2 Project Case Studies A number of projects were selected as mini-case studies in order to shed light on important issues in more detail.

In particular the case studies focused on themes of central importance to the evaluation:

• Complementarity

• Implementation

• Outputs

• Impact

• Key themes

In selecting the case studies we have attempted to choose a group of projects which will cover the widest possible number of issues, while ensuring that case studies were undertaken in all the Member States. This also follows the qualitative sampling principle of maximum variation purposeful sampling. The distribution of case studies

Deloitte & Touche


across the participating countries is as shown below (again split proportionately by year and across project types):

Total % Case Studies

Belgium 121 4.30% 2

Denmark 69 2.45% 1

Germany 368 13.06% 7

Greece 126 4.47% 2

Spain 414 14.70% 7

France 323 11.47% 6

Ireland 91 3.23% 2

Italy 374 13.28% 7

Luxembourg 31 1.10% 1

Netherlands 89 3.16% 2

Austria 70 2.48% 1

Portugal 86 3.05% 2

Finland 76 2.70% 1

Sweden 77 2.73% 1

United Kingdom 369 13.10% 7

Iceland 23 0.82% 0

Liechtenstein 3 0.11% 0

Norway 55 1.95% 1

Europe 43 1.53% 2

Cyprus 9 0.32% 0

CEEC 100 3.55% 4

TOTAL 2817 100.00% 56

2.4.4 Analysis and Synthesis This is often the most difficult aspect of any evaluation. We have taken particular care therefore to ensure reliability of the data, maintaining multiple triangulation systems throughout the research process and holding detailed records of all questionnaire and field notes relevant to the evaluation.

The data has been analysed using a number of standard research techniques,

Deloitte & Touche


• Statistical aggregation of data enabling trends and measures to be presented in tabular and graphical form;

• Coding of qualitative data enabling the distillation of key themes, generation of conceptual insights and grounded theory;

• Cross-case comparison as a basis for comparative evaluation research. The element of comparison is also an essential part of the generation of grounded insights into the programme. We have analysed the data, thematically selecting important standard evaluation criteria related to key issues for organisational and programme performance and impact.

During the analysis process we have sought to identify three major characteristics within the programme.

• Areas of maximum variation and difference;

• Context-determined characteristics; and

• Areas characterised by similarity and synergy.

In this report we therefore bring together evaluation research which is firmly grounded in and generated by the data, and illustrative, more descriptive measures, which have been deduced from the statistical data available to us.

Deloitte & Touche


3 Vocational Education & Training Policy Context of Leonardo da Vinci I

3.1 Introduction The EU vocational education and training (VET) policy framework into which the Leonardo da Vinci programme was launched in 1995 was not a static one-dimensional structure, but rather a dynamic plural system embracing a variety of actions with the potential for synergy in contributing to the further development of socio-economic policy at the European level.

This framework – which has developed over a period of some 30 years and has continued to develop intensively during the lifespan of Leonardo da Vinci 1 – includes policies, decisions, opinions and initiatives in the domains of employment and economic development; education, personal development and lifelong learning; and social affairs, including equal opportunities. Over the course of the life of the programme policy framework has been increasingly oriented towards the EU employment strategy developed through the Luxembourg process.

The coming together of the economic and socio-educational aspects of EU strategy was clearly articulated in March 2000 at the Lisbon Launch Conference of the Commission working paper 'Implementing Lifelong Learning for Active Citizenship in a Europe of Knowledge'. This working paper defines lifelong learning as "all purposeful learning activity, whether formal or informal, undertaken on an on-going basis with the aim of improving knowledge, skills and competence". It also reflects upon the conclusion of the European Council, at the end of the 1996 European Year of Lifelong Learning, that: "the principles underpinning a strategy of lifelong learning are derived from the fundamental aim of education and training to serve individual, cultural, social and economic well-being and to enhance the quality of life for all citizens."

The working paper goes on to state:

"the Socrates and Leonardo da Vinci programmes, now entering their second phase, are firmly placed within the context of lifelong learning. The main objectives of Leonardo da Vinci are closely linked to the first three pillars of the Luxembourg process with equal opportunities as a horizontal dimension. Within these objectives the Commission will pay particular attention to proposals aiming to develop new approaches to lifelong learning, the transnational dimension of the implementation of the employment guidelines, with a special emphasis on equal opportunities and the development of generic skills aimed at the new technologies and the environment."

The lifespan of the Leonardo da Vinci 1 programme coincided with an evolution and maturation of policies and principles in respect of lifelong learning and its relationship with the maintenance and growth of employment in the European Union. While this will doubtless benefit successor programmes, the policy context existing in 1995 at the launch of Leonardo 1 was rather less focused, having developed in the way illustrated by the process and the chronology below.

Deloitte & Touche


3.2 The origins of the policy framework The basis of EU vocational training policy is to be found in Article 128 of the Treaty of Rome (1957), which provides for the Council of Ministers to establish

"general principles for implementing a common vocational training policy capable of contributing to the harmonious development both of the national economies and of the common market."

The subsequent Council Decision of 2 April 1963 elaborated ten principles around which the then Member States agreed to begin to harmonise their actions on vocational training. The second of these principles included a combination of the individual social and educational aspects of learning with the economic effects of skills training

"to broaden vocational training on the basis of a general education, to an extent sufficient to encourage the harmonious development of the personality and to meet requirements arising from technical progress, new methods of production and social and economic developments;

to enable every person to acquire the technical knowledge and skill necessary to pursue a given occupation and to reach the highest possible level of training, whilst encouraging, particularly as regards young persons, intellectual and physical advancement, civic education and physical development."

Flowing from the Council Decision was a raft of provisions for co-operation between Member States, together with the setting up of the Advisory Committee on Vocational Training. Between 1963 and 1983, the development of the framework of vocational policy in the EC took the form of several non-binding resolutions. The result of the first of these was the establishment of the European Centre for the Development of Vocational Training (CEDEFOP) in 1975. The purpose of the centre was to provide information and expertise on vocational training and to stimulate research, particularly on the transferability of training standards to support the free movement of workers within the European Community.

The growth of unemployment in the 1980s led to the raising of the profile of vocational training as a strategic response to the phenomenon of long-term joblessness in the EC, and the need to support the restructuring of the labour market, by equipping workers with the capacity to adapt to technological change. In addition to taking account of the effects of new technology, labour market policy was faced with new challenges posed by the substantial growth of unemployment amongst young people and by the increasing economic activity amongst women.

A Council Resolution of July 1983 emphasised the strategic role of vocational training as:

"An instrument of active employment policy to promote economic and social development and adjustment to the new structure of the labour market;

A means of ensuring that young people are properly prepared for working life and their responsibilities as adults;

An instrument for promoting equal opportunities for all workers as regards access to the labour market."

Deloitte & Touche


For young people, in particular for those without qualifications, Member States committed themselves to providing basic training and work experience for at least a six month period, for those who wished to participate.

While the July 1983 resolution was articulated at the European level, decisions regarding the method and pace of implementation remained the preserve of Member States themselves, a principle (following legal testing as described later in this chapter) that continues to underpin EU vocational education and training policy.

3.3 Chronology of vocational training and relevant education action programmes 1986-19933

Following some 20 years of non-binding resolutions regarding vocational training, the European Communities' institutions agreed in the mid 1980s to devise a legal base, founded upon Article 128 of the Treaty of Rome, for further developments in the field. As a result of this flowed a variety of action programmes:

COMETT I (1986-1989) Programme on co-operation between universities and enterprises regarding training in the field of new technology, with a budget of MECU45. It financed 1,300 projects and established 125 university-enterprise training partnerships. COMETT II (1990-1994) had a budget of MECU200 - including an EFTA contribution.

ERASMUS (1987-1995) Programme setting up the European Community Action scheme for the mobility of university students, with a budget of MECU500.

PETRA (1987-91) Programme for the vocational training of young people and their preparation for adult and working life, with a budget of MECU40. Participants numbered some 75,000, along with more than 10,000 teachers and trainers. PETRA II (1992-1994) had a budget of MECU104.2

LINGUA (1990-1994) Programme to promote foreign language competence in the European Community, with a budget of MECU153. More than 7,000 language teachers received in-service training and 33,000 teachers participated in exchanges.

EUROTECNET (1990-1994) Programme to promote innovation in the field of vocational training resulting from technical change in the European Community, with a budget of MECU9.2. It provided funding for the networking of innovative projects in vocational training and for transnational research on specific themes linked to vocational training.

FORCE (1991-1994) Programme concerned with the quality and quantity of continuing vocational training, with a budget of MECU88. There were 720 projects, involving the transfer of innovation and expertise in continuing vocational training.

HELIOS I (1988-1991) and HELIOS II (1993-1996) promoted the social integration and independent lifestyle of people with disabilities. HELIOS II had a budget of MECU37, enabling 1,150 organisations working for disabled people to take part in discussions and exchanges with similar organisations in other Member States, with particular emphasis on vocational training and preparation for working life.

3See CEDEFOP: An Age of Learning: vocational training policy at European Level. Thessaloniki 2000

Deloitte & Touche


3.4 Clarification of the legal aspects of vocational training policy The use of Resolutions based upon Article 128 of the Treaty of Rome, backed up by Article 235 (which provided legal cover where specific treaty articles did not exist), disturbed some Member States concerned about the extent of Community competence in the field of vocational training. A legal action arising from 1989 Commission proposals to adapt ERASMUS culminated in a judgement by the European Court of Justice that Article 128 did indeed provide for legal measures that could be binding on Member States in the field of vocational training. The judgement also stated that a common vocational training policy could, broadly, include higher and university education.

The anxieties of Member States concerning the extension of legally-binding vocational training measures at a European level were confirmed by the Social Charter of Fundamental Workers' Rights, adopted under the French presidency in 1989. This document, while non-binding in itself, proposed a right of access by all workers to continuing training systems and to training leave.

The debate on the scope of European-level binding vocational training policies was resolved in the run up to the Maastricht summit which revised the social chapter of the Treaty of Rome, including Article 128. In Article 127 of the Treaty of the European Union, signed at Maastricht in 1992, the previous common vocational training policy was replaced by a Community vocational training policy, explicitly designed to support and supplement the vocational training activities of the Member States, but preventing harmonisation at a European level. Article 126 of the Maastricht Treaty removed education from the legal ambit of vocational training policy, again excluding harmonisation of Member State policies and actions. Where actions under Article 128 had required a simple majority, actions under the new Article 127 were to be decided in Council by qualified majority and after consultation with the European Parliament.

Thus, the formal policy framework into which the Leonardo da Vinci programme was later to be launched is as follows:

Treaty on the European Union 1992 - Article 127

1. “The Community shall implement a vocational training policy which shall support and supplement the action of the Member States, while fully respecting the responsibility of the Member States for the content and organisation of vocational training.

2. Community action shall aim to:

a) facilitate adaptation to industrial changes, in particular, through vocational training and retraining;

b) improve initial and continuing training in order to facilitate vocational integration and reintegration into the labour market;

c) facilitate access to vocational training and encourage mobility of instructors and trainees and particularly young people;

d) stimulate co-operation on training between educational and training establishments and firms;

e) develop exchanges of information and experience on issues common to the training systems of the Member States.

Deloitte & Touche


3. The Community and the Member States shall foster co-operation with third countries and the competent international organisations in the sphere of vocational training.

4. The Council, acting in accordance with the procedure referred to in Article 189c and after consulting the Economic and Social Committee, shall adopt measures to contribute to the achievement of the objectives referred to in this Article, excluding any harmonisation of the laws and regulations of the Member States.”

3.5 Vocational training - EU fora Alongside the development of the legal policy framework concerning vocational training in the European Community/Union, a number of relevant bodies have been created over the past thirty years, each with a different purpose and composition, but each concerned discretely with vocational training policy and practice at a European level.

The Advisory Committee for Vocational Training (1963), established following the 1963 Council decision. Its task is to provide the Commission with an opinion before formal vocational training proposals can be made.

The Management Board of CEDEFOP (1975) has the role of managing CEDEFOP as the EU information and research body in vocational training, with a tangential relationship to policy developments.

The Governing Board and the Advisory Forum of the European Training Foundation (1994), which has the task of managing the ETF's work in support of training developments in Central and Eastern Europe.

The European Forum for the Transparency of Vocational Qualifications (1997) was established as a result of an initiative by CEDEFOP and the Commission with the task of progressing the transparency of qualifications across the EU.

The Social Dialogue - working party on education and training, established to provide a forum for the social partners to meet and to produce common opinions on vocational training policy and issues.

The Directors General for Vocational Training of the Member States meet once under each presidency, with an agenda normally associated with the priorities of the presidency. This is an informal forum for senior civil servants to exchange views and information without formal outputs but with influence on the progress of vocational training policy development.

The Committee of the Leonardo da Vinci Programme (1995) has the task of guiding the Commission in the implementation of the Leonardo programme.

Each body occupies a separate position in the European-level framework of vocational training, with little, if any, structured dialogue between them.

3.6 Other Policy Influences

Two 1990s White Papers helped raise the profile of vocational training into a central position in the EU policy arena:

Deloitte & Touche


• Growth, Competitiveness, Employment - the challenges and way forward into the 21st century (1993)

• Teaching and learning - towards the learning society (1995)

1993 Growth, Competitiveness, Employment - the challenges and way forward into the 21st century

This White Paper stressed the importance to the EU of intangible investment in education and research, proposing a strategy to achieve more employment-intensive economic growth. The paper argued a role for vocational training in addressing issues of unemployment and competitiveness. In the light of inadequacies identified in existing vocational education and training systems the white paper proposed the adoption of policies and practices of continuing training and lifelong learning. The European Year of Lifelong Learning, one result of the paper, took place in 1996 with the overall aim of promoting:

− "personal development;

− the integration of individuals into working life and society;

− the participation of individuals in the democratic decision-making process;

− the ability of individuals to adjust to economic, technological and social change;"

With a number of sub themes advocating, for example, the provision of vocational training for young people, continuing education and training, co-operation between commerce and education for SMEs, measures aimed widening and increasing opportunities for lifelong learning, and the development of a European dimension in initial and continuing education .

The Council conclusions on the strategy for lifelong learning proposed the following framework of areas for further development:

− "challenges for the school system;

− economic and social considerations;

− continuing education and training;

− pathways and links between general and vocational education;

− access, certification and accreditation; and

− teachers and adult educators."

1995 Teaching and learning - towards the learning society The Competitiveness White Paper themes of lifelong learning were further emphasised in this second White Paper, which noted three major developments affecting the European social economy: the globalisation of trade, the emergence of the information society and, the progress of science and technology.

Deloitte & Touche


The White Paper proposed that education and vocational training policies should be aimed at broadening the knowledge base and improving individual employability. The paper identified five objectives towards a learning society:

− "to encourage the acquisition of new knowledge through the introduction of a European system of accreditation of technical and vocational skills

− to bring about closer co-operation between schools and business through apprenticeships and training

− to combat social exclusion by offering second-chance education to those affected

− to ensure proficiency in three Community languages; and

− to equalise treatment of capital and training investment."

3.7 Other EU Policy Influences In addition to the economic and employment-related elements mentioned above, the European Union is based on several overarching principles, a number of which are of particular relevance to the Leonardo da Vinci programme. The principle of subsidiarity has particular relevance for education and training, as we have shown in describing the debate about legal base. The Amsterdam Treaty (1997) reinforced the principles of European citizenship, a commitment to tackle social exclusion, and the development of an integrated approach to the development of a common employment strategy. The Leonardo programme was conceived and implemented within the aegis of these main principles of European Integration, as well as building on the “traditional” Community approach of transnational demonstration projects, and upgrading significantly the opportunities for personal mobility.

The matrix below summarises these key themes, showing the way in which Leonardo has translated these key EU principles into specific programme elements.

Principles of the EU relevant to Leonardo

Feature of Leonardo

Interaction between European citizens Trans-nationality

Subsidiarity National Co-ordination Units and national project selection

Development of the concept of European citizenship

Mobility, placement & exchange

Development of a common strategy on employment & co-ordination of national strategies

Attempt to develop a European Vocational Training policy

Development of a Community vocational training policy

Linkage of programme to labour market policy.

Deloitte & Touche


Commitment to tackle social exclusion

Targeting of specific marginalised groups

Promote equality between men and women

A stated objective

Relevant Principles derived from the Structural Funds approach

Additionality Co-financing

Partnership Transnational projects & NCU role

Important Policy Domains

SMEs

Social Partners

Eastern & Central Europe

Innovation & technology

All present in Leonardo

We will return to these different themes at various points throughout the report. However, it is worth noting at this stage that Leonardo accorded greater value to some rather than others during its implementation.

3.8 Launch of Leonardo da Vinci By 1995 the EU vocational training and education policy and activity framework had evolved to:

• provide an agreed revised legal basis for Community actions in this field, not seeking harmonisation but respecting the autonomy of Member State provisions while providing for value added at the European level;

• encompass the twin themes of development of the individual as citizen and the development of the individual as economic actor through the principles of lifelong learning;

• support several permanent vocational training policy/practice discussion forums;

• acquire a practice base through a range of funded programmes which would be substantially brought together under the new Leonardo programme

On 6 December 1994 the Council approved the Leonardo da Vinci programme as the first Community Vocational Training Action Programme since the revised TEU of Maastricht resolved the legal framework.

3.8.1 Leonardo da Vinci I – Common framework of objectives Leonardo da Vinci began with a common framework of objectives provided for in Article 1 (3), the aim of which was to contribute to the achievement of the objectives of Article 127 of the Treaty by pursuing the 19 different objectives in the Decision. In accordance with the recommendations of the expert group convened by the

Deloitte & Touche


Commission to guide this evaluation, we can cluster the objectives around four themes: • European Union level objectives

• National level objectives

• Target group objectives

• Horizontal objectives

The matrix below shows how the 19 Leonardo Objectives may be clustered around these themes:

Clusters Objectives

European Union level objectives:

• developing the European dimension in vocational training and vocational guidance

• supporting vocational training policies in such a way that all workers in the Community have access to continuing vocational training throughout their working life without any discrimination

• promoting the gradual development of an open European vocational training and vocational qualifications area, particularly through the exchange of information and experience on obstacles to application of the free provision of services by training bodies. This is also a horizontal objective

• -promoting co-operation on skill requirements and training needs, and

- encouraging the acquisition and transparency of qualifications and an understanding of the key skills relevant to technological development and the functioning of the internal market, including the free movement of goods, services, persons and capital, the competitiveness of undertakings and the requirements of the labour market. This is also a horizontal objective

National level objectives

• improving the quality and innovation capacity of Member States' vocational training systems and arrangements

• giving all young people in the Community who so wish the possibility of one or, if possible, two or more years of initial vocational training after their full-time compulsory education, leading to a vocational qualification recognised by the competent authorities in the Member State in which it is obtained. This is also a target group objective

Target group objectives

• promoting lifelong training so as to encourage ongoing adaptation of skills to meet the needs of workers and undertakings, contribute to reducing unemployment and

Deloitte & Touche


facilitate personal development

• encouraging specific vocational training measures for adults without adequate vocational qualifications, in particular adults without adequate education

• promoting vocational training for young people and preparing young people for adult and working life, taking account of the requirements of society and technological change

• encouraging specific vocational training measures for disadvantaged young people without adequate training and in particular young people who leave the education system without adequate training

• promoting equality of access to initial and continuing training for persons disadvantaged by socio-economic, geographical or ethnic factors or by physical or mental disabilities; special attention must be given to persons affected by several risk factors likely to cause their social and economic exclusion;

• promoting equality of opportunity as regards access for migrant workers and their children and the handicapped to vocational training and their effective participation therein;

• promoting the development of vocational guidance facilities with a view to providing every individual with the opportunity to have lifelong high-quality vocational guidance;

Horizontal objectives • enhancing the status and attractiveness of vocational education and training and promoting parity of esteem for academic diplomas and vocational qualifications;

• promoting equality of opportunity as regards access for men and women to vocational training and their effective participation therein, in particular so as to open up new areas of work to them and encourage them to return to work after a career break

• promoting vocational training in the light of the results of technological research and development programmes, particularly by means of co-operation between universities and undertakings in the sphere of training in technologies, their application and their transfer

• fostering the development of methods of self-training at the workplace and of open and distance learning and training, in particular to facilitate access to continuing vocational training;

• encouraging the development and integration of key

Deloitte & Touche


skills in vocational training measures, with the aim of promoting the acquisition of flexible qualifications and personal skills necessary to worker mobility and the needs of undertakings

• supporting activities aimed at developing linguistic skills as part of vocational training measures

In addition to the large number of objectives there were also 4 strands:

• support for the improvement of vocational training systems and arrangements in the Member States

• support for the improvement of vocational training measures including university – enterprise co-operation concerning undertakings and workers

• support for the development of language skills, knowledge and the dissemination of innovation in the field of vocational training

• support measures,

In addition to this list of desirable Leonardo activities, a more focused set of five priorities was established for the programme in 1996 and 1997: 1. The acquisition of new skills;

2. Forging closer links between educational/training establishments and enterprises;

3. Combating exclusion;

4. Promoting investment in human resources;

5. Promoting access to skills through the information society in the context of lifelong learning.

This evaluation provides an opportunity for stakeholders to assess the degree of coherence in the above context, within which Leonardo da Vinci I was implemented.

The large number of objectives, priorities, and strands created a confused picture for potential applicants. There was no clear hierarchy of objectives, some of the “priorities” were - in reality – objectives, and many of the “objectives” were, in real terms, more akin to activities.

In Leonardo 2, the number of objectives has been reduced to three. However the picture remains confused. Priorities, strands and objectives provide a problematic intersection which translates with some difficulty into practical project proposals and activities. Those closely involved with Leonardo in the Commission also report some difficulty in unravelling the hierarchy of objectives. Goal-based evaluation is also extremely difficult in the absence of a clear hierarchy of objectives.

Deloitte & Touche


4 Complementarity with other EU instruments In this chapter we outline the main EU programmes which are related in their objectives and purpose to Leonardo. This enables us to situate Leonardo within the broader programming environment and thereby demonstrate its particular niche and complementarity with other programmes. We show that Leonardo can be seen as belonging to a hierarchy of programmes in training and social investment, and that its distinctiveness vis-à-vis other European programmes can be shown by its offering of large-scale opportunities for trainee mobility and of funding for active testing of innovative ideas. By understanding the way in which Leonardo is embedded within the broader programme context we are able to demonstrate the potential for cross-programme knowledge sharing and dissemination of good practice.

The Leonardo programme came into being alongside a number of related EU programmes. In this chapter we give particular focus to the European Social Fund, the ADAPT and EMPLOYMENT initiatives, the Research and Technological Development Framework Programmes, and the related parallel programme SOCTRATES. In the paragraphs which follow, however, we first situate Leonardo in its context by giving a brief overview of the different programmes which were integrated into the Leonardo programme.

4.1 Relationship with previous programmes Initiatives in the area of vocational training across Europe are numerous. A brief search in the CORDIS4 database of EU programmes found 46 programme records related to the key words “vocational training” from 1989 to now. While a small number of these concern broader policy issues such as EU-US co-operation, the vast majority are EU programmes with a direct involvement in vocational training. Equally, the database is not exhaustive and will not pick up some of the smaller EU actions such as that on the Third System and Employment or the ESF Article 6 pilot projects. This gives, however, a broad indication of the range of activities being carried out on this topic within the Commission and an indication of the task required to ensure complementarity

The Leonardo da Vinci programme was aimed at a rationalisation of Community action in the area of vocational training. It took forward initiatives developed under COMETT, PETRA, FORCE, LINGUA and EUROTECNET and added new dimensions.

In very broad terms these programmes were respectively concerned with the development of University/Education Training partnerships, the transition from school to work, re-qualification of the workforce, the development of language skills and promoting innovative actions in the field of vocational and continuing training.

Each of these prior programmes can be clearly identified within the strands of Leonardo da Vinci.

The Leonardo da Vinci Technical Assistance Office was also responsible for the transition from the programmes to Leonardo da Vinci. One of the initiatives they undertook in this area was the development of a database of products from the past

4 Community Research and Development Information Service (www.cordis.lu)

Deloitte & Touche


programmes. However, this is no longer accessible due to the difficulties that have beset the programme.

4.2 Relationship with other Community interventions

4.2.1 the Structural Funds The majority of the EU's investment in the Member States is done through the medium of the Structural Funds (The European Regional Development Fund, The European Social Fund, The European Agricultural Guarantee and Guidance Fund and the Financial Instrument for Fisheries Guidance). While all these funds contribute in some form to training, a significant proportion of this entails investment in physical infrastructure. The investment in the human infrastructure is generally carried out through the European Social Fund. 4.2.1.1 The European Social Fund (ESF) The ESF is the EU’s main financial instrument for investing in people. Its mission is to help prevent and fight unemployment, to make Europe's workforce and companies better equipped to face new challenges, and to prevent people losing touch with the labour market.

In the programming period 1994 to 1999, the ESF priorities were expressed in Objectives 3 and 4 of the Structural Funds. For Objective 3 the fund supported actions which:

• facilitated the occupational integration of long term unemployed persons,

• facilitated the occupational integration of young people in search of employment, including up to two years initial vocational training

• promoted integration of people excluded from the labour market

• promoted equal opportunities for men and women on the labour market

The types of actions supported included

• vocational training, pre-training, guidance and counselling

• temporary employment aids

• the development of training, employment and support structures.

Objective 4 was aimed at facilitating the adaptation of workers to industrial change through:

• anticipation of labour market trends and vocational qualification requirements

• vocational training and retraining, guidance and counselling

• assistance for the improvement and development of appropriate training systems

with a focus on the specific needs of SMEs.

Thus the ESF and Leonardo da Vinci had similar goals, especially in terms of promoting and improving training, education and counselling and the development of vocational qualifications. The major difference lies in the immediate target beneficiary of the projects, the approach and the types of project assisted. The ESF channels EU money into helping the Member States. Thus the focus and outcomes are designed to have an impact at a national, regional, or local level. The

Deloitte & Touche


ESF has wider objectives than Leonardo da Vinci covering support for infrastructure, improving vocational education and training systems, and the promotion of a culture of training. The ESF’s greatest impact is arguably on labour market structures and institutions. Leonardo da Vinci, conversely, was designed with a more specific EU focus in the sense that it represented an opportunity for the formulation of EU-wide, transnational Vocational Education and Training approaches. This transnational dimension was unique to Leonardo da Vinci.

In terms of scale of resources, the ESF between 1994 and 1999 had some 57191 million ECU at its disposal.5 (i.e. nearly 100 times more than Leonardo da Vinci 1) 4.2.1.2 ADAPT AND EMPLOYMENT Community Initiatives are intended to act as a kind of laboratory within the ESF for the development of new approaches for helping people access, and retain work. The projects within the two Initiatives Adapt and Employment were all based on five key principles:

• working with partners in different Member States;

• developing innovative approaches;

• local involvement;

• creating a multiplier effect with results;

• complementarity with other EU programmes.

There were two Community Initiatives in the field of Human Resource Development (HRD) which had a relationship to the Leonardo da Vinci area – ADAPT and EMPLOYMENT. Of these, ADAPT had the closer relationship.

The ADAPT Initiative (with a budget of ECU 1.4 billion), drawing on previous experience with EUROTECNET and EUROFORM, complemented Objective 4 of the Structural Funds, assisting innovative approaches to the challenges of preparing workers for industrial change. It required that projects be undertaken in a context of transnational collaboration between linked projects.

A new priority within the ADAPT Initiative -ADAPT-Bis- was added to address the labour market implications of the Information Society.

A reserve budget was drawn on to finance the new strands; this provided an additional ECU 460 million for EMPLOYMENT and ADAPT. ADAPT projects were designed to fall into four categories of measures:

♦ Supply of training, counselling and guidance

♦ Anticipation, promotion of networking and new employment opportunities

♦ Adaptation of support structures and systems

♦ Information, dissemination and awareness actions

5 Source:'The impact of structural policies on economic and social cohesion in the Union 1989-99'. European Commission 1997.

Deloitte & Touche


The number of projects funded under the ADAPT Programme was roughly similar to that funded by Leonardo da Vinci. However as the budget was over 20 times larger, it can be seen that the average size of the projects was significantly greater.

ADAPT projects were promoted and organised by partners who reflected the whole range of companies, organisations and representative and intermediary bodies which make up the contemporary labour market and those who provide it with training, information, and other resources and services. Member States developed their own Operational Programmes and indicated their own priority promoters, in some cases excluding others from direct eligibility, but generally the following types of organisations were involved in active partnerships:

• companies (especially SMEs), groups of companies, sectoral organisations, chambers of commerce, crafts and agriculture;

• trade unions and other worker organisations;

• public and private training organisations and universities;

• local and regional authorities;

• local development agencies;

• non-governmental and voluntary organisations;

• organisations concerned with equal opportunities.

Thus there was a clear overlap in terms of target project promoters between the two programmes.

As in Leonardo da Vinci, innovation was a key issue for ADAPT projects, defined in terms of making changes and introducing new practices in the face of technological and other changes and transforming the ability of workers themselves to respond to them.

Although both ADAPT and Leonardo da Vinci had a core element of transnationality, this was implemented in different ways. Within ADAPT the partnership took the form of linked projects involving at least 2 Member States and requiring approval of each project by the Member State concerned. Under Leonardo da Vinci the projects were required to be true partnerships – one of the factors which the participants reported as most valuable in Leonardo da Vinci. ADAPT did not exhibit or require this to the same extent during the selection process. The third important element for ADAPT was the notion of mainstreaming. This was less central for Leonardo da Vinci.

In terms of dissemination, it was acknowledged by the ADAPT programme that this was one of the key elements of the programme and the intention to relate the products of innovation to mainstream practice had to be built into the projects from the start.

The EMPLOYMENT Initiative approved in 1994, with an ESF budget of ECU 1.5 billion, contained three interdependent strands, two of which were developed from the original NOW and HORIZON Initiatives:

• NOW - to promote equal opportunities for women;

Deloitte & Touche


• HORIZON - promoting pathways to employment for persons with disabilities and others threatened with exclusion;

• YOUTHSTART - integrating young people under 20 into the labour market, especially those without adequate training or qualifications.

The structure was modified with the introduction of a new strand, INTEGRA, specialising in the employment barriers faced by immigrants, refugees and other people at risk of exclusion. HORIZON continued to serve the needs of people with disabilities. All of these focused more on direct employment projects rather than on training.

4.2.2 The Research and Technological Development Framework Programmes

EU Research and development expenditure is carried out in the context of multi-annual Framework Programmes, comprising a series of research programmes. Under the Fourth Framework Programme (1993-1997), and the current Fifth Framework Programme (1998-2002), there were two main areas of research funded which had a close relationship to the work of Leonardo da Vinci. These approximate to actions relating to the human element and those relating to the technology/innovation element.

Under the Framework Programmes research is funded on a shared cost basis for projects proposed by transnational partnerships similar to the type of partnership required for Leonardo da Vinci projects. Also, as in Leonardo da Vinci there is the possibility to involve the associated states, leading to a wider public than just the EU Member States. As an element of their applications, projects are required to present a Technology Implementation Plan which details how the outputs from the project will be exploited, an element which was not present in such a detailed form under Leonardo da Vinci.

In terms of the human elements, the most relevant programme under the Fourth Framework Programme was Training and Mobility of Researchers (TMR), especially the Marie Curie Fellowships. This programme involved a relatively small number of participants, highly targeted in the field of scientific research. The placements were on average longer than Leonardo da Vinci placements and there is little real overlap. The Targeted Socio-economic Research Programme (TSER) had a key action on improving the socio-economic knowledge base. This produced some 200 projects on a range of themes such as innovation in education and training via technology, Human resource development and competence development. As an example the theme of innovation in education and training via technology produced 7 projects, all of which could in essence have been funded under Leonardo da Vinci.

Under the Fifth Framework Programme this area has been integrated into the new Programme improving human research potential and the socio-economic research base. This works in 3 areas:

♦ Improving the human research potential (which contains the mobility actions)

♦ Improving the socio-economic research base, and

♦ Support for scientific and technological development policies in Europe.

The Programme has a budget of €1280 million over 5 years.

Deloitte & Touche


In terms of technology and innovation, under the Fourth Framework Programme, the Telematics Application Programme had a strand on Telematics for Education and Training looking specifically at aspects of technology in training and education. This programme also had a supporting measure focusing specifically on the dissemination of the results of the projects.

Under the Fifth Framework Programme this has been incorporated into the Information Society Technologies (IST) Programme, which also absorbed the ESPRIT and ACTS programmes. PROSOMA is a multimedia showcase for the results of these programmes, also aimed at promotion of the take-up of the outputs.

Also under the Fifth Framework Programme, the JRC workplan budget line "underpinning European Competitiveness" is financing an action at the IPTS in Seville regarding the development of a "European Observatory of Knowledge" designed as the core activity to combine all essential sources of information on education, training and skills available in the European Union.

Its objectives are:

• Inventory and networking of existing observatories on "Education", "Training" and "Youth". Improvement of data compatibility, retrieval and information exchange procedures. In an advanced stage of development the Observatory will be able to provide access to statistics, to indicators on the quality of education, to contribute to the analysis of the current state of the art of the European educational system and to consensus generation on best practices.

• Monitoring organisational changes in public administration and industry, the evolution of the utilisation of new types of services, the impacts on the management of supply chains and the related emerging demands for new skills in society.

• Development of benchmarking tools to analyse the impacts of current skills shortages on competitiveness and innovation at company level (focus on SMEs) and in entire supply chains at regional, national and EU level.

• Analysis of the dynamic links between the emerging demands for new skills and the need for a corresponding timely and flexible adaptation of the education and training programmes.

• Analysis of the impact of emerging educational, training methods and techniques on human knowledge acquisition ability.

• Monitoring, analysis and testing of emerging data mining techniques and knowledge management systems.

• User friendly dissemination and interactive exchange of information on skills, education and training for citizens, SMEs, R&D providers & users, National/Regional policy decision-makers. Particular focus will be devoted to disabled citizens.

As such it offers an opportunity for the dissemination of the results of Leonardo da Vinci in the future. CEDEFOP is a partner in the project and thus there is the start of a co-ordination mechanism in place, should the Commission decide to use it.

Deloitte & Touche


4.2.3 SOCRATES SOCRATES and Leonardo da Vinci were in many senses parallel programmes. The former focused on education and the latter of vocational training. In reality these distinctions are less clearly maintained. Approximately 18% of Leonardo da Vinci projects were held by universities for example. This issue is discussed in more detail in Chapter 6.

The issues on complementarity, synergy and possible overlap between SOCRATES and Leonardo da Vinci can be considered from different perspectives, such as:

♦ the content of the programmes;

♦ the actors and beneficiaries involved;

♦ the structure and management of the programme.

Each of these will be briefly discussed below.

As regards the content, it is quite obvious that SOCRATES and Leonardo da Vinci could never have been designed in a fully complementary way. Indeed, the distinction between the concepts which are at the heart of both programmes - “education” for Socrates and “vocational training” or “training” for Leonardo da Vinci - has become increasingly blurred over the last decades – and moreover covers different realities in different countries. It is clear that this situation provides opportunities for both synergy and overlap.

A clear example of the possible overlap is the funding for activities in the field of “Adult education” (SOCRATES) and “Continuing (vocational) training” (Leonardo da Vinci) where the distinction has never been clarified. We did not find any evidence of a clear and accepted policy – either at European or at Member State level – that would ensure complementarity. In fact, under SOCRATES 2 and Leonardo da Vinci 2 this parallelism has been continued and even reinforced. Other areas where complementarity was unclear concerned open and distance education, language training, industry-university collaboration and certain mobility activities.

On the other hand, it appears to us that little overlap has actually occurred in the field – but also little synergy. This is first because of the uneven access policy, whereby the formal “education” community could take part in both, but where SOCRATES was restricted to them alone. Further, because of the activities supported by the two programmes only addressed a very tiny part of the possible target groups.

Another characteristic of complementarity concerns the organisations involved in both programmes. There, the boundary line is, in principle simple. In SOCRATES, with a few exceptions, only education institutions can participate; in Leonardo da Vinci in principle any organisation can participate - with a strong preference for organisations in need of vocational training. In reality, it is the education institutions that often take the lead in Leonardo da Vinci (in some countries up to 90% of projects are co-ordinated by them). This holds in particular for higher education institutions, who often pursue not so much a “programme logic” (“what can we get out of a programme”), but rather a “portfolio logic” (“what do we want to do, and where can we get funding for it”). There is little evidence that this practice leads to a lot of “double funding” or “overlap” between projects supported, but institutions often tend to optimise the deployment of their human resources by applying to different funding mechanisms.

Deloitte & Touche


It should also be noted that, although SOCRATES is almost exclusively involved with education institutions, in certain actions, other types of organisations can be involved (e.g. Thematic Networks; or Mobility of Students which may include a practical placement in a company).

Complementarity and synergy could also be considered from a structural level, i.e. the way the programmes have been organised, managed and implemented. From this perspective, synergy has been poor. Both at European (Committees, Commission and TAO) and national levels (with a few exceptions), management and organisation of the programmes was not closely co-ordinated, with virtually no contact at policy and implementation level. This can be seen in simple operational things like different funding mechanisms and rules for similar activities, or different timing and formats for selection procedures. But also little structural contact occurred at policy level, for instance in the decisions on what types of actions ought to be centralised / decentralised, and in the design of both SOCRATES 2 and Leonardo da Vinci 2. The people who are knowledgeable about SOCRATES and Leonardo da Vinci - in the Commission, in the TAOs, in the Agencies, … - are very few in number. Some beneficiaries, with a long tradition in obtaining European funding, appear to be better informed about the similarities, differences and scope for overlap and synergy than the policy makers themselves.

The distinction between SOCRATES and Leonardo (and also YOUTH) is therefore predominantly one of organisations and policy-makers, rather than one of beneficiaries and target groups. From that perspective, there is scope for both complementarity and overlap. In practice, some complementarity was achieved in the field, rather than at policy level – and overlap was not high. But synergy between both programmes has been weak, at all levels: European policy level, Member State level, and organisational level.

4.2.4 Other related activities, such as SME programmes and Social Fund pilot projects

In addition to the mainstream programmes there are a number of small actions financed by the Commission which have some relationship to vocational education and training.

For example, under Article 6 of the current ESF Regulation, 1% of the ESF budget is set aside to provide funding for pilot projects that develop innovative ideas for promoting employment and social integration. These projects are selected centrally and there is a programme of network animation associated for the exchange of experience and accompaniment of the projects. Equally in the case of the budget line for pilot projects in the field of the Third System and Employment, there was a required element of transnationality but also a system of network animation and valorisation of the outputs. Both these elements are relatively small in terms of the number of projects supported.

Equally some pilot actions undertaken in the framework of the multi-annual programme for SMEs have an element of training for SMEs. Again these are relatively small and often do not have any mechanism for the dissemination of the results.

DG Enterprise’s expertise and level of resources available for its own developmental work in the training field are too limited to allow it to make a real impact. The

Deloitte & Touche


importance of the SME dimension of training policy is not to be underestimated (as the Leonardo da Vinci Decision noted), but the actions in the SME Programme are too limited to contribute in a real way to European thinking. There was a genuine lack of knowledge among stakeholders in the Member States about the training part of the SME Programme. Nor is there evidence that the managers of Leonardo da Vinci really included or benefited from that work in their own planning, despite their DG’s role as the lead agency for training policy development in the Commission. In the other direction, the influence of the lessons of Leonardo da Vinci in the SME area was not easily visible in the planning for DG Enterprise’s Communication on training for SMEs, despite former DG XXII’s view that greater discussion between the two Directorates General would have been valuable. It seems this dialogue now longer exists at all.

4.3 Duplication or synergy/value added According to DG Employment and Social Affairs:

“Policy development and implementation under the Structural Funds are also linked to other EU programmes, especially Leonardo da Vinci Da Vinci, which supports and complements actions in the Member States to improve the quality of training policy and practice through transnational projects. Close synergy has been developed with ESF operations, especially under the Employment and Adapt Initiatives. Socrates which is intended to raise teaching standards, adapt training systems to new technologies and promote life-long learning is also close to ESF Objectives.

The prevention of social exclusion, a main theme of the ESF, also forms part of the Leonardo da Vinci and Socrates objectives and has been supported by other EU programmes in this field such as the actions initiated under HELIOS to promote the integration of people with disabilities."6

In practice we found very little evidence of these admirable intentions being implemented. At a European level, and in terms of the strategic relationships between the programmes, internal Commission structures, pressures on officials, and the sheer administrative weight of managing their programmes, reduced the capacity for Commission officials to engage in inter-departmental collaboration. Despite some formal procedures (such as the sharing of national Structural Fund draft programmes with DG EAC before finalisation), the evidence of effective synergy is limited. This is not surprising: for many years this incidence of low synergy between DG EAC and the major spending departments has been pointed out as a significant weakening of the Community-level potential in the training field. In addition the implementation processes were very different – centralised for Leonardo da Vinci and decentralised for EMPLOYMENT and ADAPT - which also had a significant impact on the potential for synergy both at European and national levels. Indeed at Member State level an example of this lack of synergy can be illustrated by the experience of Sweden which had located the co-ordination of all the relevant programmes in one single office, hoping to benefit from synergies. In fact none were forthcoming and so for the new generation of programmes there will be two offices –

6 Commission website http://europa.eu.int

http://europa.eu.int

Deloitte & Touche


one for Leonardo da Vinci and SOCRATES and one for the rest, as this structure is felt to be more in keeping with the different requirements.

The only example of co-operation we identified was the joint call for multi-media projects in 1997 involving also the Community Initiatives and the Framework Programme.

In terms of duplication, this is very difficult to assess, as the mechanisms for checking are quite rudimentary. Nevertheless a system of consultation on the selection of projects does exist and the opportunity is there to signal any examples. Whether the managers of the large programmes have the capacity to identify overlap is not clear – certainly there is no mechanistic way of doing this, and it relies on the experience of those involved.

Having said this, there is no evidence of any significant overlap. Project promoters seem to be quite sophisticated in selecting appropriate funding mechanisms. No application processes are simple and a considerable investment is required from the promoters. It is not, therefore, in their interest to apply indiscriminately.

In the opening paragraphs to this chapter we observed that Leonardo da Vinci was located within a hierarchy of EU programme initiatives in the vocational training field and related sectors. The concept of a continuum between the various funding regimes is a particularly useful one when we consider the issue of complementarity. This would categorise programmes as follows:

Nature/aim of intervention Specific programme

Experimental ESF Article 10, Third System, SME Programme

Research/Innovation Leonardo da Vinci

Development/Test Marketing Community Initiatives (now EQUAL)

Mainstream implementation ESF

In each case the lessons learned from a programme should be built on in the next programme at a different scale. Continuity between programmes in fact represents a policy opportunity that, if accurately harnessed, can yield valuable conduits for the dissemination and spillover of best practice and both strategic and operational lessons. Encouraging the utilisation of conduits of information is a recognised pathway for optimising systemic knowledge management strategies within the overall vocational and educational training polity.

Recognition of the hierarchy of programmes and the opportunity for continuity and information channels also has an implication for the selection and evaluation criteria for each type of programme. Under Leonardo da Vinci, projects are supposed to be innovative and to test out new ideas. This implies an element of risk, and of freedom to take risks - the space to innovate. A conclusion that an idea does not work need not imply that the project has been a failure – it may have successfully tested an idea with the conclusion that it is not feasible to continue with the approach. In addition ideas which do not work well in one context may always be of use in different contexts. The application of this kind of learning entails well-structured and effective systems of knowledge management to be put in place.

Deloitte & Touche


4.4 Take-up of lessons by direct funding programmes There is little evidence of take up of lessons by direct funding programmes at European level. However, unless and until programmes such as Leonardo da Vinci are in a position themselves to draw lessons from their experiences it is difficult to see how mainstream programmes can be expected to do so.

Thus, if the purpose of Leonardo da Vinci is to act as a laboratory of innovation, the mechanisms need to be developed to describe in a useful and synthesised manner, the outputs of the programme and the lessons learned. This requirement needs to be built into the programme from the beginning as it affects the management, monitoring and reporting requirements and process design.

In contrast, there was some take-up of lessons reported at participant state level particularly, but not exclusively, in those states where the vocational training system was less well developed.

Deloitte & Touche


5 Implementation The Leonardo da Vinci programme is extremely complex both in its formulation and in its implementation. Many of these complexities are embedded in the 1994 Decision, while others are related to the management systems of the European Commission. A large number of stakeholders are involved, but an equally large number of entities are involved in the decision making and administration procedures.

5.1 Structures and Procedures In order to administer the programme effectively a number of organisations were set up to assist the Commission. This reflected an externalisation of a large proportion of the administrative work both to a Technical Assistance Office (TAO) in Brussels, and to National Co-ordination Units within the participating countries.

This mirrors to a large extent the systems used under the previous generation of programmes, and was a relatively common model within the Commission, not only for Programmes within the field of vocational education and training but for programmes in other Directorates General.

The TAO was set up as the result of an open call for tenders in accordance with both the European Procurement Directives and the Commission's own internal procedures. The designation of the National Co-ordination Units was the responsibility of the participating states. Despite the fact that the EU budget provided much of their funding, the Commission had no decisive power in their selection. On the other hand, the contractual relationship was between the Commission and the individual NCUs.

During the course of the Leonardo da Vinci programme there were a number of events which culminated in the closure of the TAO and the transfer of their functions to a unit within the Commission – overseeing the Cellule Leonardo (CLEO) which took over the administrative responsibilities of the TAO. In the course of this evaluation we have obviously had to take into account the factors leading to these developments and the impact of these on the operation of the Programme. However, we have attempted to distinguish those issues which related to Programme design, those which related to institutional factors at an EU or national level, and those which related specifically to the events surrounding the problems of the TAO.

It should be made clear that the underlying principles and processes of managing the projects did not change fundamentally with the change from the TAO to CLEO.

In addition to these various structures, there were two different application processes according to the different strands of the programme.

The remainder of this section looks at the various structures and their function, the processes associated with the implementation of the Programme, the risks or critical areas, the costs and benefits of the system, and finally examines the comments of the interim evaluation and the extent to which these have been taken into account.

We also comment briefly on the introduction of Leonardo 2 in the context of highlighting those issues which have already been addressed in the implementation of the new programme phase.

Deloitte & Touche


5.1.1 Technical Assistance Office The Technical Assistance office was set up following an open call for tenders at the end of 1994. It took the form of a five year service contract renewable annually, financed from A-budget lines (ie not included in the operational budget of the Programme). The contract started in June 1995, and was terminated unilaterally by the Commission in January 1999.

Under the previous programmes 7 separate Technical Assistance Offices had existed and their functions were transferred to the new TAO with the launch of the new Programme, and to the TAO for Socrates and Youth which was set up following a similar procedure. 5.1.1.1 The Contractor The Contract for the TAO was awarded to Agenor, an organisation set up specifically to manage this type of operation at the start of the FORCE programme, with shareholders from a range of organisations with an interest in the field of vocational training. The major shareholder was CESI, a major French vocational training organisation through its associates SECOF and RESOURCES SA. The remainder were DIHT (Deutscher Industrie und Handelstag), AIFPI of Greece, FPE of Spain, Sistemi Formativi Confindustria of Italy and LO Danish Confederation of Trade Unions – all organisations with an involvement in vocational training.

Agenor had previously held the contract for the Technical Assistance Office for the FORCE Programme, and thus could reasonably be expected to be competent in the administration of European Programmes of this type. 5.1.1.2 Role of TAO (terms of reference) The work programme of the TAO was set out each year in an annex to the contract. Initially the tasks fell into seven broad areas:

♦ Setting up the infrastructure for the Programme, including all the documentation for the Programme

♦ Assistance in the calls for proposals

♦ Assistance in selection of the projects according to the relevant selection process (see below)

♦ Administrative and financial management of the contracts

♦ Actions other than calls for proposals

♦ Information actions

♦ Transition (building on the experience of previous programmes), planning, monitoring and assistance with evaluation actions not performed by the TAO.

These tasks were gradually refined over time reflecting the different stages of the Programme and the maturity of experience of both the Commission and the TAO.

At the end of the TAO’s existence the work programme had developed to cover:

♦ Administration of the projects (including project selection procedures)

♦ Information

♦ Dissemination

♦ Meetings

Deloitte & Touche


♦ Information Technology

♦ Closure of contracts from the previous programmes

The work programme had a clear focus on: contracting (including payments and monitoring), preparation of information for use in the final evaluation of the previous programmes, provision of an effective tool for monitoring and management of the projects (by the IT system of the TAO), and preparation for the involvement of the pre-accession countries. 5.1.1.3 Structure of TAO The TAO had approximately 75 staff in a number of departments dealing with the various aspects of its work. This staffing complement was reinforced by external experts for processes such as the selection of projects.

Figure 5.1 The Structure of the TAO

Two criticisms that have been made of this structure are that insufficient resources were dedicated to the area of contracts and project management given the workload involved, and that the structure was not flexible enough to deal with the evolution of the programme and changing needs as it proceeded along its life-cycle.

Management

4 people

Analysis and thematic reports

5 people

Evaluation and Monitoring

9 people

NCU Network animation

6 people

Contracts and Finance

10 people

Dissemination/Virtual Products Centre

6 people

Personnel/Administration

5 people

Information and Publications

7 people

IT

4 people

Logistics and planning

4 people

Deloitte & Touche


To a great extent this is a reflection of the Commission’s contracting procedures in that in response to a need to adhere closely to initial requirements, defined before full knowledge of the situation was possible, external contractors are not able to be as flexible as necessary in the execution of contracts. This situation is not in any way unique to the Leonardo da Vinci TAO, but does illustrate a contradiction in attempting to gain additional flexibility by externalising work, but then constraining it such that the potential is lost.

The initial structure also reflects a certain lack of clarity in terms of the boundaries of responsibility between the Commission and the TAO, particularly as concerns the policy aspects of the Programme. 5.1.1.4 Budget The initial budget of the TAO was 9.1 million ECU for the first year. This gradually reduced over time, 8 MECU in the second year, 7.3 MECU in the third and 5.9 MECU for the final year. This was the budget for the operation of the TAO, all of which was claimed from the Commission on the basis of actual expenditure with the exception of a small management fee of about 100,000 ECU per annum.7

In addition to this funding from Budget line A, the TAO was responsible for the management of some 98 million ECU per annum from budget line B, being the subventions to the actual projects and which passed through a TAO bank account.

This implies that the Agenor consortium incurred costs and managed funds totalling over 100M ECU per year, employing 75 people, with only a small management fee of 100 000 ECU per annum to cover its corporate risks. This is highly unusual, to say the least. 5.1.1.5 Special circumstances, and the establishment of CLEO In the normal management procedure of the TAO, at the end of each year the final accounts were checked by the Commission before the balance of their claim for that year was paid. At the end of 1997 it became apparent that some of the experts working for the TAO, primarily in the evaluation of project applications, worked for companies which were involved in projects under the Leonardo da Vinci programme.

In order to be sure that there were no potential conflicts of interest, this information was sent by DG XXII to DG XX and UCLAF to enable a check to be carried out on this. An audit was thus undertaken in the TAO between February and April 1998 where all the project evaluators were checked against all the partners in the projects. As a result of this process 4 projects were questioned but subsequently accepted. Following this DG XX audited a selection of 30 projects and no problems were found.

During this period no progress could be made on the signature of the 1997 Projects or on the renewal of any 1995 projects, which led to a backlog of administrative work. The contracts were eventually cleared for signature in mid April (726 projects) and mid May (the remaining 4 projects).

Subsequent to this audit, allegations of irregularities arose which were investigated in an audit by DG XX and UCLAF which was undertaken in the first half of 1998. This was completed in July, and finalised in October following the responses from DG XXII. This was submitted to the Parliament at the end of December 1998.

7 Source: Commission Audit Report

Deloitte & Touche


This report recommended a number of changes to the management of the TAO which were not implemented by the contractor. Therefore, following discussions, the Commission decided not to renew the contract between the EU and Agenor SA. At the same time the Belgian Courts decided to pursue the investigation of the fraud allegations, which led to the sealing of the files held by the TAO.

This meant that there was a another period of some 6 months between January and June 1999 when no activity could take place on Leonardo projects managed centrally while the files were reconstituted and the new Cellule Leonardo da Vinci (CLEO) was established.

This Cellule was set up within the Commission and took on a proportion of the original TAO staff (although not the management staff). The structure of the CLEO was designed to focus on the processing of payments and the final rounds of project selection. The staffing of CLEO is organised as follows:

Figure 5.2: The Structure of CLEO (12/99)

In the context of this evaluation, the only question of relevance in the terms of reference of the audit report related to the promotion of the calls for proposals and the selection procedures to ensure the best possible selection of projects.

5.1.2 National Co-ordination Units In each Member State there was a National Co-ordination Unit. These bodies were nominated by the Member State and contracted to the Commission. The NCUs were co-funded from the Leonardo da Vinci budget, the rate of subvention initially varied

Co-ordination 4 people (inc Head of

Unit B.2)

Reconciliation

5 people

Content

17 people

Logistics

13 people

Finance

16 people

Contracts

8 people

Other

3 people

Deloitte & Touche


between 30 and 60% in different Member States although there has been a convergence around 50% recently. The cost of the NCUs represents some 8% of the Leonardo da Vinci budget (Budget line B only), or about 11.2 MECU per annum. NCUs were also set up in the pre-accession countries. 5.1.2.1 Structures There was no model structure for a National Co-ordination Unit. In fact, there was a single contact point only in 7 Member States, the others having several either with a geographical responsibility (eg Belgium) or a functional responsibility by strand (eg the UK). The incidence of a single NCU was much more marked in the pre-accession countries.

There was equally no single model for the form of the NCUs. These varied from private or other independent organisations (eg NGOs) contracted to provide the services, to Ministries or special inter-ministerial units set up specifically for the purpose.

In some cases the Leonardo da Vinci NCU was based with other EU Programme management units (eg ADAPT or Socrates) but in many instances this was not the case.

Despite the fact that in some instances it was difficult for newcomers to the Programme to locate the NCU in their countries, the level of satisfaction with the support from NCUs was very high in the projects we sampled. These were all projects administered by the TAO in terms of payments etc, so we cannot comment on the views of the Mobility projects managed directly by the NCUs.

Exhibit 5.1 Project Satisfaction with the NCUs

• “National co-ordinating unit was very helpful. Not much contact with central TAO. Vademecum more distant, of interest at the very initial stage” - Finnish project

• “National Co-ordinating Unit very helpful and very quick turnaround at all times. Central technical assistance unit – decision making process far too lengthy” - UK project

• “The support of our National Co-ordination Unit was fundamental for the successful development of the project – this was greatly appreciated by all of the partners” - French Project

5.1.2.2 Functions In each of the participating countries, the National Co-ordination Unit had three main functions:

♦ An information role, responsible for the promotion of the Programme within the country concerned

♦ A selection and management role for the projects selected under Procedure 1

♦ An advisory role for the projects selected under Procedure 2.

In addition they had the role of smoothing the transition from the previous generation of programmes.

Deloitte & Touche


In terms of the information role, the NCUs were crucial in order to ensure that information on the programme reached the widest possible audience, particularly all the key players in the country. Although the European calls were published in the Official Journal, this was not necessarily a source of information readily accessed by some of the key actors targeted by the Programme.

In addition to this, the NCUs provided active help and background material on the application process, which was quite complex and in most cases also provided guidance and support during project implementation. Once the programme was more established many NCUs also undertook a range of activities aimed at promoting the results of the programme. These included seminars, publications and databases on websites.

Some of the websites were quite rich in information (eg UK, Italy) while others were extremely basic.

The role of the NCUs in the selection process is described in section 5.2 below.

5.1.3 The Leonardo Committee Article 6 of the Decision provides for the establishment of a Committee composed of two representatives from each Member State, chaired by the Commission. The Social Partners also participate in the Committee as Observers. This Committee is responsible for deciding on the general guidelines for the financial support to be supplied (amounts, duration and recipients), on matters relating to the internal breakdown of the programme, the procedures for selection, follow-up, evaluation, dissemination and transfer of results.

It is also responsible for setting the priorities for the programme.

Thus the principal decisions taken by the Committee relate to the setting of the annual priorities – a not-insignificant task, given the complexity of the programme – and the approval of the proposed projects from each annual call for proposals.

With the extension of the Programme to the Pre-Accession countries, major questions need to be asked about the capacity of the Committee to guide the Programme due simply to the sheer size of the committee and its resultant unwieldiness. Even the extensive use of sub-committees does not enable the degree of flexibility which would be desirable.

We encountered much criticism of the Committee for its narrow focus on the issue of distribution of Leonardo Da Vinci resources “between” the Member States, rather than playing a useful role in guiding the programme in terms of thematic focus, linkages with other activities and so on.

5.2 Project selection and management Projects were selected and managed on the basis of two different procedures depending on the type of project. These were organised under the responsibility of either the Member States (Procedure 1) or the Commission (Procedure 2) as follows:

Deloitte & Touche


Procedure 1 Member State

Strand I (mobility)

Support for the improvement of vocational training systems and arrangement in the Member States

Commission Remainder of Strand I

Strand III.1 Development of Language Skills

Strand III.3.a Multiplier-effect projects

Procedure 2 Commission Strand II

Strand III.2

Support for the improvement of vocational training measures, including university/industry co-operation, concerning undertaking and workers.

CEDEFOP Strand III.3.b Transnational Study Visits

The projects concerned with Transnational Study Visits were administered by CEDEFOP.

In general calls subject to both selection procedures were published at the same time for maximum coverage and transparency.

5.2.1 Procedure 1 This procedure is overwhelmingly concerned with the placement and exchange projects previously funded under the PETRA programme. Under this procedure a call for proposals was organised by the Member State. The Member State would produce an operational plan for the programme which would form the basis of their budgetary envelope and the selection of the projects.

Project promoters sent an application to the National Co-ordination Unit of behalf of the participating state, which then managed an assessment of the projects and produced a list of pre-selected projects with a supporting report on the selection process. This list was then discussed with the Commission (who also received copies of the applications and assessed them), and a final draft list was produced which was submitted to the Leonardo da Vinci Committee for approval.

This procedure covered approximately 1700 projects, with states selecting between 1 and 40 projects per annum, with a maximum total number of projects being managed by a participating state of 214 over the 5 years.

Projects funded under this procedure represented approximately one third of the Programme budget.

5.2.2 Procedure 2 The remainder of the projects (some 3000 projects) were selected using procedure 2. Under this procedure the Commission published a call for proposals in the Official Journal. The call was also promoted through the NCUs and any other available

Deloitte & Touche


networks. Calls were generally published in December with a final decision taken by the Commission in November of the following year.

Under this procedure, project promoters would send copies of their applications to the TAO and to the NCU. On reception each application was evaluated following a detailed evaluation framework by Commission officers and by 2 independent external experts nominated by the participating countries. This evaluation framework was gradually refined over the life of the Programme. The applications were also evaluated by the participating countries.

The Council Decision sets out a division of budget between different thematic categories:

Criterion Minimum %

Initial vocational training 40%

Continuing vocational training 32%

Lifelong learning 13%

Pilot Projects 35%

Mobility Projects 30%

In addition to this a balance was sought between the participating states in terms of number of projects and budgets. Although no formal quotas were established per participating state, an informal system existed and formed the basis of the Leonardo Committee’s decisions.

Following the evaluation of the projects they were awarded a mark between 1 and 4, with 1 being “excellent” and 4 being “not acceptable”.

On this basis a list of proposed projects to be accepted was drawn up and communicated to the participating states. There then followed a process of bilateral negotiation where the results of the central and national evaluations were discussed and a final draft list was drawn up for presentation to the Leonardo da Vinci Committee for approval. The rate of acceptance of the projects varied from 16.49% in the first year to 35% for 1999, although overall it was in the region of 30%.

An inter-service consultation also took place within the Commission to try and ensure that there was coherence between the projects selected and projects selected under other EU Programmes managed elsewhere in the Commission. Chapter 4 has discussed the degree of success of this approach.

Once the opinion of the Leonardo da Vinci Committee was received, the written procedure was launched for the formal approval of the Commission.

Deloitte & Touche


The Timetable for Leonardo da Vinci project selection, 1999

ID Task Name1 Call for Proposals

2 Procedure I

3 Receipt, data input, analysis and preselection

4 exchange of proposals EC/NCUs

5 Lists of eligible proposals to Commission

6 Proc I eval report and preselection list to EC

7 Procedure 2

8 Receipt, input and evaluation by Commission

9 Prep of NCU report to Commission

10 Exchange of lists of proposals EC/NCUs

11 NCU Evaluation report to Commission

12 Analysis of proposals by DG XXII

13 Evaluation of Proc II proposals by experts

14 Evaluation of Proc I andII eval reports by DG XXII

15 Selection of proposals

16 Prep of f iles for bilateral meetings

17 Meeting of Leonardo Committee

18 Bilateral Meetings

19 Establishment of draf t selection list

20 Interdepartmental consultation on the list

21 Draft list to Leonardo Committee

22 Meeting of Leonardo Committee

23 Written Procedure in the Commission on list

24 Decision by the Commission on the selection list

25 Contract negotiation

12/05

21/05

21/05

19/10

24/11

30/1107/1214/1221/1228/1204/0111/0118/0125/0101/0208/0215/0222/0201/0308/0315/0322/0329/0305/0412/0419/0426/0403/0510/0517/0524/0531/0507/0614/0621/0628/0605/0712/0719/0726/0702/0809/0816/0823/0830/0806/0913/0920/0927/0904/1011/1018/1025/1001/1108/1115/1122/1129/1106/1213/1220/1227/1203/0110/0117/0124/0131/0107/0214/0221/0228/December January February March April May June July August September October November December January February Ma

Deloitte & Touche


Thus the selection process took somewhat over 8 months, with contracting procedures taking a further 3-4 months with up to 7 months being not unknown. By way of comparison, the procedure for selection of projects under the Fifth Framework Programme takes 3-4 months (including 4-6 weeks for Committee opinion) with a further 6 weeks to 4 months for the contracting process. As with Leonardo da Vinci, some aspects of the timing on the contracting process lie in the hands of the projects themselves.

Looking at the timetable shown above, it suggests that by altering the timing of the call slightly, it might be possible to avoid the two-month hiatus over the summer period.

5.2.3 Contracting and payments Once the list of projects was approved the successful promoters would be informed and the contracting process would begin. In the first four years of the programme it was normal for projects to be accepted but for the budget allocation approved to be significantly lower than that originally proposed. Thus the "contractualisation" process often involved a renegotiation of the project and may have led to quite significant changes in the project’s work programme. This practice was criticised by the auditors, and had adverse effects in that it was not unknown for projects to inflate their budgets artificially to take account of the expected reduction. In the final year of the Programme a decision was taken to fund projects 100% of the requested budget, except in cases where the budget was clearly shown to be overstated in the evaluation process, or on the specific advice of the participating state. Although there was an intention to complete the contractualisation process before the end of the calendar year (see flow chart above), it was often difficult for the projects to complete the complex forms and budget revisions in time for this to take place and normally contracts would be signed in the first quarter of the following year. In the case of 1998 and 1999 projects the situation was further complicated by the events surrounding the TAO.

Projects that were not successful did not receive any feedback as to why their project was not accepted.

Exhibit 5.2 Delayed Start Dates For The Projects

• “Lots of beneficiaries mention the fact that they have only received the first instalment/payment. The missing money in many cases blocks the project.” (German project)

• “The decision about the grant arrived far too late, the written decision arrived 7 months after the beginning of the first project phase”.(Austrian project)

This lengthy process had two major problems – firstly it was not uncommon for projects to find that situations had changed significantly between the design of the proposal and the point where the project could start. Secondly, a practice, much criticised in the audit reports, of backdating contracts, sometimes by as much as 7 months, had grown up. This made life extremely difficult for projects who faced a decision between starting the project before the contract was issued (thereby incurring significant risk), or of starting after signature of the contract, but being unable to meet the timetable.

Deloitte & Touche


Each project would then receive an advance payment. The final payment would be made on the basis of the final report and detailed financial statements. In the case of projects with a duration of more than 12 months, an interim payment was also payable based on an interim report and financial statement.

These reports and financial statements were the subject of close scrutiny and evaluation, and detailed reports on them were drawn up by the TAO.

5.2.4 Follow-up and animation This is acknowledged as one of the weakest areas in the central administration of the Programme. To some degree this is a result of the events surrounding the closure of the TAO in that the resource to undertake much of this work was lost just at the time when projects were coming to fruition.

Paradoxically, the establishment of CLEO has given the possibility to maintain a Unit dealing with the administration of the projects beyond the paper life of the overall programme. Technical assistance offices normally end with the end of the Programme leaving the closure and dissemination phases to be undertaken as an incidental activity by the TAO for the successor programme. This situation seems unlikely to change in the light of current discussions on the future of TAOs.

Contact, other than purely administrative, was not maintained with the projects during their life-times, leading to a feeling of isolation reported by our sample.

In the participating states, and even between groups of states (eg the Nordic countries) a number of animation activities were undertaken, but this was not consistent across the countries and was not drawn together at European level in any meaningful way.

5.2.5 Strand III3b – Study Visits Strand III3b of the Leonardo programme – the programme of Community study visits for vocational training professionals – is handled separately from the Procedures 1 and 2 above, being managed by CEDEFOP, the European Centre for the Development of Vocational Training. The Community study visits programme pre-dated Leonardo by some 10 years, with the aim of activating information flows and exchange of experience amongst vocational training specialists of Member States and associated countries. The study visit programme is funded by means of a ring-fenced allocation in the Commission’s annual grant to CEDEFOP. This money was not part of the Leonardo budget.

Selection of participants for the study visits is the responsibility of National Liaison Officers, civil servants nominated by their governments to work with CEDEFOP on the implementation of guidelines and organisational procedures.

CEDEFOP, as a Community body established by a Council of Ministers’ decision of 1975, has a formal role as a source of information and centre of reference in contributing to the development of vocational training in the European Union. CEDEFOP does not have a formal role in respect of the Leonardo Programme’s operations or pedagogical objectives beyond strand III3b.

Deloitte & Touche


5.3 Problem Areas

5.3.1 Structural The structure of the programme was extremely complex. This caused problems for potential promoters in identifying which procedure, which strand was relevant for their project. This situation has not been improved by the supporting documentation which was also extremely hard to follow, especially for project promoters working in languages other than their first language.

Some of the NCUs produced additional supporting information which made the procedures much clearer, but in general it must be said that the documentation was not “user friendly”. This need for clearer documentation was repeatedly expressed in our survey.

There is a need for project promoters to build partnerships when drawing up proposals. However, in the case of the pilot projects, very little assistance was given in this process other than the partner search database (which came on stream quite late in the life of the programme) and the annual fair (“contact and information days”) in Brussels. An active role could be played by the NCUs in exchanging information on potential partners. The current system is biased in favour of applicant organisations which have existing networks (not that these should be discouraged) but makes the development of new and sustainable partnerships comparatively difficult.

There is no real mechanism either at European or national level that is specifically designed to take forward the lessons from the Programme. Activities have taken place in some participating states but this has been lacking in co-ordination at a European level. This relates not only to assistance/insistence on the diffusion or commercialisation of products, but more generally to liaison with the wider constituency of business, government and trade unions to mainstream the valuable learning outcomes.

This lack of any real strategy at any level, from project to the overall programme, to use the outputs of the programme is a serious weakness and, in terms of the long term impact of the programme, probably the major one.

The administrative burden of the programme has meant that the negative aspects associated with it have become the dominant factor, rather than a facilitating process. It is not an exaggeration to state that many potential applicants for EU support are not prepared to endure the delays and bureaucracy that the current systems display, and are thus refusing to participate in the future.

5.3.2 Financial The financial circuits of the programme for the centrally administered projects were immensely long. Dossiers had to be verified by officials in three different departments prior to any payment being made (TAO, DG XXII, and DG XX). This meant that the time taken to pay claims was extremely long, even without the additional delays caused by the investigations into the TAO.

Exhibit 5.3 Payment delays

• “We have been waiting for the outstanding payments since the end of the project. We do not understand the continued absence of written replies about this matter in spite of the countless written requests we have made” (Italian project)

Deloitte & Touche


• “Due to loony periods of bureaucratic work in Brussels we have had an 8 month delay! The interim report was not evaluated in accordance with the agreement (60 days) That constitutes a breach on the Commission’s side (it took 4 ½ months)” (Danish project)

• “Immense delays, through excessive bureaucratic procedures, especially payments. Unfulfilled promises of payment dates only appears to be in the interest of financial aspects, little if any in the technical aspects” (UK project)

The length of time taken to pay claims is extremely important as it has a strong influence on the type of organisation which is able to act as a project promoter. SMEs and non-profit organisations often simply do not have the cash flow to withstand this type of problem, and thus improving their participation rate in the programme will be extremely difficult until this issue is resolved.

Over the life of the Programme approximately 11500 payments had to be processed centrally. Currently CLEO has a target of clearing 300 dossiers per month whereas the TAO was managing to deal with approximately 100. It should be noted that payments will need to be processed until December 2001.

Despite the fact that CLEO has achieved this level of processing, there are still two next steps of the process – the verification and payment by the financial services departments of DG Education and Culture and DG Financial Control. There has been much internal debate between the services concerned. However, it seems that now some progress has been made on reducing the backlog although there appears to be a level of about 200 payments which is unlikely to be reduced given the resources available. This suggests forcefully that the designers of programmes should have a more realistic understanding of the resource implications of implementation, and that processes should be designed to be as streamlined as possible.

Payments to the projects initially passed through the TAO. This added an extra stage to the process. The TAO was thus charged with making payments to bodies with which it had no direct contractual relationship. Further, questions were raised during the audit as to whether the money advanced to the TAO was being managed in the most productive manner. These issues were addressed following the audit report and substantial changes were made. This situation (of allocating funds to the TAO to manage) obviously no longer applies. It is doubtful whether it would apply to any future technical assistance regime.

The whole co-funding regime was extremely onerous, requiring minute details in justification of expenditures which were not always easily identifiable (eg additional salary costs). In addition the involvement of the pre-accession countries complicated matters in terms of financial reporting as it was necessary to account separately for all related expenditure. We understand that steps have been taken in Leonardo 2 to simplify the co-funding regime to attempt to reduce this problem. In addition, advances in the enlargement process have reduced the administrative burden of dealing with the pre-accession countries.

5.3.3 Procedural In terms of the procedures, the main areas of weakness seem to lie in the following areas:

Deloitte & Touche


The Selection Process By working with the NCUs it seems that information on the calls reaches a wide range of operators and that the information is easy to access. Having received the information, however, the complexity of the application process may discourage some potential participants, especially those who have not previously been involved in EU programmes.

The main concern however, lies in the fact that it took (in cases where no special circumstances intervened) a period of almost one year between the publication of the call and the notification of the successful projects. During this time many changes can take place in the circumstances of the project/partnership, and what was innovative at the time of the proposal may be less relevant at the end of the process. While there is clearly a need for a transparent evaluation process, the degree of consultation and negotiation involved in this case led to this extremely extended timetable.

This was made worse in the eyes of the projects by the fact that no feedback was received, either during or after the process. It may be worth noting that in the case of the Fifth Framework Programme calls, attempts have been made to introduce a process of rapid response so that applicants are informed in a short time period whether their application has passed the eligibility criteria and is going forward for consideration.

Finally the practice of backdating contracts is not conducive to the proper implementation of projects.

Bilateral negotiations The outcomes of negotiation in the bilateral meetings meant that although projects were initially selected on the basis of objective criteria, the ensuing discussion introduced a number of other factors more related to national priorities. In some cases this may have added clarification to the expert analysis but the process moved away from the principle of total transparency, introducing a set of non-documented criteria.

Category 4 projects An extreme example of this is the case of the Category 4 projects. As described above, the evaluation process graded projects between 1 and 4, with 4 being classed as “unacceptable”. However, in the course of the bilateral negotiations a number of these projects were selected for inclusion on the final list. This clearly casts doubt on the process of independent evaluation and was criticised in the audit report. These projects may have been selected for their strategic importance, but this is not clear, and leaves the whole selection process open to criticism. Should the evaluators refrain from grading any projects as type 4 to avoid this criticism, the whole process would be undermined.

Reduction of budgets Once projects were selected, it was unusual in the 1995-1998 cycles for the full budget requested to be allocated to the project. This then resulted in a process of negotiation which could mean that the project as implemented was substantially different from the project as evaluated – not a desirable situation. The alternative of trying to implement the project as proposed but with reduced resources is equally undesirable. A third negative effect of this practice is that in subsequent applications projects deliberately overstate the financial requirements as they expect to be reduced if successful, leading to an elaborate game and wasting time and resources.

Deloitte & Touche


For the 1999 call this process was changed with only those projects with clearly inflated budgets having their allocations reduced.

Lack of feedback and contact The problem of lack of feedback and contact mentioned above in the selection process was equally an issue for projects administered by the TAO. It is our understanding that this was less of an issue for the decentralised projects, which is understandable as the scale of the management task was much smaller.

Reporting Although the projects produce very detailed final reports, and these are extensively analysed, this analysis is undertaken in a very mechanical way and there is no mechanism for recording or using the results of this analysis. This is discussed in more detail below.

5.3.4 Institutional Commission ability to process payments Quite apart from the capacity of the TAO/CLEO to process the payment and contracting elements of their tasks, serious questions also arise about the capacity of the Commission itself to process the number of payments involved within a reasonable timescale. This issue relates to both the resources of the Commission and the internal processes which are currently acknowledged to be long and slow.

The structures for programme administration need to be designed with the capacity of the Commission itself in mind to avoid the creation of bottlenecks. We have seen examples of other programmes where the reporting and verification requirements are much less cumbersome, while remaining open and transparent.

Responsibility of Member States/Commission The division of responsibility between the Commission and the participating states is the cause of several limitations in the programme implementation. Apart from the cumbersome consultation procedures, at a practical level it has led to a series of weaknesses, for example:

♦ Data is not consolidated and held centrally. This makes it difficult to analyse and build on the results of the decentralised projects. This weakness was compounded by the lack of a well structured data management system at the centre in which to keep this data, had it been collected.

♦ While some interesting work was done by the NCUs in terms of animation and dissemination, this was not co-ordinated or built on in any significant way. There was no feeling of an implementation partnership, but more of a separation of powers. This meant that many interesting opportunities were lost.

5.4 Costs, benefits and value for money In total, and not including the costs of the Commission personnel involved in programme management and financial control, the EU contribution to the administration of the Leonardo da Vinci programme came to nearly 20 MECU per annum, which represents over 13% of the Programme budget.

Deloitte & Touche


According to the DG XXII Audit report of the TAO, the average cost of a person at the TAO was 83 000 € compared to 108 000 € for a Commission official8. Thus, in principle, the externalisation of the programme management represented a significant saving for the Commission.

One can analyse the programme in terms of cost per participant, or other yardsticks. However, it was not possible for us to make a detailed analysis of the outputs (see chapter 6 below). However, the cost effectiveness of producing something that is not exploited beyond the actual project has to be estimated as low.

5.5 The Interim Evaluation The interim evaluation9 made a number of comments on the selection process, on monitoring and evaluation and on the overall management of the programme. These remained largely unaddressed in the remainder of the programme, and indeed in some cases the problem was intensified rather than resolved. While this was often due to the extraordinary circumstances surrounding the programme, nevertheless there is little evidence of any attempt to take the recommendations of the evaluation on board.

The evaluation made a number of recommendations regarding dissemination, highlighting already that this was a weak point in the programme, but for various reasons, including crisis management and resource shortages, these were not implemented.

8 SEC(1999)881 Vade Mecum Relatif aux Bureaux d’Assistance Technique 9 First external evaluation of the Action Programme for the implementation of a European Community Vocational Training Policy, Ecotec Research and Consulting Ltd, March 1997

Deloitte & Touche


6 Outputs

6.1 What were the outputs This evaluation uses the term ‘outputs’ to describe the activities that took place in Leonardo da Vinci, where, when, involving whom, and at what cost. This information is presented below in tabular form. In addition to the activities listed, it considers those outputs described as ‘products’ with particular respect to the pilot projects.

6.2 Outputs recorded centrally The majority of the tables relate only to the data held centrally – thus do not include information on projects operated under procedure 1. This information is reviewed in 6.3. Final data on the 1999 procedure 1 projects is not yet available, as the projects have only just finished and the reporting cycle is not yet complete. Over the period 1995 to 1999, some 3745 projects were financed, at a total cost to the EU of €487.69 million. A further €151.04 million was spent on placements and exchanges administered by the participating countries, and €87.78 million on support and other measures (including the NCUs), giving an overall total of €726.7 million. The total investment in Leonardo da Vinci, including the project participants’ own contributions, was € 801,190,597. Thus the EU contribution represented approximately 66%. In total the projects involved approximately 3000 contracting organisations. Overall involvement in the partnerships totalled 77,274 organisations altogether. This figure includes some duplication (especially in the case of the mobility projects where organisations may figure as potential hosts in several projects) but is nonetheless impressive.

In terms of priorities, it is difficult to present the data, as there was a change in priorities between 1995 and the remaining years. However, figure 6.1 below shows the breakdown for the years 1996-1999.

Deloitte & Touche


Figure 6.1: Number of Projects by Priority

0

50

100

150

200

250

300

350

1 Acquisition ofnew skills

2 Closer linksbetween

education andtraining

establishmentsand enterprises

3 Combatingexclusion

4 Investment inHuman Resources

5 Use ofinformation toolsand technologies

96979899

In terms of the breakdown by strand10, the situation for 1995-1999 was as follows:

10 see end of chapter for a list of the strands

Figure 6.2: Distribution of projects by strand

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

I.1.1.a I.1.1.b I.1.1.c I.1.1.d I.1.1.e II.1.1.a II.1.1.b II.1.1.c II.1.1.d II.1.2.a II.1.2.b II.1.2.c III.1.a III.1.b III.2.a III.3.a

% of Budget% of Projects

Deloitte & Touche


Strand II.1.2.a – in-company training placements for university students and young graduates had the largest number of projects (and budget), although the pilot projects on training to cope with technological change and investment in continuing vocational training for workers also represented a significant proportion of the resources. The detailed expenditure by strand is shown in table 6.2.1 below.

In terms of distribution by country, a broad pattern is clear, with France leading the largest number of projects. However, there is some variation between years, indicating that there was at least marginal flexibility in the distribution of projects:

Figure 6.3: Distribution by country by number of projects

0%2%4%6%8%

10%12%14%16%18%

Belgium

Denmark

German

y

Greece

Spain

France

Irelan

dIta

ly

Luxe

mbour

Netherl

andAus

tria

Portug

al

Finlan

d

Sweden

United

King

doIce

land

Liech

tenste

i

Norway

Cyprus

Czech

Rep

ubEsto

nia

Hunga

ry

Litua

niaLa

tvia

Poland

Roman

ia

Slovak

Rep

ubBulg

aria

Sloven

ia

19951996199719981999

Deloitte & Touche


A similar representation on the basis of budget shows that although France had consistently the most projects, the greatest budget went to Germany, with France, Italy and the UK at similar levels following them.

Figure 6.4: Division of budget by country by year

0%

2%

4%

6%

8%

10%

12%

14%

16%

Belgium

Denmark

German

y

Greece

Spain

France

Irelan

dIta

ly

Luxe

mbourg

Netherl

ands

Austria

Portug

al

Finlan

d

Sweden

United

King

dom

Icelan

d

Liech

tenste

in

Norway

Cyprus

Czech

Rep

ublic

Estonia

Hunga

ry

Litua

niaLa

tvia

Poland

Roman

ia

Slovak

Rep

ublic

1995199619971998

Deloitte & Touche


TABLE 6.1 MEASURES GLOBAL

1995 1996 1997 1998 1999 TOTAL

Measures Number of Projects

LEONARDO Grants (M€)

Number of Projects


Number of Projects


Number of Projects


Number of Projects


Number of Projects


I.1.1.a 93 10.90 79 11.84 71 10.22 83 12.64 82 8.82 408 54.42

I.1.1.b 58 6.55 60 9.24 51 7.60 69 9.07 57 5.24 295 37.71

I.1.1.c 20 2.02 21 2.47 22 2.89 25 3.34 18 1.74 106 12.46

I.1.1.d 18 1.99 14 2.09 12 1.80 18 2.47 6 0.67 68 9.02

I.1.1.e 38 4.26 56 7.83 41 6.25 47 6.86 40 4.45 222 29.65

II.1.1.a 127 14.31 103 14.90 78 11.94 71 11.66 45 4.58 424 57.39

II.1.1.b 81 9.72 110 16.67 88 13.00 83 11.79 56 6.21 418 57.39

II.1.1.c 57 8.49 69 10.13 45 7.07 38 6.60 35 3.20 244 35.49

II.1.1.d 20 2.33 19 2.74 12 1.71 13 2.20 12 1.14 76 10.12

II.1.2.a 102 13.75 144 20.59 150 23.11 137 16.59 180 23.58 713 97.62

II.1.2.b 11 0.38 11 0.37 21 0.74 13 0.27 21 0.63 77 2.38

II.1.2.c 5 0.10 12 0.46 13 0.45 11 0.43 14 0.31 55 1.75

III.1.a 43 5.05 30 4.39 38 5.79 40 5.39 23 2.63 174 23.25

III.1.b 3 0.10 3 0.03 4 0.32 1 0.00 2 0.10 13 0.56

III.2.a 47 6.85 41 6.03 47 7.92 35 5.42 0 0.00 170 26.22

III.3.a 26 2.86 21 2.34 37 4.54 37 4.19 161 18.37 282 32.30

TOTAL 749 89.66 793 112.14 730 109.22 721 94.24 752 81.67 3745 487.69

Source: European Commission, DG EAC

Deloitte & Touche


TABLE 6.2 COUNTRIES GLOBAL (all amounts in €) Countries 199511 1996 1997 1998 1999 TOTAL

Number of Projects

LEONARDO Grants

Number of Projects

LEONARDO Grants

Number of Projects

LEONARDO Grants

Number of Projects

LEONARDO Grants

Number of Projects

LEONARDO Grants

Number of Projects

LEONARDO Grants

Belgium 31 3,358,755 24 3,649,178 26 3,810,651 28 3,820,684 25 2,709,605 134 17,348,873 Denmark 29 2,734,395 20 2,381,300 16 2,154,060 16 1,999,042 14 1,804,304 95 11,073,101 Germany 143 16,168,646 103 16,822,573 86 15,818,667 85 12,290,358 81 12,899,744 498 73,999,988 Greece 59 4,993,941 35 5,157,900 28 4,523,570 22 3,349,009 26 3,595,110 170 21,619,530 Spain 112 10,228,773 86 11,214,134 82 10,583,897 70 8,690,802 76 8,864,066 426 49,581,672 France 151 13,687,112 117 14,521,429 113 14,034,766 100 11,097,282 104 11,172,558 585 64,513,147 Ireland 32 3,331,803 27 3,617,458 21 2,939,157 18 2,194,438 16 2,313,430 114 14,396,286 Italy 137 13,724,022 105 14,308,125 94 13,022,925 81 10,701,513 82 10,518,441 499 62,275,026 Luxembourg 11 1,448,200 8 1,237,850 7 1,246,103 7 882,337 4 590,626 37 5,405,116 Netherlands 40 4,412,364 25 4,463,321 19 3,747,813 19 3,103,405 22 3,285,593 125 19,012,496 Austria 30 3,078,035 20 3,220,000 20 3,912,653 22 2,884,610 20 2,951,394 112 16,046,692 Portugal 45 4,004,224 27 3,985,925 22 3,768,937 20 2,648,775 24 3,115,464 138 17,523,325 Finland 32 3,066,287 20 3,117,673 19 3,115,539 18 2,517,726 25 2,448,373 114 14,265,598 Sweden 30 3,037,980 23 3,191,798 18 2,944,805 17 2,378,410 16 2,300,256 104 13,853,249 United Kingdom 138 14,015,851 105 14,311,789 87 13,054,088 81 10,795,192 72 10,641,608 483 62,818,528 Iceland 7 750,925 5 626,398 7 814,426 5 513,365 5 543,213 29 3,248,327 Liechtenstein 2 210,000 1 170,000 1 154,000 1 160,000 1 100,000 6 794,000 Norway 21 2,100,542 15 2,190,165 15 2,255,210 11 1,525,021 9 1,072,837 71 9,143,775 Bulgaria 9 1,114,429 9 1,114,429 Cyprus 4 613,311 5 431,282 2 265,050 11 1,309,643 Czech Republic 2 343,600 19 2,078,612 18 1,548,474 39 3,970,686 Estonia 4 467,751 2 296,226 6 763,977 Hungary 14 2,492,406 14 1,422,995 15 1,448,365 43 5,363,766 Lithuania 6 808,774 10 775,409 16 1,584,183 Latvia 3 520,900 5 598,150 8 1,119,050 Poland 17 2,747,663 22 2,875,780 39 5,623,443 Romania 2 273,600 25 3,118,764 26 2,589,492 53 5,981,856 Slovenia 5 509,495 5 509,495 Slovak Republic 14 1,755,505 17 1,638,593 31 3,394,098 EUR projects 24 2,323,010 27 3,890,806 27 3,446,259 0 0 0 0 78 9,660,075 TOTAL 1,074 106,674,865 793 112,077,822 730 109,070,443 728 94,904,215 753 94,586,085 4,078 517,313,430

11 Includes 1995R project renewals

Deloitte & Touche


TABLE 6.3 Number of beneficiaries for the selected projects by country and by strand

Country Strand I Strand II Strand III Total

B 2.059 1.059 3.118 DK 789 559 1.348 D 16.434 7.103 164 23.701 EL 1.045 1.016 2.061 E 4.525 4.214 8.739 F 6.381 7.108 13.489 IRL 656 1.498 2.154 I 6.743 5.058 4 11.805 L 89 263 352 NL 1.425 2.012 3.437 P 1.518 1.299 2.817 UK 6.550 4.351 37 10.938 S 1.042 1.049 2.091 A 1.521 1.595 3.116 FIN 864 1.270 2.134 N 995 585 1.580 IS 224 255 479 FL 13 13 RO 819 175 994 HU 357 107 464 CZ 464 40 504 CY - 12 12 PL 480 61 541 SK - 122 122 EE 69 69 LT 154 45 199 LV 255 17 272 BG - - SI - - EUR – Projects - 282 282 TOTAL 54.513 40.616 205 95.334

For Strand I : only figures for 1995 – 1998

II and III :figures for 1995 - 1999

Source: European Commission, DG EAC

Deloitte & Touche


TABLE 6.4 Number of selected projects by country and by strand

Country Strand I Strand II Strand III Total B 200 23 1 224 DK 116 21 137 D 1.940 99 4 2.043 EL 71 17 88 E 373 11 484 F 471 171 2 644 IRL 92 32 124 I 289 94 1 384 L 18 4 22 NL 116 22 138 P 173 28 201 UK 404 75 5 484 S 117 20 137 A 123 23 146 FIN 120 33 153 N 132 15 147 IS 67 8 75 FL 1 1 RO 75 15 90 HU 37 7 44 CZ 59 4 63 CY - 1 1 PL - 1 1 SK 16 4 20 EE 8 8 LT 18 4 22 LV 53 1 54 BG - SI - - EUR – Projects - 13 13 TOTAL 4.994 622 13 5.829 For Strand I : only figures for 1995 – 1998 II and III :figures for 1995 – 1999 Source: European Commission, DG EAC

Deloitte & Touche


Figure 6.5 Contractors by type of organisation

27%

18%

9%

7%

7%

5%

4%

4%

3%

3%

3%

2%

2%2%

2% 2%

1%

1%

Training organisation

U niversity

U niversity enterprise training partnership

O ther organisations

Public authorit ies

Small and medium sized enterprise ( less than 2 50workers)Research centres or institutes

Professional organisations/ federations/ groupings

Chamber of commerce, industry, agriculture

Trade union organisation

Employer organisation

Large enterprise (2 50 workers and more)

Group or association of companies

Joint body

Regional consortium

Sectoral organisation

European organisations

O rganisations concerned with cert ification andrecognit ion of qualifications

Deloitte & Touche


6.3 Outputs recorded at national level For the period of 1995-199912, placement and exchange projects under Strand I were co-financed with an amount of €162.49 million (€61.36m for young people, €83.74m for young workers and €17.38m for instructors). The beneficiaries were 51,047 young people, 21,486 young workers and 10,750 trainers, giving a total of 83,283 participants from about 6,516 projects.

Under Strand II young graduates, staff from universities or enterprises and inter-company exchanges, received €105.10 million in co-financing, for 822 projects involving 40,616 beneficiaries.

Strand III for language instructors involved only 205 beneficiaries from 13 projects at a cost of €0.99 million.

Thus in total nearly 125,000 beneficiaries participated in these mobility actions, at a cost of some €270 million. More than a third (35%) of these projects were led from Germany – 1243 out of a total of 5829, with France and the UK having the next highest proportion at 644 and 484 respectively.

No information was available on where the participants undertook their placements, although this should be available from the national reports.

However, although the NCUs have data on participants leaving their country and destinations, they have no information on participants arriving in their countries from 12 1999 figures are estimates subject to confirmation

Figure 6.6 Type of organisation as partnerSmall and medium sized enterprise (< 250workers)University

University enterprise training partnership

Training organisation

Large enterprise (250 workers and more)

Other organisations

Public authorities

Research centres or institutes

Professional organisations/ federations/ groupings

Chamber of commerce, industry, agriculture

Group or association of companies

Trade union organisation

Employer organisation

Regional consortium

Sectoral organisation

Joint body

Organisations concerned with certification andrecognition of qualifications

Deloitte & Touche


elsewhere – a useful piece of information for which there were apparently no central collection facilities foreseen.

6.4 Quality of outputs The emphasis on detailed quantitative reporting at the European level appears to have resulted in downplaying systematic appraisal of the quality of the Leonardo da Vinci processes (managerial/networking etc); of the personal experiences (placement and exchange); or of the products (pilot projects). The evidence gathered from questionnaire responses and from structured interviews suggests, however:

• a significant level of satisfaction amongst promoters with the opportunities provided by Leonardo da Vinci for development of project management skills, establishment of networks and information exchange at the European level;

• a significant level of satisfaction amongst participants, irrespective of age, in the personal development opportunities afforded by the mobility projects;

• a feeling amongst promoters of pilot projects that the quality and utility of their products was neglected by Brussels, which paid far more attention to administrative and procedural considerations.

6.5 Products - Dissemination From the launch of the Leonardo da Vinci programme emphasis was placed by the Commission on the importance of dissemination of those outputs described as products. Each pilot project was required to specify anticipated products of the training /development activity in the framework of a dissemination plan. There was no definitive ex-ante classification of products, but Commission guidance provided in the course of Leonardo da Vinci 1 suggested that the term “can be taken to mean the tangible outcomes of projects funded by the European Community. Examples are training manuals, training course curricula, training software etc., or surveys and analyses funded as research”. (In the terminology of this evaluation the term 'outcomes' is used to refer to benefits accruing to a target group or groups as a result of intervention; the tangibles are termed 'outputs' as noted at the beginning of this section.)

The guidance (produced in 1997) went on to offer pointers regarding issues concerning the commercial exploitation of products. The Commission view was that dissemination also applied to the processes involved in pilot projects – managerial lessons learned, be they positive or negative (in the spirit of the programme as a 'laboratory of innovation'). Equally, in mobility projects, the guidance considered that there were 'experiences, methodologies and lessons' worthy of dissemination. Thus each type of project would be expected to have some tangible message to relay to the outside world.

Methods of dissemination suggested by the Commission included:

• press

• newsletters

• publications

• fairs and exhibitions

Deloitte & Touche


• conferences and seminars

• internet

Such dissemination was to be supported by the Commission's own communications strategy, in association with the central TAO, including:

• databases of past projects, products and results which 'can be used by promoters past, present and future, to see what other work is being done that connects with their work, thereby creating the potential for links between projects and avoiding repetition. For example, the Leonardo da Vinci Technical Assistance Office has developed a database of products from past programmes (ie PETRA, COMETT, FORCE, EUROTECNET and LINGUA) which can be consulted at the Multimedia Centre”;

• product fairs at a European level;

• a Multimedia Centre, established by the TAO “where visitors can look at products developed under previous programmes (and, when the projects are completed and products received, also from the Leonardo da Vinci programme), and examine the pedagogical, methodological and technical aspects of product development”.

The Multimedia Centre was able to take account of the tendency amongst promoters to change their product media in the wake of fast-moving ICT developments: initial emphasis on audio and videocassettes for example, gave way to proposals for CD-ROM products. Expense and quality issues relating to the latter led to the later interest in the development of web sites.

Problem Areas The Commission's ambitions with respect to the development, classification and dissemination of Leonardo da Vinci products appear to have been frustrated by the programme's emphasis on managerialism and unnecessary bureaucracy and by the collapse of the Technical Assistance Office. Survey respondents have commented, for example:

'Nobody seems interested in the quality of the work' (private sector promoter, UK)

'The administrative demands detract from matters of substance' (professional association promoter, Denmark)

'Only appears to be interest in the finance aspects; little, if any in the technical aspects' (private sector promoter UK)

'Dissemination of results by the Leonardo da Vinci administration is very inadequate' (training organisation promoter, Spain)

The Commission's own Leonardo da Vinci communications strategy was interlinked with the work of the collapsed TAO, so that, for example:

• the envisaged product databases are not available to visitors;

• the multimedia centre no longer exists;

• for those Leonardo da Vinci projects which have finished, product evaluation is limited to verifying technical conformity with contract;

Deloitte & Touche


• there is no complete catalogue/database of Leonardo da Vinci products so far;

• there has been only one product fair at European level. A second, planned for January 2000, was cancelled because 'a suitable room' could not be found in Brussels.

CLEO, the successor technical assistance unit, is under pressure to resolve the substantial remaining administrative backlog in relation to projects and does not have the resources to continue work on product classification and dissemination. Earlier initiatives, such as one to publish up to 20 examples of good practice based upon research and analysis projects, have had to be abandoned. As a result of their work on evaluation/verification of final reports CLEO staff know broadly what exists in respect of outputs from finished projects and take the view that there are high quality products amongst them. It appears that there is, however, neither the operational priority nor the organisational resource to do anything about them.

The effects As it is, the Leonardo da Vinci programme appears to have lacked (or lost):

• a Europe wide approach to the dissemination of products

• a catalogue of products, even as a simple list

• the provision of good-practice case material and guidance

• high profile marketing events

• a realisation of the growing political importance of vocational training particularly in respect of the EU employment strategy

The loss of leadership and resource in classification and dissemination of Leonardo da Vinci products at a European level has further adversely affected the programme's credibility amongst promoters and NCUs.

Perhaps more important, it could diminish the authority and capacity of the programme to inform and influence the development of Vocational Education and Training policy at the EU level at a time when vocational training is recognised more and more as a crucial input to the Union's socio-economic policies, particularly in respect of the maintenance and growth of employment. It is no exaggeration to state that not having even a basic directory of the products and outputs of a programme that absorbed over €500 million of public money is a completely unacceptable lapse.

The need In order to help re-establish the professional potential of the Leonardo da Vinci programme there is a need for the Commission to lead a new initiative, perhaps in conjunction with CLEO, CEDEFOP, the NCUs and the Leonardo da Vinci networks, to produce a capitalisation strategy for the products of both Leonardo da Vinci 1 and II. This initiative could include

• a properly segmented marketing plan at a European level;

• a comprehensive, inventory of Leonardo da Vinci products, classified by category;

Deloitte & Touche


• a resumption of work on good practice models and guidance;

• a re-instatement of the European Leonardo da Vinci product fair;

• a programme of promotional events, including a joint Commission/Parliamentary seminar to emphasise the political importance of Vocational Education and Training.

Given the obvious interest in vehicles like the CEDEFOP Electronic Training Village (over 7000 registered residents to date) and the new initiative for a European Learning Gateway, it would be logical to ensure that the fuller development of these tools make provision for an effective capitalisation of Leonardo’s results.

Deloitte & Touche


Leonardo da Vinci 1 – The strands

Strand 1

support for the improvement of vocational training systems and arrangements in the Member States

I.1.1.a Projects to improve initial training systems

I.1.1.b Projects to improve continuing training systems

I.1.1.c Projects concerning vocational information and guidance

I.1.1.d Projects to promote equal opportunities

I.1.1.e Projects to combat exclusion

I.1.2.a Placement programmes for young people in initial vocational training

I.1.2.b Placement programmes for young workers

I.1.2.c Exchange programmes for training programme instructors, planners and administrators.

Strand 2

support for the improvement of vocational training measures, including university/enterprise co-operation, concerning enterprises and workers

II.1.1.a Projects concerning innovation in vocational training for workers

II.1.1.b Projects concerning investment in continuing training content and methods

II.1.1.c Projects concerning industry/university co-operation

II.1.1.d Projects to promote equal opportunities

II.1.2.a In-plant training placement programmes

II.1.2.b/c Exchange programmes

Strand 3

support for the development of language skills, knowledge and the dissemination of innovation in the field of vocational training

III.1.a Language training projects

III.1.b Projects for the exchange of language teachers

III.2.a Survey and analysis projects

III.2.b Exchange of comparable data

III.3.a Multiplier projects for the dissemination of prior results

III.3.b Programme of study visits

Deloitte & Touche


7 Impact

7.1 Approach In this chapter we outline the impact of the Leonardo programme. The chapter examines impact at each of the different levels of the project: European, Member State, projects, and participants. In addition, the impact of Leonardo is discussed for different target groups (women, social partners, SMEs, marginalised groups).

We can employ a number of different approaches to understanding the impact of Leonardo at these different levels. In Chapter 6 we have already identified and described the outputs of the programme. Programme outputs are an important aspect of impact, constituting the measurable and quantifiable side of impact analysis. As we saw in Chapter 6 it has been possible to quantify the numbers of activities undertaken by different projects, to measure the numbers of participants in the different strands and so on. Impact, however, extends beyond what can be measured and quantified. A programme like Leonardo can have a far-reaching impact on the lives of the individual participants, and can lead to unexpected side effects and spillovers into separate but related areas of activity for the projects. At the level of the Member State and EU, the impact of Leonardo may reshape new policies and programmes and may extend to other policies and programmes. In this sense the impact of Leonardo reflects an intrinsic process of organisational learning. Impact is therefore equally conditioned by the receptiveness of the implementation environment as much as by the design and quality of the programme.

In the sections that follow we therefore draw on data which reflects the understandings of the key stakeholders as to their experiences of the programme. We employ a holistic approach, drawing on multiple aspects of the programme environment in order to put together a comprehensive picture of the impact and dynamics of the programme. By drawing on the qualitative indicators of impact we can direct greater attention to context, and to the development of in-depth understandings of the complex interaction between programme setting and programme design. This is important as it shapes and determines the nature and depth of programme impact at the different levels and spheres of operation.

In later sub-sections we give particular emphasis to the experiences and interpretations of Leonardo held by the projects and participants. The experiences of Leonardo tend to be shared experiences, subtly differentiated by context but – nevertheless - meaningful for the participants and projects that have engaged directly with the projects. In the paragraphs which follow we therefore stress the shared elements of experiences of the programme at the different levels of its operation. This chapter, in conjunction with our analysis of programme outputs (Chapter 6), provides us with a complete and comprehensive understanding of impact which is built on both quantitative measures and qualitative holistic data.

Before considering impact at the level of projects, participants and target groups, we first turn our attention to the European and Member State levels of programme operation. At these levels there is clear evidence that the programme impact has been directly conditioned by the organisational dynamics which have shaped the implementation environment as well as by the structural characteristics of the programme which we have already outlined in Chapter 4.

Deloitte & Touche


7.2 Impact of Leonardo at the EU and national levels Time has shown that the capacity of Member States to operate European programmes, coupled with their varying ability to integrate European activities into national policy making, and the tendency to allow programmes with major budgets to dominate attention, all played key roles in reducing Leonardo’s ability to command serious attention at high level within the Member States. In addition, an influence on policy, or general practice can be achieved through direct access to senior officials (assuming that they are really in a position to develop innovative policy and to change systems) or through the spreading of high quality concepts and ideas at grass roots level. In the case of Leonardo da Vinci, it is too early to assess if such a cause and effect relationship exists.

We have referred above to the rather disappointing role played by the Leonardo Committee.

Not surprisingly, many of the project representatives state their view that the programme has influenced the national environment (part of a justifiable sense of pride in their work, perhaps?). Those NCUs who were in a position to promote the themes and initiatives of LEONARDO (such as Germany) also feel that the programme had an impact in their Member State. But this is not (yet) discernible in policy or system terms. Given the fact that many projects are not yet finalised, and that the products have not been effectively presented, it would be illusory to expect much evidence at this stage. There remains the reality that the more individuals, organisations and policy makers work in a European Framework, the more likely it is that European-level message will filter through into their national environment. This process is given a useful push by the debate surrounding the creation of the Euro, the convergence of economic policies and the Luxembourg employment process.

7.3 Impact of Pilot Projects

7.3.1 Motivation and Expectations It is interesting to compare the main motivations and expectations held by the projects at the start of their engagement with the Leonardo Da Vinci programme, with their reported experiences of its impact. On the whole the projects appear largely satisfied that their objectives have been met, while there is widespread dissatisfaction with the delays and financial costs of the project.

Although there was general dissatisfaction with the financial mechanisms of Leonardo we are reminded by many of the projects that the prospect of financial support was instrumental in ensuring the start up and continuation of many of the projects - as one project says: “the project would not have been implemented without the support of Leonardo” (Dutch project)

It is regrettable that stakeholders who clearly welcome and often depend on the Community financial contribution are so often subjected to worry and delay because of a dysfunctional administrative system.

Deloitte & Touche


7.3.2 Perceptions of Innovative Character One impact of the Leonardo da Vinci programme, which was possibly not anticipated by the original designers of the programme, concerns the widespread perception held among the pilot projects that their projects were particularly innovative. This is an inevitable result of the selection processes applied to the pilot projects which placed emphasis and standing on the projects which were ‘innovative’. Perhaps not surprisingly, the pilot projects consistently reported that their projects were ‘innovative’.

It is a common place in European development programmes that the definition of innovation is something which is particularly hard to achieve and is determined very considerably by implementation context. We find a similar phenomenon in the vocational and educational training domain, and in Leonardo projects particularly. For this reason it becomes very difficult to assess the genuine innovativeness of projects, as what may be considered innovative in one context is less so in another.

Projects gave a number of reasons for the innovative character of their activities. These may be summarised as follows:

• The training methods, products and materials were perceived to be intrinsically innovative

• IT was used in the projects

• The trans-national element of the project was perceived to be innovative,

• The training material was felt to be innovative because it was the first time it had been applied in the domain in which the project operated

• The project was held to be innovative because it was inherently flexible

• The project was perceived as innovative because it responded to need

7.3.3 Trans-nationality and partnerships

“a good learning opportunity although frustrating at times” (UK project)

One of the most important outcomes of the Leonardo programme was, as we have seen earlier, the institutionalisation of a wide network of trans-national contacts and the sharing and exchange of skills working practices and cultures across the European arena. This must be regarded as a highlight of the programme which has almost universally been applauded and welcomed by both individuals and pilot projects alike. The success of the trans-national element of Leonardo may be grouped around a number of causal factors:

1. The projects benefited from exchange of experience and co-operation with other projects i.e. learning new skills

2. The trans-national dimension entailed opportunities for learning new cultures and this was also highly valued

3. The involvement of the projects in both national and trans-national partnerships was also highly valued

Deloitte & Touche


The projects make many statements about the benefits they have gained from participating in the partnerships and transnational exchanges which are inherent to Leonardo. Exhibit 7.1 overleaf gives exemplars of these reported experiences. The words of the projects themselves perhaps illustrate this best. We can translate these reported experiences into statements about impact:

1. There is evidence of organisational learning in terms of:

acquisition of new substantive, work related skills;

learning about inter-organisational relations – negotiation, bargaining, co-operation, team working, the value of and skills required for networking;

administrative learning centred on managing complex administrative procedures and multiple teams from different countries and organisations.

2. Projects have been increasingly sensitised to the added value that can be obtained by being open to learning from other parts of Europe. Leonardo has therefore been an educational experience which has contributed the process of “Europeanisation” of civil society and has raised the level of European consciousness.

3. A process of partnership capacity building and enhancement has been initiated or continued among the Leonardo projects. Understanding and successfully participating in partnerships is something which requires time to learn and develop. The Leonardo project has contributed to continuing learning about partnership in the Vocational and Educational training sector.

Exhibit 7.1: Transnational Partnerships in Leonardo

1. Benefit of exchange of experience and co-operation with other projects i.e. learning new skills

• “the company has gained considerably from working with the partner organisations” (Irish project)

• “Leonardo has improved and increased the definition and the exchange of products and methodologies at European level” (Italian project)

• “The co-operation with the partners was especially important for us. It has been possible during the project implementation to collect new knowledge and exchange experience about professional training in Europe”. (Austrian project)

• “Results of the project: new methods/approaches in connection with new situation at work, for training and in the social field were developed. & Important international networks exist now”. (German project)

2. Trans-national dimension learning new cultures also valued

• “all our advisers increase your expectatives about European dimension” (sic) (Spanish project)

• “Leonardo funding made it possible to create a common project with five nations represented. This was a necessary condition to achieve a “pan – European”

Deloitte & Touche


standard for securing of loads” (Swedish project)

• “Intensification and increase of the meaning of European topics in training and education. For this, following condition is required: Chambers of Commerce as responsible/competent body for European projects”. (German project)

3. Partnership element valued

• “we have created a consortium to co-operate in new projects thanks to the Leonardo programme” (Spanish project)

• “We found now stable and reliable partners with whom we want to implement and manage other European projects.” (Austrian project)

The pilot projects also frequently observed that partnerships were also burdensome to administer and to manage. Complaints were made about absent partners who left the burden of activity on the shoulders of the remaining partners. However, given what we already know about the developmental trajectory necessary for the acquisition of partnership capacity, we can see these complaints as demonstrating evidence of the early stages of the formulation of partnership capacity.

• “there was difficulty with partners from different countries to keep to deadlines and maintain momentum” (UK project)

• “in theory it was a partnership of equals but the ineffectiveness of one partner upset the equilibrium of the partnership”(Irish project)

• “From the beginning it was a partnership of equals but from the ‘Leonardo affair’ most of the partners lost interest and the project was dominated by the lead organisation” (Swedish project)

7.3.4 Managerial issues: Institutional performance In spite of the very positive message coming from the projects about the valuable impact of Leonardo in stimulating organisational leaning and Europeanisation at the project level, when we consider the impact of Leonardo in the sphere of managerial and financial matters the experience is very different.

“The strategy and objectives of the Leonardo programme as a concept is brilliant. The way it is organised is dreadful” (Irish project)

Almost without exception projects have spoken of:

• staff intensive management & high administration costs;

• complicated administrative and reporting requirements;

• administration not handled effectively – massive delays;

Deloitte & Touche


The questionnaires and project case studies provide ample testimony to the difficulties experienced.

Staff intensive management & High administration costs

Inevitably this has not been without significant financial and time costs to projects. Inability to predict the cost has been a major issue for the projects.

“The energy required for management of the project is at least the double as foreseen to reach the fixed objectives.” (German project)

Complicated administrative and reporting requirements.

Here projects suggest different solutions to the problem. Some ask for a help desk, others suggest streamlined management in favour of the projects, the provision of guidelines and models for the completion of the administrative documents and so on.

• “the requirements on the reports to Brussels were far too complicated” (Swedish project)

• “involvement of our organisation in the Leonardo programme was very worthwhile… any simplification of administration/paperwork would be most welcome!” (Irish project)

• “the financial administration of the project was trial and error. There should be more adequate instruction on this, perhaps even a help desk” (Dutch project)

Administration not handled effectively – massive delays

Projects make specific complaints about both the Commission and the TAO. The projects speak very directly on the subject of payment delays.

• “speed of response has been slow and always has been. However I feel staff have been both fair and thorough” (UK project)

• “Our dissatisfaction is due to the delays in payments, the excessive bureaucracy, and the consequences which arose out of the intervention of the Belgian Court of Justice.” (Spanish project)

• “The management of the budget has become intolerable for the contractor” (Belgian project)

• “Percentage funded by Commission? - We will tell you as and when the Commission pay the last tranche they are 2 years late.….we are currently envisaging legal action if a solution to our final payment is not found rapidly” (French project)

The operation of the TAO has also been the subject of pointed criticism by the projects:

Deloitte & Touche


• “One administrative task of TAO (especially the payments) is extremely delayed, requests for information are not answered only until we decided to call in a collection agency (Dutch project)

• “The TAO worked very inefficiently, contacts were very late. No transparency in the way reports were processed, little feedback. Payment late” (French project)

Conversely, a smaller group of projects have applauded the excellent support of the NCUs (see Chapter 4).

The chief impact of the operational difficulties experienced by the Commission in keeping to deadlines and meeting its payment obligations is that many projects report that they would not repeat the experience of Leonardo da Vinci because of the delays. The same projects that say that they will not participate in Leonardo again also report the benefit they have obtained as the result of the trans-national exchanges and partnership arrangements which were a feature of Leonardo. It follows that many projects who would not become involved again with Leonardo in its current form, would nevertheless be interested in involvement in subsequent programmes which draw out the positive aspects of the Leonardo experience without repeating the negative administrative experiences.

7.3.5 The Flexibility and Adaptability of the Programme Projects allude to the flexibility of the Leonardo programme. This has undoubtedly been a direct benefit to the pilot projects. We can identify at least three ways in which the innate programme flexibility has impacted beneficially on the activities of the projects:

1. it has created space to ensure that the projects are able to remain at the forefront of technological advances;

2. it has allowed the projects to adapt to changing market needs;

3. it has created the preconditions to allow the projects to become a platform for innovation.

7.3.6 Achievements A significant number of the projects report that Leonardo allowed development of new competencies and expertise that would not have been achieved otherwise. The projects provide testimony to advances in vocational and educational training which they attribute directly to their involvement with Leonardo. This is an important finding because it enables us to conclude that in spite of the serious operational difficulties experienced by Leonardo and in spite of delays in payments, the projects gained in terms of:

1. partnership capacity enhancement and organisational learning, and

2. significant advances and innovation have been made in terms of the particpants’ own working areas.

In the words of the projects:

Deloitte & Touche


• “Leonardo added a new dimension to our work that otherwise would have taken years to develop”(Irish project)

• “we are grateful to the fact that the Leonardo gave us the chance to create an initial vocational training specialisation in Greece which can also be offered at a distance”(Greek project)

• “participation in the Leonardo programme has given us a strong impulse to develop the important part of our educational activity and to strengthen its European dimension; we began some research projects connected with our tasks within Leonardo, work on the Leonardo project have interested our staff” (Polish project)

• “Leonardo has been a huge support allowing us to achieve significant advances in our work integrating underprivileged young people into the labour market” (Spanish project),

7.4 Impact of the Mobility Projects The mobility projects record many similar experiences to the pilot projects. Impact has not therefore been dissimilar for this group of projects. The mobility projects have similar goals to the pilot projects but, not surprisingly, an additional, important, goal also entails the aim of setting students and trainees on track to start their professional careers and working lives. In addition, the mobility projects also aim to enhance the employability of students. Other goals are broadly shared with the pilot projects as they were set out in 7.4.1 above.

As with the pilot projects, the trans-national dimension of Leonardo was also highly prized by the mobility projects. Value was placed on the international side of Leonardo as it facilitated the broadening of opportunities, learning and employability of the students and placement participants.

• “The programme responds to a current need for students of demanding professional training. The companies who host the trainees can make contacts with other regions and other European high schools” (German project)

As with the pilot projects, the satisfaction of the mobility projects was also somewhat lessened by the difficulties associated with delays and operational difficulties of Leonardo. Projects note that:

• “The programme should be absolutely managed with more flexibility, for the interest of the beneficiaries” (Austrian Project)

• “the delay in payments issue is of extreme concern to us” (Irish project)

• “The problems with the technical assistance management caused a lot of problems with the management of the project The problems even exist today” (Dutch project)

Deloitte & Touche


The notion of quality control is an important one for the mobility projects. This is founded on projects conferring status and trust to the regulatory bodies (Commission, Universities, other bodies). It is therefore important that damaging administrative delays are not allowed to damage and tarnish the position of the Commission in this respect.

“Moreover , to act under the shelter of a European programme is a guarantee for sending and hosting organisations , and an added value for all participants” (Italian project)

In line with the experience of the pilot projects, the mobility projects also claim to have been innovative, to have appreciated partnership but found it difficult to manage, and to have been highly motivated by the Leonardo grant money. They credit Leonardo with facilitating actions and developments that would not have been possible without the support of the programme.

• “We were able to offer the trainees funding for their practical placement periods, which would not have been possible without Leonardo” (German project)

• “As a direct result of the Leonardo activities a degree of knowledge of the labour market and effective links with European companies have been formed” (Portuguese project)

• “It enables all our students to gain international placements in sectors of industry and commerce which would not otherwise have been open to them. It allowed them to be far more mobile physically and intellectually than before” (Irish project)

The above reflects general project – level satisfaction with the mobility projects, which allowed tens of thousands of people to have a transnational training experience. But there is little evidence of any framework to collect the lessons of these projects, and channel them into a policy context. Why is mobility favoured? Is it merely to give a broader perspective to the participating individuals? Where are the policy analyses that should flow from this great flood of projects?

7.5 Impact for the Participants As we outlined in Chapter 2, we consulted as widely as possible in the data collection phase of the research. We therefore sent out a questionnaire to placement and exchange participants. Their responses can be grouped around 3 key benefits:

• the trans-national dimension was greatly appreciated and provide the basis for enhanced learning, cultural awareness and reflection about Europe;

• Leonardo provided the basis for enhanced maturity, independence and self confidence;

• Leonardo is perceived to enhance the employability of participants;

Deloitte & Touche


As with the pilot projects, the trans-national dimension of Leonardo was very important for the exchange and placement participants. They report that the scheme has:

• broadened their horizons – quite often they note that they would like further new work experiences in other countries;

• enhanced their cultural awareness – under Leonardo they have developed better understanding of different European countries;

• expanded their opportunities for communication with other professional people from abroad;

• enhanced their understanding of the supra-national dimension of the EU; and

• enhanced their reflection about Europe and its diversity.

The participants observe, for example, that:

• “after this experience I will try to communicate to others this sense of Community unity which few people feel” (Spanish man, to Italy)

• "a placement in a foreign country makes us more mature and responsible. Moreover, I got the opportunity to discover a new culture" (French woman, to Spain)

• "Each time I had the possibility of exchanging knowledge with colleagues from different European countries, the possibility for enhanced technological understanding was also increased" (French man, to Spain)

• "Exchange between countries is obligatory to achieve a better understanding of the remaining countries" ( Spanish man, to France)

The Leonardo programme has undoubtedly impacted significantly on the lives of the participants. The benefits are clearly age related. Young adults report that they have gained in employability, maturity and self-confidence whereas more mature participants tend to record enhancement in work related skills.

There were, however, some responses critical of aspects of the placement and exchange projects, particularly that that quality of the experience was adversely affected by the lack of preparation (knowledge and awareness of the host country) of participants and the lack of systematic monitoring by the organisers. Amongst the concerns of participants themselves were:

• a mismatch between work and learning needs (problem of time available for mentoring), more training wanted.

• a lack of choice about placement for participant/ more appropriate placement/ more information in advance of starting;

• an imbalance of too much work, not enough learning;

• the grant was too small; and

• the placement was too short to fit in all the learning;

Deloitte & Touche


Nonetheless, on this level we must regard this strand of Leonardo as a success. The vast majority of the participants record that they will recommend Leonardo to their friends. It is important to emphasise that the role of the European Commission and of the institutes / universities in Leonardo are regarded as quality control mechanisms for the programmes by the participants

"placements under an European programme offer guarantees and quality. It reassures students that every detail is under control, backed up by university, and the European Commission"

7.6 Impact in Central and Eastern Europe While it is too early to estimate the actual impact of the CEE countries’ involvement in the Leonardo programme it is possible to discern the following:

Bulgaria

• a very direct impact on vocational education and training policy and the harmonisation of training and employment policies in a general framework strategy;

• a change in the general culture how to work; a change in the style and mentality in vocational education and training (in the former system people went to a vocational education and training school, have got a training provision but they were not aware of their opportunities how to use the knowledge and skills they acquired, and efficiency and efficacy was not a goal;

• people learned in vocational education and training that evaluation and dissemination of results must begin at the very beginning of a project not at the end.

Czech Republic

• a new awareness and enthusiasm in the secondary school/vocational training sector for the benefits of mobility/exchange activity previously confined to the university sector;

• a major impact on individuals: there are currently 100 or so mobility projects in the Czech programme;

• an influence, at the very least inferred, upon the development of the Czech vocational training policies and practice;

• an opening of the Czech vocational training field to EU ideas.

Hungary

• The most important impact for the country is that the Hungarian vocational education and training now is present on the European “education market”, the Hungarian promoters and the outputs of the joint projects as products and services are present on international fairs, conferences and symposia.

• Another important factor was the acquisition of know-how of the new ways of project management in vocational education and training institutions (higher education institutions went through the same within TEMPUS and later World Bank projects). These institutions had got some of these experiences during the

Deloitte & Touche


PHARE vocational education and training and the World Bank projects, but in those cases there was always an external partner responsible for the management of the projects (donor/recipient relations in assistance programmes), but now projects have management responsibility themselves.

• Despite the classical “academic” orientation of the actors in vocational education and training institutions, a relatively large number of persons tried to run a project as promoter. Out of 150 applicants 43 projects were approved with Hungarian promoters, and another 247 institutions work as partners in projects with a promoter from another European country.

• A further important impact what one could call a kind of having a “Leonardo consciousness” among the applicants or projects’ participants. Having the label “participated in Leonardo da Vinci 1” is a source of pride for most of the vocational education and training institutions.

Poland The majority of pilot project promoters believe that their projects will have the strongest impact in the following areas:

• development of curricula designed to facilitate young people’s adaptation to changing conditions of the labour market;

• development of co-operation between SMEs on the one hand, and vocational education and training institutions on the other;

• growing involvement of social partners in vocational education.

Romania Even though at national level the priorities for the country’s economic and social development has been the development of continuous vocational training and regional development, most of the projects dealt with the acquisition of new professional skills (mainly within the frame of mobility and placements strands), development of new curricula, of social partnership and build institutional capacities. According to the NCU director this is explained by the fact, that there is no relevant legal regulations still, and also that the newly established institutional structures serve to solve this problem, and therefore the maybe promoters are concentrating on the national project more for the moment.

Concerning the projects they focus considerably on initial training, as this was regarded as having a “solid base” at operational level, due an effect of the PHARE vocational education and training projects, which started before Leonardo da Vinci .

The Leonardo programme served as a catalyst for change in all the CEE countries, facing the huge problem of transforming their economic and political systems in a much shorter period than the present EU Member States had. The challenges of globalisation and the rapid changes in technology and knowledge require a raising of the skills of the labour force in order to create a learning society.

All these countries had a long and well-established vocational education, mainly on the basis of the European continental traditions, which was broken up with the introduction of centrally planned economic and one party based political systems

Deloitte & Touche


Within that system closed links were developed between vocational education and training schools and economic actors (state owned industrial and agricultural enterprises), but one must not hide the fact that this system was rather efficient and serving well the needs of that economic system. When the systems collapsed, the whole education and training sectors were influenced. Neither the school system, nor the employment policy within the planned economic system were able to respond to the challenges.

The Leonardo programme could not help to foster change on a large scale, but could catalyse the processes, and via the co-operation and working together with different partners helped in setting up a “treasury of knowledge” about each other’s vocational education and training systems, and shared common values about working methods and different qualities of life.

Now the main concern of the CEE countries in their development strategies lies in the development of human resources, or linking education policy with labour market and employment policy. But due to the very rapid changes in the labour market and the traditional and somehow inherent slowness of reform in education, the task is not a simple one. From this point of view, even the rather late participation of the CEE countries in Leonardo da Vinci , as a “laboratory for innovation”, provided an excellent opportunity for complement their overall system level reform processes.

Leonardo da Vinci ’s priorities were designed to meet the needs of the EU Member States, but one can find matching priorities in the CEE countries.

From the interviews carried out in the CEE countries it is clear that there is both institutional and operational capacity to take part in the Leonardo Da Vinci programme. However, there has been and there are still problems in the financial capacity of CEE countries. In most of the countries, at the policy-makers level, the financial contribution to the programme is considered as an intangible investment. As an indicator of trust one must pay for it, because the outputs and outcomes bring added value worth more than the cost. This is especially true in the light of the belief of all the countries concerned possibly this co-operation helps them to take one more step towards accession to the European Community.

Considering the “EU policy” elements of Leonardo, in the CEE countries at policy makers’ level all the responsible people are very well informed and aware about those. However, there is a lack of information at school and social partners level, “at policy level there are very nice ideas, but at implementation level not so much” as was said in one of the Ministries. But if we look at the transnational character of Leonardo da Vinci and that it is considered as a laboratory for innovation these two elements may have a rather good impact even at the implementation level, due to the simple fact, that working together with others and exchanging ideas and confronting problems in a different system can develop the capacity of the partners to see their system from an outsiders’ point of view, and by this try out new approaches and may be change or modify his own system. This way even at the implementation level, at schools and at social partners the program had a real impact (or better to say will have an impact, because most of the projects are not yet finished, but there are some indicators that this will happen).

7.7 Impact in respect of Equal Opportunities Since the time of the launch of the Leonardo 1 programme in 1995 the Commission has developed its stance on equality of opportunity as a horizontal aspect of funded

Deloitte & Touche


programmes, as indicated in the earlier chapter of this evaluation report concerning the policy context. However, in Leonardo 1 the issue was simply addressed in two ways:

• a question to promoters on their plans for ensuring equal opportunities within their projects;

• the funding of fieldwork projects on the equal opportunities theme.

On the first, there is no record of monitoring procedures or follow-up of this question. Hence there is no data available for this evaluation.

On the second, the Commission has published a Leonardo da Vinci Compendium of Projects 1995-1996-1997 on Equal Opportunities for Men and Women in Vocational Training, detailing 118 projects (1.57 per cent of the total number of projects), 49 of which were procedure 1 and 69 procedure 2 selections. The Compendium states:

" ..most are pilot projects intended mainly to yield training tools for women in growth areas and cross-sectoral skills. There is also a significant number of survey and analysis projects which usually focus on investigating training needs at regional and sectoral level, with a view to adjusting training and vocational guidance for women, and examining the career obstacles for women within companies."

The Compendium notes the unevenness in the geographical distribution of equal opportunities projects, 38 percent of which were accounted for together by the UK (26 projects) and Italy (19 projects). These results are attributed to certain national characteristics and

"also reflect the different national sensibilities and meet the priorities established by the Member States for each annual call for proposals."

On January 16-17 1998, the Finnish Leonardo Centre organised a transnational thematic seminar on "Equal Opportunities in Vocational Training" in Tampere, in conjunction with the Commission and the Dutch and UK Leonardo NCUs. A significant part of the content of the event was a presentation of the projects as in the Compendium above.

7.8 Impact of specialist strands The programme featured a number of “specialist” strands which deserve brief analysis in addition to the above. These were:

• statistics actions;

• surveys and analyses

The statistics action is largely welcomed. A key player in European training statistics said “most of the data available now is there because of Leonardo’s contribution”. For a modest financial outlay, the results are impressive and useful. Already there is evidence of results through publications such as the CEDEFOP 1997 publication on statistics, the forthcoming publication on Initial Training, and CEDEFOP’s input to policy-making through various papers which now benefit from a better statistical base than before. These publications top CEDEFOP’s sales lists, and the collaboration with EUROSTAT has proved very fruitful.

Deloitte & Touche


The surveys and analyses strand, on the other hand, has not been as successful as might have been hoped. There was little or no linkage between this strand and the actual projects in the field, thereby minimising the opportunity to test some of the research theme results in live projects. The strand may also have suffered from confusion regarding the relative roles to be played by the EU’s Research & Development programmes and this activity within Leonardo. This area of Leonardo could have built on some of the useful track record of programmes like Eurotecnet in championing high-quality research on relevant themes, and bringing experts together to achieve exchange objectives. Once again, the Commission’s resource constraints meant that the results have not (yet) been disseminated or used for further work. The Leonardo 2 Programme has effectively suppressed this strand, perhaps a sign that the R&D programmes will now be the prime place for this kind of work.

7.9 Impact with respect to the Social Partners The involvement of the social partners as project promoters or major contributors within the Leonardo programme took place, with the support and encouragement of the Commission, within the framework of the European Social Dialogue on Training. Since the 1985 Delors move to revitalise the European social dialogue, vocational training has been expressed as a priority by the social partners at both interprofessional and sectoral levels.

For its part, the Commission has supported, for example, the independent dialogue between the social partners on education and training set up by the Social Dialogue Committee and also the involvement of the social partners in sectoral survey activity in both the FORCE programme (1990-1994) and the Leonardo programme.

In terms of practical involvement of the social partners in the Leonardo programme, the available data show that of the 2,272 projects resulting from the 1995,1996 and 1997 calls for proposals, a total of 413 (18 per cent) were identified broadly as social partner projects in the light of three criteria:

1. Projects directly submitted by social partners, in which the contracting/co-ordinating organisation was a trade union, employer, or joint organisation. Under this criterion 164 (7 per cent of the total number) projects were identified.

2. Projects submitted by equivalent organisations, such as trade union research centres or training bodies.

3. Projects promoted by other organisations in which social partners play a key part as parties to the social dialogue and/or disseminators or multipliers of project results and/or where the social partners are the target group.

Of the 164 ‘strictly’ social partner projects the majority was contracted to trade union organisations, followed by employers’ associations with joint bodies in third place.

Most of the social partner projects were pilot projects submitted under strand II, covering “support for the improvement of vocational training measures … concerning undertakings and workers”. The projects undertaken reflected the following priorities of the social partners themselves, developed within the support framework of social dialogue on training between 1994 and 1996:

• Planning of in-company training

• Training in SMEs

Deloitte & Touche


• Development of work organisation and training

• Specific occupations

• Integration of young people

• New forms of regional co-operation

• Validation/recognition of skills/qualifications

• Training materials and methods

• Allocation of training resources

• New responses to individual needs

to which four further themes were added after 1996:

• Lifelong learning

• Vocational guidance for young people and adults

• Qualifications, including the anticipation and analysis of needs, validation and certification

• Resources, including financing, joint responsibility and joint investment

In the estimation of the Commission, the body of the 413 social partner projects in the Leonardo programme reflected all the above priorities and concerns, demonstrating the value of the Leonardo da Vinci programme for the social partners.

7.9.1 Issues It is clear that the purposes of the Leonardo da Vinci programme were consonant with the principles of the European social dialogue framework and the involvement of the social partners has been an important element in the overall quality of the programme. However, while the range and variety (if not the number) of social partner projects in the Leonardo programme may have satisfied the Commission of the value of the programme to the social partners. ETUC suggested, inter alia, that

a) the overall trade union involvement in Leonardo da Vinci had been ‘unsatisfying and unsatisfactory’

b) Bureaucratic procedures deter both trade unions and management side of enterprises from participation

c) Actors at the enterprise level are unused to procedures such as are involved in applying for projects: this provides hurdles to participation

d) SMEs have difficulty, for example, because of limited resources and problems with worker release, in integrating vocational training within work organisation.

7.10 Impact with respect to sectors It is very difficult to make any substantial comment on the impact on various sectors by Leonardo da Vinci since the necessary data is lacking. For the centrally funded projects we can break down the projects by sector as shown below:

Deloitte & Touche


While this serves to give some indications, the dominance of training organisations leaves a large number of projects where we have no information on the sector addressed.

If a similar exercise is carried out excluding the training sector, the results show a distribution as follows:

Figure 7.1: Number of Projects by Nace code

0

200

400

600

800

1000

1200

1400

Agric

ultu

re H

untin

g an

d Fo

rest

ry

Fish

ing

Min

ing

and

quar

ryin

g

Man

ufac

turin

g

Elec

trici

ty, G

as a

nd W

ater

Sup

ply

Con

stru

ctio

n

Who

lesa

le a

nd R

etai

l Tra

de

Hot

els

and

Res

taur

ants

Tran

spor

t Sto

rage

and

Com

mun

icat

ion

Fina

ncia

l Int

erm

edia

tion

Rea

sl E

stat

e, R

entin

g an

d Bu

sine

ss A

ctiv

ities

Publ

ic A

dmin

istra

tion

and

Def

ence

Educ

atio

n,

Hea

lth a

nd S

ocia

l Wor

k

Oth

er C

omm

unity

, Soc

ail a

nd P

erso

nal

Ser

vice

Act

iviti

es

Priv

ate

Hou

sehl

ds w

ith E

mpl

oyed

Per

sons

Extra

-terr

itoria

l org

anis

atio

ns a

nd b

odie

s

Figure 7.2: Breakdown by sector (not including training)

050

100150200250300350400450500

Agr

icul

ture

Hun

ting

and

Fore

stry

Fish

ing

Min

ing

and

quar

ryin

g

Man

ufac

turin

g

Ele

ctric

ity, G

as a

nd W

ater

Sup

ply

Con

stru

ctio

n

Who

lesa

le a

nd R

etai

l Tra

de

Hot

els

and

Res

taur

ants

Tran

spor

t Sto

rage

and

Com

mun

icat

ion

Fina

ncia

l Int

erm

edia

tion

Rea

sl E

stat

e, R

entin

g an

dBu

sine

ss A

ctiv

ities

Publ

ic A

dmin

istra

tion

and

Def

ence

Hea

lth a

nd S

ocia

l Wor

k

Oth

er C

omm

unity

, Soc

ail a

ndP

erso

nal S

ervi

ce A

ctiv

ities

Priv

ate

Hou

sehl

ds w

ith E

mpl

oyed

Per

sons

Ext

ra-te

rrito

rial o

rgan

isat

ions

and

bodi

es

Deloitte & Touche


While this is a far from perfect measure, it shows that there is a strong domination by the service sector, including IT. There is also a large involvement in activities in the area of community, social and personal services.

It would be useful to analyse this with more complete data – which should be possible under Leonardo da Vinci 2.

Deloitte & Touche


8 Conclusions and Recommendations In this chapter we set out our conclusions and recommendations. For ease of reference this is done sequentially, with the main findings presented in subsections relating to the individual chapters. Recommendations are also linked to these findings. In the paragraphs which follow we begin by revisiting the objectives and the different actions in Leonardo da Vinci before discussing the key findings and recommendations arising from the main evaluative chapters concerned with the themes of complementarity, implementation, outputs and impact. The chapter concludes with an overview of the key evaluation themes that have guided and shaped the evaluation: outputs and impacts, effectiveness, efficiency, complementarity and synergy. Generic programme - level recommendations are also provided.

8.1 The Programme’s objectives revisited It is generally accepted amongst the stakeholders that the 19 different objectives with which Leonardo da Vinci 1 began were little more than a cumulative listing of aspirations and principles inherited from the group of programmes which preceded Leonardo da Vinci 1 . There is some suggestion that, had the ratification of the Maastricht Treaty not been delayed, the Commission might have had the time, and opportunity, to reduce and refine the range of objectives of the Leonardo da Vinci 1 programme to something more coherent. The adoption of the Council decision probably more reflected the Member States’ wish not to lose any of the elements of the previous generation of programmes, rather than a well-planned and thoughtful approach to a complementary European-level contribution to policy-making at a time of change.

The external interim evaluation of Leonardo suggested that the multiplicity of objectives would make any final evaluation impossible. We agree in part. The fact is that because the 19 'objectives' statements were not clear objectives, targeted and measurable in terms of performance, it is not feasible to attempt to provide any assessment of their achievement in quantitative terms.

In the light of that, this evaluation has concentrated upon what, as far as we can learn, was actually achieved by the Leonardo expenditure, with what effect and to what end.

In looking towards the future, we anticipate that the operation and evaluation of Leonardo da Vinci 2 is likely to benefit from the reduction of programme objectives from 19 to 3, enabling much clearer performance review.

Recommendations: Programme Objectives

• Programme objectives must be patently measurable and achievable; should be distinguished from aspirational aims and should conform to the SMART criteria described later in this chapter.

• There should be a clear distinction between process type objectives, e.g. concerned with the incremental development of an EU level vocational training policy, and task-type objectives, e.g. concerned with numbers of training places created for excluded groups.

Deloitte & Touche


8.2 Types of action During Leonardo different types of action took place:

• Policy development actions

• Pilot projects or demonstration actions

• Mobility actions

• Publications, Databases and other Information actions

Of course, it is not always possible to completely distinguish between the four types of action in practice. This reflects the sophisticated nature of many of the interventions in Leonardo. Actions with multiple dimensions make typologies of interventions difficult to apply rigorously. In many cases, for example, pilot actions are an integral part of the policy process where they can be used to test policy responses to problems, which have been identified through policy analysis and research. Some publications are linked to other actions as well. Nevertheless, the distinctions are generally useful, in our opinion, for identifying the different types of activity undertaken under the programme.

8.2.1 Policy Development Under “policy development” we include all measures from research actions through conferences and seminars and the production of statistics, research reports and analyses of specific areas. It also includes the analyses of outputs from other pilot and project-type actions.

Given the economic policy context, it is clear that the EU dimension has an important role to play in the elaboration of training policy for Member States. The European dimension to policy research is of enormous value, because it allows for exchanges of experience between Member States and analysis on a European scale. “Europe” can compare and contrast the very different experiences encountered in a range of locations and then help to analyse which solutions have proved effective. Such analysis (and the results of the research itself) must however be disseminated widely and discussed with the Member States in order to maximise its value. This is because the appropriate policy response will in all cases have to be delivered at national or regional level.

Recommendations: Policy Development

• Analysis and comparison of best practice across Europe should be undertaken

• This must be disseminated and discussed with the Member States to facilitate national and regional level take up of best practices.

8.2.2 Pilot Projects As “pilot projects” we include all test or demonstration actions funded by the programme. The direct quantitative impact of pilot actions is typically localised and

Deloitte & Touche


limited in relation to the potential number of beneficiaries in the EU as a whole. The main value of the actions lies in the potential to influence a policy debate and to be replicated by other actors, or by the Commission itself, if they are successful. They are usually used to test different policy responses to identified problems and are often called “demonstration projects” because of their ability to demonstrate suitable courses of policy action for the future.

There is a strong case that pilot actions should only be used to test potential policy responses (rather than merely funding “interesting” ideas on a demand-driven basis), and only those which are not already being tested elsewhere. This means that they should only be launched after an exhaustive analysis of the relevant policy options.

Our own view is that pilot actions must indeed be linked to pioneering innovative policy development and not just launched in the absence of this strategic focus. Furthermore, the results of the pilot action must be examined in great depth and communicated to all relevant policy making bodies and interested parties. Finally, the objectives of the pilot actions must be clear and if possible, performance indicators should be designed to measure the success or otherwise of the pilot, before it is even launched. Leonardo has not (yet) demonstrated that it has followed these principles.

Recommendations: pilot projects

• Pilot actions should only be used to test potential policy responses not being tested elsewhere. Exhaustive analysis of relevant policy options is therefore first required.

• Results of pilot actions must be disseminated.

• Performance indicators should be established for the pilot projects from their inception.

8.2.3 Mobility actions Mobility actions are those which involve actors in a project that forms part of the general delivery of existing programmes, rather than being a special action with a hoped-for larger effect. There is no fundamental problem with the Commission being involved in such actions, as long as there is a clear identification of a need at European level and a clear strategy as to objectives, quality assurance and use of results. Obviously, EU actions should be integrated as far as possible into other mechanisms in order to gain greater visibility and greater efficiency and to avoid duplication and wastage of resources.

Such actions must, in our view, be mounted on a scale which is sufficient to make a real impact. This is necessary to achieve economies of scale and administrative efficiencies. In order to achieve the necessary volume of activity and market penetration, these actions should also, in our opinion, be well marketed and resourced and planned on a multi-annual basis for sustained impact. The mobility actions under Leonardo, while they succeeded in “moving” a large number of people, were generally rather isolated in the sense that they occurred without close monitoring or capitalisation of their results.

Deloitte & Touche


Recommendations: Mobility Actions

• Analysis needs to be carried out of the desired outcomes for Leonardo’s mobility actions

• Analysis of what works is also needed. This requires the establishment of monitoring systems and procedures that focus on desired outcomes and best practice. This will enable the mobility interventions to be focused on innovative areas and avoid the risk of duplication of effort with other EU and national programmes.

8.2.4 Publications, Databases and other Information Actions Most of the publications and other information actions produced by the programme are directly related to its dissemination strategy. This type of action should be very focused on specific policy purposes, if they are to have real value. The studies, surveys and analyses work of Leonardo was mixed in terms of its real contribution in this sense. This comment ignores the obvious lack of widespread dissemination of results of the Leonardo projects.

Recommendations: Publications, databases, other information actions

• Ensure actions are focused on specific policy purposes, aimed at specific target audiences and adequately disseminated

8.3 Complementarity At a conceptual level the Leonardo programme was complementary to other EU programmes as we saw in Chapter 4, but almost as a result of accident rather than careful planning.

However, we have noted that:

• The other training-related programmes funded by the EU budget (principally ESF and R&D) had budgets greatly superior to that of Leonardo. They inevitably occupied Member State authorities (and national training bodies) more than Leonardo did.

• Socrates and Leonardo were not co-ordinated at official level, either in Brussels or in the Member States. Given the high incidence of participation in both programmes by formal education institutions, some complementarity was secured at actor level. But this was not widespread.

Deloitte & Touche


• The incidence of effective knowledge sharing through inter-departmental contact within the Commission remains low. This is a regrettable handicap on the potential for the large number of EU-level training-related initiatives to combine and yield added value for the benefit of all.

The different EU programmes tend to operate in parallel rather than in an integrated fashion. Providing the opportunities for the exchange of best practice and lessons between programmes and departments would entail the provision of resources and installation of mechanisms for the flow of information between programmes. This implies the necessity for programmes to have a degree of porosity in boundaries which would enable the communication channels to work effectively. We make a number of recommendations below, which would assist the establishment of mechanisms for effective knowledge management (policy think tank, profile raising activities, workshops, dissemination functions, use of Internet and CEDEFOP). Such devices would assist the Commission to maximise the benefits of having complementary programmes by ensuring integration and a more holistic approach to vocational education and training.

We have noted that continuity between programmes in fact represents a policy opportunity that, if accurately harnessed, can yield valuable conduits for the dissemination and spillover of best practice and both strategic and operational lessons. The establishment and utilisation of information channels between programmes and related policies is essential for optimising knowledge management strategies within the overall vocational and educational training polity. Although Leonardo did not necessarily achieve great synergy between ministries at national level, it did begin a process which may yield fruit in the future.

The notion of seeing different European interventions as a continuum, playing really complementary roles in a policy process is a key conclusion in this context.

Recommendations: complementarity

• High-level political commitment to strategic integration of the whole range of current EU training activities.

• Commit to a policy continuum approach involving research, field testing, and mainstream delivery, to replace current uncoordinated EU interventions.

• Dissemination of best practices across other EU programmes should also be undertaken by the establishment of an effective knowledge management strategy

8.4 Implementation The implementation of Leonardo has arguably been the most problematic area of the programme. As we saw in Chapter 5 the programme suffered from a number of structural and design difficulties which impeded effective programme delivery.

Deloitte & Touche


In the area of implementation we can cluster the main conclusions around two separate phases of the policy process: programme design and programme delivery. Programme delivery relates to the mechanics and structures of implementation. An implementation experience is conditioned equally but the nature of programme design (i.e. its implementability) and by programme delivery structures and procedures. Both programme design and programme delivery therefore have a direct impact on the implementation experience. Conclusions and recommendations arising from our analysis of the implementation of Leonardo are therefore grouped around the related themes of programme design and programme delivery.

8.4.1 Programme design In the area of programme design, our conclusions are centred on four key points:

• The programme lacked a user-focus. It therefore did not succeed in accommodating or responding to the particular needs of the targeted participants and groups;

In Chapter 5 we saw that target groups were adversely affected or even excluded by the design structures. For example SMEs often lacked the necessary resources to wait months for payment. Projects without pre-existing networks struggled to create partnerships. Delays constitute bias against target groups. Complexity of the administrative procedures was daunting and unmanageable for the target groups. Documentation was also often late and incomprehensible. The complexity of the system severely limited the involvement of some types of organisation. In Chapter 7 we saw that lack of feedback to the projects did not help sustain user morale or maintain a positive image for the Commission

• The programme did not incorporate effective structures for the dissemination and roll-out of project results or best practice strategies;

A major weakness of Leonardo da Vinci lies not in what it did but in the fact that it does not know /cannot show what it did, and cannot learn from it.

• The programme design did not take into account the capacity of the Commission to implement the programme.

Both programme design and structures for programme administration need to be designed with Commission capacity problems in mind to avoid bottlenecks.

• The programme was not designed with adequate monitoring or evaluation systems and therefore lacked any form of sophisticated evaluation capacity. Recommendations from previous evaluations have not been implemented, despite the increasing Commission attention to evaluation practice. Proper evaluation of what has been done is important for the programme and proper monitoring and centralisation of data storage is needed for this.

These four major conclusions in the area of programme design lead us to recommend the following actions:

Deloitte & Touche


Recommendations: programme design

• A needs analysis should be undertaken to ensure that the programme is accessible to the target group.

• Simplification of all procedures is urgently required. Efficient payment systems must be developed.

• Administrative practices should be tested early in the programme, changed as required in the light of experience

• The actions for mobility projects need to be put into a context where they can contribute to policy ideas.

• Dissemination of project results and best practices is essential.

• Ex-ante targeting needs to be increased.

• Formative utilisation evaluations should be introduced.

• Information systems should be defined from the start of the programme to meet the potential evaluation and exploitation needs, as well as the administrative processes.

• A proper monitoring system which is focused on outputs and outcomes and their use needs to be developed.

8.4.2 Programme delivery In Chapter 5 we outlined a number of findings related to programme delivery. Our conclusions are centred on the following key points:

• The programme was successful in terms of provision of:

♦ an important tutelage function for projects

♦ an important information brokerage function for the projects

This was one of the most successful aspects of the implementation experience and was vested in the NCUs. Continuity is necessary at the centre and in the NCUs to ensure that the information base is not lost

• Programmes and policies evolve and mature. During the life of a project goals and objectives should be able to evolve and respond to changed circumstances. The programme did not recognise this reality. (see Chapter 5). (The TAO’s tasks, however, were refined over time reflecting the different stages of programme and maturity of experience of Commission and TAO. This is a positive finding.) There was insufficient flexibility for the programme structure to deal with changing programme needs at different stages in its life cycle. This over-rigidity was symptomatic of a lack of organisational learning, knowledge management,

Deloitte & Touche


and structures which were overly internally focused on organisational matters rather than adapting to changing user and market needs.

• There was a tendency to over-regulate certain aspects of the programme. This effectively operated to dampen down innovation opportunities and constrain the implementation capacity of the programme. The adherence to contracting procedures that were defined before full knowledge of the situation was possible, and which did not recognise the reality of the programme participants, caused a loss of programme responsiveness. The Commission tradition (imposed by the Parliament) of taking resources away from a project at the end of its life, regardless of the fact that activities may continue for at least a further 12 months means that there is little chance of proper recognition or use of the outputs.

• Decisions about the allocation of resources within the TAO did not always reflect the existing administration requirements - insufficient resources were allocated to project selection and proposal management for example. This led to a significant backlog in payments, and contributed to slow approval procedures.

• Inter-organisational boundaries between the TAO and the Commission and between administering bodies and projects were not clear as they were not adequately and clearly defined.

• Vicious circles were set up - budget reductions interfered with the successful completion of the project work programmes and led to lengthy negotiation processes. Setting unrealistic project deadlines as a result of late starts following extended contracting procedures can lead to project failure. The project selection procedure was sub-optimal: project quality was sacrificed in favour of geographic spread and an attempt to fund as many projects as possible. This applied to both pilot and mobility projects following selection procedure II.

The programme objectives were lost sight of and were substituted by an over-emphasis on procedures and the administrative process. Too great a focus was placed on procedures and not enough on the overarching goals (ends). The administrative issues were prioritised and uppermost. The issue of quality in mobility projects needs to be addressed in Leonardo da Vinci 2. This does not imply that there was necessarily a problem with quality for these projects but that the mechanism for quality control was absent.

• The opportunity for strategic European planning was lost in the Leonardo Committee, and there was no other effective policy channel for review of results and capitalisation thereof. Some information is detailed in the Compendium, but this is essentially just a list and it does not describe outputs of projects. This does not provide an adequate base for strategic European planning in the vocational educational and training system. The large size of the committee inevitably made strategic direction difficult.

• The financial arrangements for the TAO, whereby profit was effectively excluded, were extremely questionable and risky. This was an important factor impacting on the operability of the TAO.

• Participants were effectively discouraged from seeking to participate in Leonardo again. There was no real interest in their environment on the part of Brussels, and the programme suffered from a very inappropriate management system and no

Deloitte & Touche


strategy for the crisis when it came. This meant that the system was stacked against the supposed target groups.

Conclusions related to the area of programme delivery are also detailed further in the matrix below:

Recommendations: programme delivery

• Build on the positive aspects, be willing to revise and modify goals. Recognise the natural evolution of programme life-course.

• Succession planning for the post-Leonardo continuation of projects should be an integral part of project planning and approval.

• A crisis management strategy should be in place.

• The roles of the different bodies should ideally be defined through the setting up of guidelines and agreed conventions.

• A system for quality management and procedures needs to be developed.

• In recognition of the partnership character of the projects, selection should not necessarily be based on the nationality of the lead partner alone.

• For Leonardo to be a true European programme the Member States must operate to agreed European guidelines in the selection procedures.

• Incentive (profit) is necessary for limiting risk, and enhancing motivation and quality in outsourced management activities. This should be considered when awarding future management contracts.

8.5 Outputs and Impact In this section we detail our conclusions and recommendations related to outputs and impact. In the paragraphs that follow we draw on our discussions in Chapter 6 and 7 of this document. Outputs and impact are two aspects of the policy cycle that are intrinsically related and so the two sets of findings are discussed together.

• The policy framework and political consensus, surrounding the role of the EU in training as an element of economic and social policy, had developed by the time of Leonardo da Vinci 1 to the extent that it could have played a coherent and integrated role in testing and sharing ideas and exposing people to other approaches. The testing and sharing of ideas was not achieved due to lack of capitalisation of project results, while the exposure to other approaches (which has a much smaller penetration) occurred for those who participated directly.

• Leonardo da Vinci 1 lost some of the specificity which the previous multiple programme approach had secured, without securing greater impact or comprehensiveness (because of its muddled design13 and failure to capitalise on project results).

13 See our comments on programme design in section 8.4)

Deloitte & Touche


• The lack of insight into products is unacceptable. A working methodology for capitalising on projects’ outputs is crucial and central to a programme such as Leonardo.

• The delays experienced during the programme had a negative effect on:

♦ project design (difficult to implement in the time left available),

♦ dissemination (the problems with the TAO consumed time allocated for dissemination),

♦ ability of target groups to be included (projects had to be self-financing or start late),

♦ innovation (projects lost their innovative edge because of delayed starts)

♦ participants’ willingness to participate in future Leonardo programmes.

• However, project participants largely felt satisfied. They created informal networks, initiated learning about partnership operation and learned about participating in European programmes. Evidence of organisational learning was identified in the areas of:

♦ work related skills,

♦ managing complex administrative procedures,

♦ inter-organisational relations (negotiation, bargaining, co-operation, teamwork, networking skills).

Exchange and placement participants also reported learning in the areas of:

♦ work related skills,

♦ knowledge of other cultures,

♦ increased independence, maturity, self-confidence, and enhanced employability.

• The statistics strand was very useful: the resources provided allowed real progress to be achieved.

• The surveys and analyses strand suffered from isolation from the rest of the programme.

• Despite everything the programme embodied an innate flexibility allowing space for the projects remain at the forefront of technological advances, to adapt to changing market needs, and to allow the projects to become a platform for innovation. This was more by luck than design however.

• The transnationality of the programme contributed mostly to the Europeanisation of civil society and the growth of European consciousness. More importantly for the vocational and educational training system, a wide network of transnational contacts was built up allowing the exchange of skills, cultures and learning.

• In eastern and central Europe additional benefits accrued as the result of the unique circumstances of these countries.

♦ There was increased harmonisation with the European vocational education and training system,

Deloitte & Touche


♦ Changes in working cultures and project management systems were implemented

♦ The practice of partnership and inter-SME co-operation was encouraged by Leonardo

♦ Projects recorded an increment in institutional capacity

• Leonardo was responsible for initiation of a number of projects which could not have come into existence without the Leonardo da Vinci programme.

As we have seen in Chapter 7, impact is nevertheless very difficult to measure because of the absence of necessary data about products and achievements. Data enabling assessment of impact needed to be collected at the outset of the programme. The pilot projects often alleged that Leonardo had had a significant impact on the national systems of the different Member States, but this is very difficult to establish in the absence of this kind of data. What is clear is that impact has been conditioned by the organisational dynamics and by the structural programme characteristics which helped shape the implementation environment and subsequent attainable impact for the programme.

We outline the main recommendations arising from this analysis below:

Recommendations: outputs and impact

• data on project services and products should be collected and best practices disseminated

• in addition mechanisms for ensuring quality of the products should be put in place.. Project results need to be capitalised upon and the programme design should be transparent to users. A user focus for the programme design is needed.

• A working methodology capitalising on the projects’ outputs would entail an approach designed to ensure that important best practices were captured, innovative trends pointed up and strategies for follow through put in place. This is especially important for projects of significant potential and value.

• Organisational learning, and participant learning was a positive impact of Leonardo. Future programmes will need to capitalise on this positive outcome through an effective dissemination and knowledge management strategy. Workshops to share information, effective web sites, and the dissemination of and reward for best practices must be seen as integral and critical for this process

• Both output and impact strands have a key input to make into the data collection and dissemination strategy for Leonardo

• The eastern and central European experiences must be built on where they were positive. This would entail the organisation of workshops and pro-active dissemination of the learning experiences.

• Appropriate data collection is of critical importance for the subsequent measurement of impact. Baseline data needs to be collected and performance measures established.

• Member States and the Commission should establish a formal mechanism for assessing results and processing them thereafter.

Deloitte & Touche


8.6 A summary In the matrix below we summarise the strengths, weaknesses, opportunities and threats for the future which relate to the Leonardo programme we have evaluated.

Strengths Involvement of key players

Trans-national dimension

Open to a large number of themes and issues

Innovative approaches to process

Organisational learning by projects

Some believe Leonardo has influenced national systems

European citizenship enhanced

Maturity, independence, skills and employability of young participants enhanced

Projects left to their devices. Although this led to feelings of isolation it also de facto created space for innovation at project level

Many of the projects would not have happened without Leonardo

Contributed to continuing socialisation of EU training actors

Weaknesses Programme not user friendly or user-centred

Delays

No use made of outputs

Focus on means not ends

Over regulation of TAO and administrative aspects

Insufficient quality control mechanisms for project outputs and products

Organisational roles not clearly defined

Insufficient recognition that goals evolve and change through the lifecycle of a project

No crisis plan

Strategic direction not achieved

Lack of quality controls

Dissemination strategy not in place

Opportunities for future European complementarity could bring great rewards if secured

Forging of sustainable partnerships: keep building the links between practitioners

Threats for future

Poor image accorded to the Commission

Projects not willing to be involved with Leonardo again

Capacity to manage the programme threatened

Partnerships heavy to manage for projects

8.7 Overall We set out below some recommendations arising from the conclusions of this Chapter. In general terms we find that future Leonardo programmes would benefit from a much greater user-focus. The projects and participants should be uppermost in the focus of the programme design. We therefore advocate that a comprehensive needs analysis is done during the phase of programme elaboration so that the programme can respond directly to the needs of the target groups. This kind of needs

Deloitte & Touche


analysis is best done by listening to the voices of the participants. We would propose the employment of focus groups, questionnaires and user panels to aid in the design and programme elaboration.

In the area of complementarity , output analysis and dissemination there is a need for considerable change. CEDEFOP, the ETF, and new internet-based initiatives could play a useful role in drawing all of the relevant material together (not just from Leonardo 2).

We also propose the convening of high level workshops to disseminate key findings and to rollout best practice to other related EU programmes. The complimentary links between Leonardo and other programmes should be officially co-ordinated. Complementarity entails recognition that EU actions in this domain area are systematically related and need to be integrated at the European level and not just at the micro level by individual participants and projects.

A high-level think tank, with the imprimatur of the Member States, could rise above some of the management and allocation functions that the Leonardo da Vinci 1 Committee carried out, and set an integrated course for the whole range of training activities that the EU is now engaged in. This would encompass the selection of major themes for policy development, research, demonstration projects, event s and so on. The continuum of programme styles could be harnessed to give real power to this work and to allow each area of Community intervention to contribute in an integrated way.

With regard to evaluation, we strongly urge the establishment of proper monitoring and data storage systems. In addition- given the need to tailor the programme to the needs of the target groups, and to ensure that evaluation findings are utilised, relevant and helpful - we would advocate the establishment of formative evaluation structures. This would entail the decentralisation of the evaluation process to more broadly include the relevant stakeholders. The Expert Group would provide a good forum for this as it comprises a fairly representative cross-section of stakeholders. Consideration should be given to broadening the group to include other important stakeholders if it were to take on this role.. Such evaluation approaches invest ownership of the evaluation in the stakeholders and tend to induce capacity enhancement as well as on-going programme improvements throughout the life cycle of the project. This evaluation approach is centred on action, rather than audit of outcomes, and is a proven mechanism for ensuring enhanced quality and better outcomes for programmes.

8.8 Specific Objectives of the evaluation In Sub-Section 2.2.2 we set out a number of specific objectives for this evaluation – to evaluate the programme and provide advice and counselling to the European Commission on five key issues:

• Outputs and impacts of Leonardo

• Effectiveness of the programme

• Efficiency of the programme

• Ways to improve efficiency and effectiveness in future programmes

Deloitte & Touche


• Ways to develop complementarity and synergy with other vocational education and training instruments and policies.

8.8.1 Outputs and impacts Leonardo undoubtedly impacted upon the lives of individual participants and was instrumental in many instances in allowing the start up of projects which otherwise could not have come into being. It undoubtedly contributed to the broad European project helping to shape the emergence of European civil society and European consciousness.

The outputs and impact of Leonardo were difficult to assess because of the absence of a clear data collection strategy; gaps in information were evident in several areas (products, equal opportunities etc). We have therefore stressed the need for evaluation to be conceptualised as a process which starts with the elaboration of the programme and is on going throughout the programme’s life cycle. This entails a more formative approach to evaluation and an acceptance of programme flexibility and adaptation during the course of its delivery,

We have also stressed the need for rigorous collection of appropriate information which should be determined in line with the questions which future evaluations seek to answer. For tracking impact, for example, it would be useful to establish baseline indicators as well as performance measures. This should be carried out in conjunction with qualitative evaluation designed to illuminate and shape processes (implementation, design, partnerships, and experiential data, for example). Appropriate dissemination and quality assessment of products is critical.

8.8.2 Effectiveness of the Programme Effectiveness is closely related to impact. Judgements of effectiveness must, by definition, be related to the goals of the programme. Questions of effectiveness normally concern a programme’s effectiveness in the achievement of defined goals and objectives.

We can observe that the programme had definite micro-level benefits and in this respect was effective in achieving some of its objectives, most particularly in the area of achievement of broadening the European citizenship and, to a degree, enhancing employability.

The degree to which the Programme was effective in achievement of its other goals is more difficult to determine for a number of reasons. Firstly, the lack of clarity, precision, and measurability of the programme’s multiple goals is a real obstacle to measurements of effectiveness. Goals should be SMART – Specific, Measurable, Achievable, Realistic and Time-achievable. Progress with the rationalisation of objectives for Leonardo has begun with the reduction of objectives from nineteen to three. We have noted, however, that there is still room for greater clarity and transparency in this area.

Assessments of the degree of innovation are also difficult because of a lack of common definition for this term. Lack of clarity about the meaning of ‘innovation’ is not undesirable, however, as the degree to which a project can be innovative is heavily contextually determined. Some guidelines for the identification of what is

Deloitte & Touche


innovative in specific contexts would however be useful if Leonardo is to be judged successful in stimulating innovation in the future.

A key objective of Leonardo was to develop a European vocational and education training policy. We have noted during the course of this evaluation that the absence of data (baseline indicators in particular), the time frame and the multi-causal nature of change in the vocational education and training system make it impossible to assess the degree of impact of the programme in this area. Approximately 50% of the projects we sampled felt that Leonardo had impacted on the national vocational and education training system, as their own projects’ products had filled gaps in vocational education and training provision. However, identification of the effectiveness of Leonardo in the Europeanisation of vocational education and training systems can only be achieved in conjunction with the prior adoption of a clear strategic focus for the programme in this area. Leonardo was not able to capitalise on this opportunity. Impact of Leonardo in this area is consequently not focused and opportunities for synergy and strategic planning are therefore lost.

8.8.3 Efficiency of the Programme There is little doubt that the efficiency of Leonardo has been seriously adversely affected by the administrative difficulties which characterised the programme during the period of this evaluation. We have recorded the adverse impact of this circumstance on the programme. Long delays have been the most serious obstacle to efficient programme operation, and in Chapter 5 we have discussed in some detail the nature of these implementation difficulties. In summary, a combination of structural features (administrative procedures) and organisational dynamics have jointly established an adverse implementation environment for the programme. This has had a significant impact on the programme’s efficiency which cannot be lightly ignored.

8.8.4 Ways to improve efficiency and effectiveness in future programmes In the previous sections of this Chapter we have set out a number of recommendations which will help improve both the effectiveness and efficiency of the programme. In summary, we have advocated that reforms should be made in the design, evaluation, delivery, and dissemination phases of the programme.

The adoption of a user–focus is central to the production of a user-friendly workable programme and meaningful and useable evaluations. A high level think tank should be established to guide the strategic direction of Leonardo within the overall European training debate (which need not be limited to EU for a), and move towards a European dimension for vocational and education training. During the delivery phase, care must be taken to ensure that the objectives are not marginalised in favour of rigorous application of administrative procedures; it is important to keep sight of the goals of the programme at all times. Dissemination of the programme’s results must be seen as critical and we have suggested that CEDEFOP, could play a useful role in this function.

Adoption of the recommendations set out in this Chapter will help to ensure that the intrinsic efficiency problems experienced are minimised in the future, and the programme becomes more effective as it is better able to capitalise on its successes and better able to achieve its goals in a more systematic and targeted fashion.

Deloitte & Touche


8.8.5 Ways to develop complementarity and synergy with other Vocational Education and Training instruments and policies

We have discussed the current situation of Leonardo in the context of other, related, EU programmes in Chapter 4. Finding ways to avoid dead-weight is a widespread problem across many European programmes. Ensuring that programmes are complementary to each other is a more manageable and productive way of approaching this issue. Inherent to this approach is the practice of knowledge management and the notion of the learning organisation. Thus best practices and important lessons from programmes should be available to other programmes and policy-makers to optimise and build upon the learning patrimony of the European Commission.

In the domain of vocational and educational training, this approach calls for Leonardo to be situated within the broader context of other vocational education and training programmes. We have alluded throughout the report for the need for programmes to have porous boundaries. This means that best practices and valuable policy and practice lessons are able to flow in and out of programmes to the mutual benefit of related programmes, also operating in this domain.

Ensuring complementarity and synergy with other programmes necessitates the creation of opportunities for inter-programme dialogue. We have suggested workshops, web fora and so on, and effective dissemination strategies. We have provided detailed recommendations for the achievement of these principles in Sub-section 8.3 of this Chapter. The tracking of related initiatives and measures in the vocational and educational training arena is also necessary.

final evaluation of leonardo da vinci i programme · leonardo da vinci 1 programme coincided with a...

Documents