Formative Evaluation of the Northern Aboriginal Economic Opportunities Program 2011-2012 to 2015-2016

Acronyms

Brief explanations and descriptions are provided with the acronyms to improve the readability and comprehension of this report.

Glossary of Terms

The following evaluation terms are defined in Treasury Board Secretariat's Results-Based Management Lexicon.^{Footnote 1}

Effectiveness

The extent to which an organization, policy, program or initiative is meeting its expected results. Related term: Cost Effectiveness - The extent to which an organization, program, policy or initiative is using the most appropriate and efficient means in achieving its expected results relative to alternative design and delivery approaches.

Efficiency

The relationship between the outputs and results of an organization, policy, program or initiative and the resources used to produce them.

Evaluation

The systematic collection and analysis of evidence on the outcomes of programs to make judgments about their relevance, performance and alternative ways to deliver them or to achieve the same results.

Expected Result

An outcome that a program, policy or initiative is designed to produce.

Input

The financial and non-financial resources used by organizations to implement policies, programs and initiatives to produce outputs and accomplish outcomes. E.g., Funds, personnel, equipment and supplies.

Logic Model

A depiction of the causal or logical relationships between activities inputs, outputs and the outcomes of a given policy, program or initiative.

Outcomes/Results

An external consequence attributed, in part, to an organization, policy, program or initiative. Outcomes are not within the control of a single organization, policy, program or initiative; instead they are within the area of the organization's influence. Outcomes are usually further qualified as immediate, intermediate, or ultimate (final), expected, direct, etc. Three types of outcomes related to the logic model are defined as:

Immediate Outcome

An outcome that is directly attributable to a policy, program or initiative's outputs. In terms of time frame and level, these are short-term outcomes and are often at the level of an increase in awareness of a target population.

Intermediate Outcome

An outcome that is expected to logically occur once one or more immediate outcomes have been achieved. In terms of time frame and level, these are medium-term outcomes and are often at the change of behaviour level among a target population.

Final Outcome

The highest-level outcome that can be reasonably attributed to a policy, program or initiative in causal manner, and is the consequence of one or more intermediate outcomes having been achieved. These outcomes usually represent the raison d'être of a policy, program or initiative. They are long-term outcomes that represent a change of state of a target population. Ultimate outcomes of individual programs, policies or initiatives contribute to the higher-level departmental Strategic Outcomes.

Output

Direct products or services stemming from the activities of an organization, policy, program or initiative, and usually within the control of the organization itself. E.g., pamphlet, research study, water treatment plant and training session.

Performance

What a government did with its resources to achieve its results, how well those results compare to what the government intended to achieve and how well lessons learned have been identified.

Performance Indicator / Performance Measure

A qualitative or quantitative means of measuring an output or outcome, with the intention of gauging the performance of an organization, program, policy or initiative respecting expected results. Quantitative performance measures are composed of a number and a unit. The number provides the magnitude (how much) and the unit gives the number its meaning (what). E.g., number of written complaints received.

Performance Measurement

The process and systems of selection, development and on-going use of performance measures to guide decision-making.

Performance Monitoring

The ongoing, systematic process of collecting, analyzing and using performance information to assess and report on an organization's progress in meeting expected results and, if necessary, adjust to ensure these results are achieved.

Performance Reporting

The process of communicating evidence-based performance information. Performance reporting supports decision-making, accountability and transparency.

Program

A group of related activities that are designed and managed to meet a specific public need and often treated as a budgetary unit.

Results

Refer to the above definition for Outcomes.

Qualitative Quantifiers Used in This Report

A portion of the information presented in this report was drawn from a relatively small number of individuals and examples; and, not from a statistically representative sample. To avoid the perception that the information is statistically representative, while at the same time providing a quantitative indication of the responses and findings, five quantitative-adjectives are used:  

• All . . . 100% of respondents, observations and/or findings;
• Majority / Most / Large . . . At least 75% but less than 100% of the respondents or observations;
• Many / Much . . . At least 50% but less than 75% of the respondents or observations;
• Some . . . At least 25% but less than 50% of the respondents or observations; and,
• Few . . . At least two respondents but less than 25% of the respondents or observations.

Updated Report

The final report was initially submitted and presented to CanNor on 16 August 2017. In March 2018, CanNor provided the evaluation team with additional information on core-funding. This information has been reviewed by the evaluators and addressed in this updated report; resulting in revisions to the conclusions and recommendation on core-funding; with the major change being clearer identification of core-funding eligibility for Indigenous organizations under NAEOP.

Purpose and Scope

Started in December 2016, the purpose of this formative evaluation was to:

• Review the implementation and impacts of Harmonization and the Transition to NAEOP;
• Identify areas for program administration and delivery improvements;
• Examine the need for core-funding; and,
• Determine CanNor's readiness for the 2018-2019 summative evaluation of NAEOP.

This formative evaluation does not identify, attribute or quantify program results. A summative evaluation, which will be conducted at a future date by CanNor, will focus on the degree to which the desired outcomes specified within the Logic Model have been achieved; and, the extent to which the policy and program have contributed to the achievement of program outcomes.

The scope of the evaluation covers the five-years of CanNor Indigenous economic development programs in the Territories:

• AED programming from 2011-2012 to 2013-2014); and,
• NAEOP programing for 2014-2015 and 2015-2016.

Methods and Methodological Challenges

The methods employed during the evaluation included:

• A Document Review of key policy, program and other documents identified by CanNor;
• 63 Project File Reviews of randomly selected project files, 21 per Region;
• 96 Key Informant Interviews;
• 18 Project Profiles, 6 per Region; and,
• Site visits in Nunavut, Yellowknife, Whitehorse and Ottawa.

Several methodological challenges were encountered which resulted in complications and delays in completing the evaluation:

• Harmonization and Transition Objectives Not Clearly Defined: During the Document Review page eight of the 2012 Vision for the Harmonization of NAEOP, Renovation in the North [2012 Vision] was identified as the base document which described the purpose and intent of Harmonization. During the planning phase of the evaluation, the 2012 Vision document was used to prepare the Evaluation Objectives, Evaluation Questions and Research Questions which formed the basis for the development of the data gathering instruments, including the interview guides and questionnaires.
  
  During the Key Informant interviews in March 2017, two documents were found which had not been identified and included within the Document Review:
  • Program Renewal, Staff Briefing, March18-19, 2014, a copy of which is presented in Appendix 3.
  • CanNor Aboriginal Economic Development Program Harmonization Transition Plan, a copy of which is presented in Appendix 6.
  These documents demonstrated that the 2012 Vision document was not an accurate or complete reflection of the purpose and intent of the 2014 Harmonization and the Transition to NAEOP.
  
  The new information resulted in:
  • The data gathering and analysis conducted prior to identifying the two documents in March 2017 being revisited;
  • The Evaluation Objectives and Evaluation Questions were rewritten;
  • Additional Key Informant interviews being conducted;
  • Additional research, interviews and analysis was conducted which resulted in identifying new findings; and,
  • The completion date for the evaluation was extended, primarily to accommodate the additional work.
• Performance Measurement Strategy Is Too Focused on Administrative Outputs: CanNor's August 2013 Performance Measurement Strategy (PMS) is too focused on administrative outputs rather than program outcomes. Only one of the 15 performance indicators identified in the PMS addresses program outcomes; while the other 14 performance indicators address administrative outputs. Also, the performance indicators are not quantitatively measurable.
• Project Files Lack Results-based Information: The review of the project files did not produce much of the sought for information. As a result, it became necessary to increase the number of Key Informant interviews from 57 to 104, an 82% increase. Some of the sought for information was identified.

Conclusions and Recommendations

This formative evaluation addressed seven Evaluation Objectives [EO] and produced conclusions and recommendations for each.

Evaluation Objective #2: Effectiveness

Has NAEOP resulted in improved administrative effectiveness of delivery? [Effectiveness]

Conclusion: Harmonization and the transition to NAEOP did not produce any measurable levels of increased effectiveness as some key implementation tasks have not been completed.

Recommendation: Use the updated Transition Plan to guide the completion the implantation of NAEOP from an effectiveness perspective.

Section 3. Purpose, Methods and Evaluation Objectives and Questions

This section:

• Identifies the purpose and scope of the evaluation;
• Presents the methods employed to conduct the evaluation and the methodological challenges that were encountered; and,
• Presents the Evaluation Objectives and Evaluation Question.

3.1 Purpose and Scope

This formative evaluation, also called an interim or mid-term evaluations, examined Harmonization and the implementation of the Northern Aboriginal Economic Opportunities Program (NAEOP). Harmonization is described by CanNor as the merging of the four AED (Aboriginal Economic Development) programs into NAEOP:

• AED was composed of four programs: ABDP (Aboriginal Business Development Program), CEDP (Community Economic Development Program), CEOP (Community Economic Opportunities Program) and CSSP (Community Support Services Program); and,
• NAEOP is composed of two streams: CROP (Community Readiness and Opportunities Planning Fund) and EBD (Entrepreneurship and Business Development Fund)

The purpose of this formative evaluation is to:

• Review the implementation and impacts of CanNor's Harmonized NAEOP programs;
• Identify areas where there is potential for program administration and delivery improvements;
• Examine the need for core funding;^{Footnote 2 Footnote 3} and,
• Identify areas where changes might be needed in preparation for a future summative evaluation.

The broad objectives of this evaluation are:

• Identifying the status of Harmonization and the implementation of NAEOP;
• Identifying the impacts of Harmonization; and,
• Proposing changes to the implementation of NAEOP and program delivery that will increase the efficiency, effectiveness and expected results.

The evaluation covers the five-years of CanNor Indigenous economic development program in the Territories:

• AED programming from 2011-2012 to 2013-2014; and,
• NAEOP programing for 2014-2015 and 2015-2016.

This formative evaluation does not:

• Identify, attribute or quantify program results. A summative evaluation, which will be conducted at a future date by CanNor, will focus on the degree to which desired outcomes specified within the Logic Model have been achieved; and, the extent to which the policy and program have contributed to the achievement of outcomes.
• Address CanNor's other programs: Strategic Investments in Northern Economic Development (SINED); Northern Adult Basic Education Program (NABEP); Community Infrastructure Improvement Fund (CIIF); nor the work being done by CanNor to promote the inclusion and use of official language minority communities.

3.2 Methods Employed and Challenges Encountered

This Sub-section describes each of the evaluation methods employed during this evaluation; the methodological challenges that were encountered, if any; and, the solutions that were adapted and used during the evaluation.

3.3 Evaluation Objectives and Evaluation Questions

To develop the Evaluation Objectives [EO] and Evaluation Questions [EQ] for this evaluation, the first part of the Document Review sought to identify the objectives and expected results of Harmonization and the implementation of NAEOP in April 2014.

To help ensure the rigor and accuracy of this evaluation, documents were sought that met the following selection criteria:

• Should predate the April 2014 NAEOP implementation date;^{Footnote 4}
• Describes the objectives and expected results of Harmonization 
• Have been approved by CanNor's management team; and
• Have been widely circulated within the organization being evaluated, thus ensuring that there is a shared understanding within the organization of the program's objectives and expected results.

The Document Review, using documents provided by CanNor, found that:

• There are no Treasury Board submission documents to refer to as NAEOP was implemented under Ministerial Authority;  
• There are no approved logic models nor evaluation matrices for NAEOP, nor for its predecessor, AED; and,^{Footnote 5 Footnote 6}
• The 2014 Performance Measurement Strategy was incomplete and focused on administrative outputs rather than program results.

Lacking other resource documents, CanNor identified page eight of the 2012 Vision for the Harmonization of NAEOP, Renovation in the North, as the base document to be used during the evaluation, a copy of which is presented in Appendix 2 of this report.^{Footnote 7}  The 2012 Vision for the Harmonization of NAEOP, Renovation in the North was used to develop the original EOs and EQs which guided the development of data gathering questionnaires and guides. The original EOs and EQs are presented within the detailed workplan that was produced at the start of the evaluation.

In March 2017, during one of the final Key Informant interviews, two documents were identified which had not been found and included within the initial Document Review:

• Program Renewal, Staff Briefing, March18-19, 2014, a copy of which is presented in Appendix 3.  
• CanNor Aboriginal Economic Development Program Harmonization Transition Plan, a copy of which is presented in Appendix 6.

Based upon the information provided in the briefing and transition plan documents, the original EOs EQs were modified in April 2017 to the following:

• EO1: What were NAEOP's original and current management and program objectives and the expected benefits to be achieved from the Harmonization of the four AED programs (ABDP, CEDP, CEOP and CSSP) into NAEOP? [Program Design and Changes]
  • EQ1.1: What is Harmonization and was it implemented?
  • EQ 1.2: What were the eligibility requirement changes that were brought about by the Harmonization of the four AED programs into NAEOP?
  • EQ1.3: What are the expected-results of Harmonization and the transition to NAEOP, and were they achieved?
  • EQ1.4: What was the implementation plan for Harmonization and the Transition to NAEOP and was it successfully completed?
  • EQ1.5: What are the organizational and administrative changes that were adopted by CanNor for the Harmonization of the four AED programs into NAEOP?
• EO2: Has NAEOP resulted in improved administrative effectiveness of delivery? [Effectiveness]
  • EQ2.1: Did the Harmonization of AED programs into NAEOP achieve more effective administrative delivery of programming?
  • EQ2.2: What are the external perceptions of Harmonization and the transition to NAEOP?
• EO3: Has NAEOP resulted in increased administrative and operational efficiencies? [Efficiencies]
  • EQ 3.1: Has the Harmonization resulted in improved program administration efficiency?
  • EQ3.2: Has Harmonization reduced the reporting burden; benefiting CanNor staff as well as funding recipients?
• EO4: What are the direct and indirect impacts of the Harmonization of the four Indigenous economic development programs upon program delivery? [Impact]
  • EQ4.1: Did the realignment of funding during the Harmonization of Indigenous economic development programs into NAEOP increase support for business over the 2012 level of 20%?
  • EQ4.2: Did Harmonization change the numbers and types of funded projects?
• EO5: What has been the impact of reduced core-funding and its planned elimination?
  • EQ5.1: Has the shift from core-funding to proposal-based funding impacted upon the ability of Indigenous communities and organizations to access economic development funding?
  • EQ5.2: Has the shift from core-funding to proposal-based funding impacted upon the ability of CanNor and its partners to deliver economic development funding?
• EO6: Is NAEOP aligned with current government priorities and federal initiatives? [Relevance].
  • EQ6.1: What are the Canadian Government's economic development properties?
  • EQ6.2: Is NAEOP aligned with current government priorities and federal initiatives relative to Indigenous economic development, in general, and more specifically north of the 60th parallel?
  • EQ6.3: Is NAEOP aligned with CanNor's mandate and priorities?
• EO7: What actions are required to improve program performance and prepare for the summative evaluation? [Program Improvements].
  • EQ7.1: What needs to be done to measure program results?
  • EQ7.2: What needs to be done to improve program performance?
  • EQ7.3: Is CanNor prepared for the summative evaluation to be conducted in 2018?
  • EQ7.4: What strategy could be employed to optimize measuring results and improving program performance?

4.1 CanNor, the Organization and Its Governance Structure

Reporting to the Minister of Innovation, Science and Economic Development, CanNor was established in 2009 to foster growth and development in the North by delivering economic development programs; and, by:^{Footnote 8}

• Collaborating with and aligning the efforts of partners in northern and southern Canada to respond to economic challenges and opportunities in the North.
• Help develop a diversified, sustainable, and dynamic economy across Canada's three territories by working with communities, community- based businesses and local entrepreneurs to develop and diversify local economies; and, take advantage of the strengths of Northern Canada.

With a staff of 87 fill-time equivalent positions in the Territories and Ottawa, CanNor's governance structure is supported by four committees:

• Senior Management Committee;
• Policy, Planning & Operations Committee;
• Extended Management Committee; and,
• Resource Management Committee.^{Footnote 9}

CanNor's headquarters is in Iqaluit, with regional offices in Whitehorse, Yellowknife, Iqaluit and a liaison office in Ottawa. CanNor is one of six Regional Development Agencies (RDAs) responding to the Minister of Innovation, Science and Economic Development Canada. The other RDAs are:

• Atlantic Canada Opportunities Agency (ACOA);
• Canada Economic Development for Quebec Regions (CED);
• Federal Economic Development Agency for Southern Ontario (FedDev Ontario);
• Federal Economic Development Initiative for Northern Ontario (FedNor); and,
• Western Economic Diversification Canada (WD).

CanNor received its suite of programs from INAC in 2009. These programs provide funding to support the development of key economic sectors such as mining, tourism, fisheries, cultural industries and business development. The programs received by CanNor in 2009 included:

• Aboriginal Economic Development (AED);^{Footnote 10}
• Strategic Investments in Northern Economic Development (SINED);
• Northern Adult Basic Education Program (NABEP);
• Community Infrastructure Improvement Fund (CIIF); and,
• Promotion of official language minority communities.

4.2 Indigenous Economic Development Programs

Through NAEOP, CanNor supports increased economic participation by northern Inuit, First Nations and Metis communities and businesses. Support under NAEOP is complementary to the support provided through programs of general application and INAC’s programs of national application. NAEOP comprises two program streams:

• Community-Based: Community Readiness and Opportunities Planning (CROP) provides project-based support to First Nations, Inuit and Metis communities and organizations, including organizations providing business development, financial or economic development services under this authority, for capacity and readiness activities so that communities can plan for, pursue and participate in economic opportunities. 
• Business (Entrepreneur)-Based: Entrepreneurship and Business Development (EBD) assists Indigenous entrepreneurs, Indigenous businesses and Indigenous organizations in the pursuit of economic opportunities for the benefit of Indigenous people. The stream includes project-based support for activities that facilitate the establishment or expansion of Indigenous businesses. 

4.3 Budgets and Expenditures, Combined AED and NAEOP

CanNor funding for 2011-12 and 2012-13 was set at $11,800,000 per year. Beginning in 2013-14, funding was decreased by $1 million per year to $10,800,000 per year.

For the five-year period 2011-2012 through to 2015-2016, CanNor has delivered 351 economic development projects valued at an average of $137,662; ranging from a low of $4,500 for the valuation of a business to a high of $1,375,300 for core support for the provision of community economic development services and advocacy.

5.1 Harmonization, Two Perspectives [EQ1.1]

Harmonization is the term employed by CanNor for the integration of the four pre-April 2014 AED programs (ABP, CEDP, CEOP and CSSP) into NAEOP, a single program which is composed of two streams CROP and EBD. The following table presents the policy perspective of Harmonization.

From a technical perspective, NAEOP is correctly defined as being a single program composed of two funds (two streams), CROP and EBD. CanNor, like other federal departments and organizations, have taken smaller programs and integrated them within a larger ‘umbrella' program; with the previous smaller programs (if retained as separate entities with different eligibility criteria and target populations) being called funds, streams, initiatives etc.

Ample evidence exists that Harmonization's policy objective of merging the four AED programs in a single NAEOP program was completed. AED no longer exists and some of the key NAEOP policy documents have been developed and implemented. CanNor is now reporting on a single program rather than four AED programs.

Some of the interviewed CanNor staff stated that, while it is true that there is a single set of Ts&Cs for both Streams, from an operation's perspective, calling NAEOP a single program with two streams is not a fully accurate description as:

• CROP and EBD have different program guidelines;
• They are delivered somewhat differently;
• They target different recipient groups; and,
• ABDP was largely unchanged and had been "rebranded" as EBD.

It was suggested that a better operation's presentation and understanding of the impacts of Harmonization could be achieved by describing the transition as:

• CEDP, CEOP and CSSP(community-based) were Harmonized (merged and updated) into CROP; and,
• ABDP evolved into EBD (business-based), with minor updates, to align some eligibility requirements with those found in CROP.

In reviewing the two perspectives of Harmonization it became evident that the policy and operations perspectives of Harmonization have merit and that this formative evaluation and the future summative evaluation of NAEOP would be better served if both perspectives were considered during the evaluations. However, it is noted that the policy and the operational perspectives of Harmonization provide useful information and insights; and, employing both perspectives provides a richer and more comprehensive understanding of Harmonization and its impacts.

For example, when examining the decision to produce a single Project Assessment Report (PAR) from the policy perspective produces quite different findings than if the operations perspective of harmonization is employed:

• Policy Perspective: A single PAR is more administratively efficient as a single tool and guide is used for both CROP and EBD, simplifying the process and reducing staff training.
• Operational Perspective: A single PAR for two different programs with somewhat different information requirements make the document repetitive and forces the applicant to say essentially the same thing in two different ways.

5.2 Eligibility Requirements [EQ1.2]

A crosswalk analysis of AED's four sets of Ts&Cs (ABDP, CEDP, CEOP and CSSP) and NAEOPs two sets of T's and Cs (EBD and CROP). The crosswalk was informed by CanNor's 2013 crosswalk, Harmonization of CanNor's Aboriginal Economic Development Programs.^{Footnote 13}  

It was found that NAEOP's Ts&Cs are more detailed and prescriptive then those under AED; making a direct comparison difficult; however, it was possible to determine that:

• Eligible Recipients:
  • Organizations that were eligible for funding under AED were also eligible under NAEOP;
  • Charitable and religious organizations are specifically excluded under NAEOP, which was not the case under AED; and,
  • NAEOP did not add other potential recipients that were not previously identified under AED.
• Eligible Projects:
  • NAEOP's Terms and Conditions allows for the funding of all project types that were available under AED; and,
  • While NAEOP does not employ the term core-funding, core-funding is permitted under CROP for funding recipients who were receiving population-based funding under the previous Community Economic Development Program (CEDP).

Under AED funding could be provided to an organization (i.e. an NGO supporting artists) where the benefit accrued to Indigenous Artists. NAEOP's updated T&Cs are more prescriptive, stating that the organization's Board of Directors would have to be mainly Indigenous to be eligible. This can be problematic if a board changes mid-stream through a project and the Indigenous-majority of the Board is lost, the organization can become ineligible and the project's funding potentially jeopardized.

An important clarification is that while NAEOP's Ts&Cs identify eligible recipients and projects, CanNor has the authority to prioritize and focus on certain groups or sub-sets of the identified eligible recipients and project types. Therefore, CanNor does not have to agree to fund an organization even though it is identified as an eligible organization.

5.3 Harmonization's Expected Benefits [EQ1.3]

The March 2014 briefing to CanNor staff identified the expected results of Harmonization:

• Improved program management efficiency through:
  • a single-window approach; and,
  • standardized program management and administration processes.
• Increased flexibility and simplified process for recipients through, for example:
  • a single application process; and,
  • multi-year agreements.

5.4 Implementation (Transition) Plan [EQ1.4]

CanNor's Aboriginal Economic Development Program Harmonization Transition Plan was drafted by CanNor's Policy Committee on March 1, 2013 and updated January 16, 2014. A more current version of the document was not found and it is not known if the Transition Plan had been submitted to Senior Management Committee for review and/or comments. The Transition Plan is presented in Appendix 6 to this document.

The Transition Plan identified and tracked many tasks including:

• Updating the Project Management Control Framework and Performance Measurement Strategy;
• Developing assessment criteria, guidelines, information reporting requirements, and communications documents for the identification of new third-party delivery partners;
• Updating administrative documents and guidelines;
• Phasing out CEDP core-funding; and,
• Signing multi-year agreements where possible.

The Transition Plan was spread over four fiscal years, from 2013-14 to 2016-17. The objectives for each of the four fiscal years were identified as:

• Objectives Year 1, 2013-14:
  • Develop and put in place program materials, processes and systems
  • Develop a third-party delivery strategy / approach (e.g., existing arrangements, potential pilots with new delivery organizations)
  • Identify program delivery needs (e.g., capacity, guidelines)
  • Manage stakeholder communications and expectations
  • Develop a formal understanding with AANDC on access to national programs
• Objectives Year 2, 2014-15:
  • Launch new program forms and assessment process
  • Work with candidates for third-party delivery partnership towards formal agreements
  • Develop strategies / mechanisms to address identified program delivery needs
• Objectives Year 3, 2015-16:
  • Manage delivery of the new program
  • Sign enhanced third-party delivery agreements where possible
• Objectives Year 4, 2016-17:
  • Manage delivery of the new program
  • Manage final year of CEDP phase-out
  • Final year of CEDP phase-out – develop necessary communication strategies / products to manage end of transition

The implementation status reports for the Transition Plan stopped at the end of Year 1, 2013-14; with no status updates provided to the evaluators for the three remaining years.

CanNor staff in the Territories and Ottawa agreed that Transition to NAEOP was successful, and that other than for the issues surrounding the decision to eliminate core-funding, few major problems were encountered. Staff noted that some tasks identified for Years 2 through 4 (2014-15 to 2016-17) of the Transition remained incomplete; and, if they were completed, administrative efficiency and effectiveness could be improved:

• Completed tasks:
  • Program Guidelines;
  • Terms and Conditions;
  • Program Management Control Framework;
  • Chart of Accounts; and,
  • Update communication tools and website.
• Tasks requiring further attention:
  • Program Performance Measurement Strategy;
  • Logic Model; and,
  • Forms, Guides and Interpretations.

5.5 Organizational and Administrative Changes [EQ1.5]

In 2012, the Federal Government introduced its Deficit Reduction Action Plan (DRAP), at about the same time that CanNor was preparing to Harmonize and Transition to NAEOP. The combined effect of DRAP and NAEOP resulted in the reduction of two positions and the reorganization of staff within the four CanNor offices.

The reorganization of CanNor staff resulted in a partial reallocation of responsibilities amongst staff; however, the way CanNor administered and delivered AED, and then NAEOP, did not change significantly. As one CanNor staff member stated; it was all the same work, just done by different people.

Most of NAEOP's administrative guidelines and interpretations were inherited from INAC when the transfer of AED occurred in 2009. During discussions with CanNor staff it was noted that:

• The guidelines were incomplete and that the interpretations often lack sufficient direction to be of use to economic development officers;
• Work had been initiated in 2015-2016 to complete the guidelines and improve the usefulness of the interpretations as well as developing a crosswalk between those documents and NAEOP's Terms and Conditions. That work has not been completed primarily because of a lack of time due to other work priorities; and, the position of DG Operations, who has the authority to review and approve the guidelines and interpretations had been vacant for a considerable period; and,
• The lack of current approved guidelines and interpretations was on occasion making it difficult to guide funding applicants on project eligibility and eligible costs.

6.1 Effective Delivery of NAEOP [EO2.1]

During the Key Informant Interviews and the Project File Reviews, information was sought that would indicate whether program delivery had become more effective with Harmonization and the introduction of NAEOP. Unfortunately, the required information was not identified nor gathered and the project files provided no indicators on effectiveness. The evaluation was unable to determine if program delivery effectiveness had increased.

Most of NAEOP's administrative guidelines and interpretations were inherited from INAC when the transfer of AED to CanNor occurred in 2009. During discussions with CanNor staff it was noted that:

• The administrative guidelines were incomplete and the interpretations to those guidelines often lacked sufficient direction to be of use to economic development officers;
• Work had been initiated in 2015-16 to complete the guidelines and improve the usefulness of the interpretations as well as developing a crosswalk between those documents and NAEOP's Terms and Conditions. That work has not been completed primarily because of a lack of time due to other work priorities; and, the position of DG Operations, who has the authority to review and approve the guidelines and interpretations had been vacant for a considerable period; and,
• The lack of current approved guidelines and interpretations was on occasion making it difficult to guide funding applicants on project eligibility and eligible costs.

Some CanNor staff noted that updating and completing the guidelines and interpretations; and, providing training on their use and interpretation, would likely produce an increase in program delivery effectiveness.

It can be argued that having one community-based program (CROP) versus the three AED community-based programs should result in increased administrative effectiveness. During discussions with the different Key Informant groups, some noted it was no longer necessary to spend time to determine which of the three different AED community-based programs would be the most appropriate for the project being proposed to CanNor. CanNor staff stated that there may be minor time savings but not enough to impact upon program delivery effectiveness.

6.2   External Perceptions of Harmonization and Transition to NAEOP [EO2.3]

7.1 Impact of Harmonization upon Program Administration Efficiency [EO3.1]

The consensus of CanNor operations staff and executive management is that the Transition of NAEOP did not measurably reduce their internal administrative and reporting burdens nor that of the funding recipients. Some of the key observations were:

• The Transition was initiated but many of the tasks identified in the Transition Plan that would have resulted in administrative and reporting efficiencies have not been completed.
• While the introduction of multi-year projects did reduce the number of projects being administered; multi-year projects are significantly more complex than single-year projects, requiring considerably more work (on a per project basis) with the funding applicants to get the project application to a point where it can be approved.
• While it is true that multi-year projects require only one project review and approval, all other administrative requirements must be produced on an annual basis. There is a shared perception that the administrative requirements for multi-year projects can be further streamlined so that the burden is reduced.
• For single-year projects, application review and funding processes require that economic development officers go through the same steps regardless of the projects' size, complexity, dollar value, and the levels of risk.
• The updates to the funding applications and the Project Assessment Report (PAR) did not adequately address the problem that the funding applications do not provide all the information needed by CanNor economic development officers to complete PARs. This means that the economic development officer must return to the funding applicant and seek additional information which could have been provided within the funding application.
• The administrative burden, for both applicants and economic development officers, could be reduced with the availability of templates that would assist in the production, for example, of funding applications and end of project reports.

CanNor policy staff noted that their administrative burden had been reduced slightly, possibly by a few days per year, because there were fewer programs to report on and fewer guidelines and administrative documents to maintain.

7.2 Reporting Burden, Funding Recipients [EQ3.2]

Some CanNor staff stated that many funding recipients had told them that:

• Reporting is too time consuming and that funding recipients are not accustomed to producing reports that included results-based information;
• Since CanNor does not fund the time for the recipients to produce the reports, some recipients have stated they do not have the money needed to pay staff or contract someone to write the reports; and,
• CanNor did not provide sufficient direction on what had to be reported on. This was proving frustrating to funding recipients, especially those with little report writing experience and capacity, because they had to go through several iterations of a report before CanNor would approve it.

Many funding recipients stated that end-of-project reporting requirements are time consuming and that:

• The level of effort and detail required in the reports seems to vary amongst different CanNor and ASDP staff;
• Smaller low-risk projects seem to require as much reporting as do larger complex higher-risk projects; and,
• Time is wasted because CanNor does not have a standardized set of reporting templates that can be used. End of project reporting templates would reduce the burden of reporting for applicants as they would know what information is being sought by CanNor

When asked whether Harmonization had streamlined reporting and reduced the reporting burden, most CanNor operations staff were unable to identify any significant improvements or changes to the end-of-project reporting processes.

8.1   Focus on Business Support [EQ4.1]

The EBD Guidelines published by CanNor in 2014 state that the goal is to Increase the number of sustainable Aboriginal businesses and provide a supportive business environment for Aboriginal people in the territories.

There is a shared belief amongst some CanNor staff that 70% of NAEOP funding is to be directed to business-based projects; largely based upon possible program design changes that were discussed between 2012 and 2014. However, no policy statements or other documents were identified that specifies a performance target or states what percentage of NAEOP funding is to be directed to business-based projects.

CanNor is tracking and reporting on the percentage of projects that are business-based, with that number increasing from about 20% in 2012-13 to about 30% in 2015-16. CanNor staff could not identify a conscious effort to seek-out or provide preference to business-based projects; nor could they explain why there was an increase in business-based project funding versus community-based projects. A couple of CanNor staff speculated that the introduction of multi-year funding favoured business-based projects; however, that could be neither confirmed or denied.

When asked if it was feasible to increase the percentage of funding directed to Indigenous-owned businesses projects in each of the Territories, CanNor staff responded yes, but only to a limited extent. CanNor's economic development staff noted that each Territory and the regions within the Territories are quite different from each other. NWT has larger cities and towns, by northern standards while Nunavut has smaller community populations and there is a stronger tendency towards community-based business than entrepreneur-based businesses. The Yukon is situated somewhere between NWT and Nunavut when it comes to economic development.

CanNor staff stated that to increase the percentage Indigenous-owned business funded projects would require the development of a focussed approach to favour business-based projects and a change to program criteria. They also stated that the targeted percentage for business-based would have to be lower than the 70% target that was envisioned in 2012; and, that the targets would have to be adjusted by Territory with NWT probably having the highest potential for increased business-based project funding, while Nunavut and the Yukon would require lower targets for business-based funded projects.

8.2 Numbers and Types of Funded Projects [EQ4.2]

The project file review of 63 AED and NAEOP projects did not result in any recognizable trends in the types of projects being funded by CanNor. However, with the introduction of multi-year funding under NAEOP in 2014 there is a trend to longer projects (two to four years) that would in the past have been undertaken under several projects rather than one longer term project; which incidentally is also increasing the dollar value of projects.

Projects were generally diverse in nature: (i) technical, feasibility, financial, marketing and environmental studies; (ii) training, capacity and human resource development; (iii) community infrastructure; and, (iv) business capital infrastructure. Most projects were related to mining, tourism, fisheries, cultural industries and business development.

The interviews of the CanNor managers and officers of the 63 project files that were reviewed observed that:

• Many CanNor funded projects are one part of a larger, more complex project which includes many different funding cycles; some of which CanNor may be involved with and others where CanNor is not involved.
• Multi-year project funding has increased the size and duration of the project and can include several sub-projects, each of which would likely have been funded under single-year project funding.
• No pattern, other than multi-year project funding, was detected which changed either the composition and/or characteristics of the projects being funded by CanNor.

9.1 Core-Funding and Allocation Based-Funding

CanNor received INAC's suite of programs from INAC in 2009; with the TS&Cs remaining mostly unchanged. Part of INAC's ‘programming approach' was to provide core-funding to build economic development capacity within the communities, especially those representing smaller communities in more remote locations that lack the capacity to effectively plan for and support economic development. 

In 2013-2014, CanNor made the decision to progressively shift the funding from allocation-based funding (core funding) toward project-based; however, while general statements were made about the funding shift, no detailed implementation plan or formal policy were developed to specify how this would happen.

The non-CanNor groups of key informants had difficulty understanding the purpose and intent of CanNor's shift from using an ‘allocation-formula' to allocate core-funding.

As described in CanNor's 2015-16 Departmental Performance Review^{Footnote 16}, with the phasing-out of CEDP, CanNor in 2014 started moving in the direction of supporting proposal-driven projects under CROP. Under CROP^{Footnote 17}, funded projects can include core-like-funding to Indigenous community-based organizations; with CROP's Program Guidelines stating that funding can be provided to increase the capacity within the community government to seize future economic opportunities.

While many CanNor staff did not make a distinction between core-funding and core-like-funding, some CanNor staff made a general distinction between core-funding and core-like-funding in the following manner:

•  Core-funding under CEDP provided fixed annual amounts for economic development activities. With the implementation of NAEOP CanNor decided to reduce core-funding to Indigenous community-based economic development organizations by 25% per year to allow those organizations to adapt to the proposal-driven approach.
•  Core-like-funding under CROP would allow the funding of activities that would support Indigenous community-based economic development organizations that lack economic development capacity.

During the Document Review and the Key Informant Interviews it was observed that the use of core-funding is often used in correspondence and other documents relative to CROP as well as in correspondence to ASDPs and others. The terms core-funding and core-like-funding are not defined or used in the Program Guidelines. This has lead to some confusion.

9.2 Access to Core-funding [EQ5.1]

Core-funding supports community economic development services and organizational capacity and is provided through NAEOP's CROP fund. The core-funding budget for 2016-17 is $1,214,181, going to 27 different Indigenous economic development organizations:

• 3 organizations in Nunavut received $779,418;
• 21 organization in NWT received $376,110; and,
• 3 organizations in the Yukon received $58,653.

Some CanNor staff stated that technically, ASDPs do not receive core-funding; rather, they do receive core-like-funding through CROP to fund core-like activities to better support the organization they work with. For example, core-like-funding allows the ASDPs to hire additional economic development officers, providing the ASDP's with the resources needed to identify economic development opportunities and develop funding strategies to undertake those projects. Another advantage of core-like-funding to ASDP's is that their work in multiple communities allows them to better identify, plan and coordinate larger projects that have regional coverage. 

CanNor economic development officers generally agree that core-like-funding to ASDPs is a positive support to economic development in the North. During the Project File Review, no factual results-based information was found that supported or refuted the usefulness of core and core-like-funding. Key Informant Interviews provided a diverse set of opinions about the usefulness of core and core-like-funding; mostly anecdotal, with few demonstrated facts to support their beliefs.

Some CanNor staff did not agree that core-like-funding to Indigenous Economic Development organizations is a positive support to economic development in the North.

The evaluation sought to identify positive results of providing core-funding and/or core-like-funding. The Project File Review and the Project Profiles was unsuccessful in identifying benefits and/or results that could be attributed to the provision of core-funding and/or core-like-funding. During the Key Informant Interviews CanNor staff agreed that information could be identified and gathered to allow for a results-based analysis.

Many of the Key Informants agreed core-funding was essential to the design, approval and implementation of successfully funded projects, citing that the applicants often did not have the capacity to undertake the economic development activity without help.

9.3 Shift from Core-funding to Proposal-based Funding [EQ5.2]

There have been problems in implementing this approach. A few CanNor staff and CROP funding recipients staff felt that the concept of proposal-based funding was insufficiently developed and that more detailed guidelines and templates may allow for the shift away from allocation-based core-funding.

Some CanNor and CROP recipient staff noted that proposal-based funding is or could be a way to:

• Have Indigenous economic development organizations develop results-based work plans that identify what and how economic development capacities are to be developed; and,
• Require those organizations to submit end-of-project reports, which is not a requirement when core-funding is provided to communities.

One of the identified major impediments to implementing proposal-based funding is the lack of clear directives and guidelines that would apply, as well as templates and training for CanNor staff who work with funding applicants and Indigenous community-based economic development organizations.

Concern was raised by the Indigenous Economic Development organizations and CanNor staff that submitting proposal-based funding applications under CROP to carry out core-like activities would require organizations to pay for at least 20% of the total cost of the project. Two views were expressed:

• The proposal-based funding approach is to replace core-like-funding that had been previously provided by CanNor. Since the applicant is seeking the core-like-funding to develop economic development capacity; * this is seen by some Indigenous economic development organizations and some CanNor staff as CanNor requiring the Indigenous Economic Development organizations to support their internal capacity by leveraging 20% from other sources, which is perceived as being unfair.
• The Indigenous Economic Development organizations that are or could be seeking proposal-based funding to replace core-like-funding are, for the large part, Inuit, First Nations and Metis organizations who have been funded from a Land Claim or from other federal, territorial or Indigenous organizations. A few CanNor staff stated that requiring a 20% contribution from the ASDPs might be appropriate.

When asked about the elimination of CEDP core-funding, some CanNor staff stated that the proposal-based approach was insufficiently developed and had been too hastily implemented; and, that CanNor should at, a minimum, delay the elimination of the allocation-based  core-funding until the proposal-based approach is fully thought out; after which, the core-funding decision can be made.

Most of the interviewed key informants with Indigenous Economic Development organizations were against the shift from the previous CEDP core-funding to a proposal-based approach. Some noted that many of the essential services they provide are paid for by the allocation-based funding they receive.

10.1 Government of Canada's Economic Development Priorities [EQ6.1]

The Government of Canada's commitment to economic development priorities has been outlined in two key documents:

• The Federal Government's 2017 Budget, which continues to support the economic priorities identified within the 2016 Budget, states that the Government:^{Footnote 18}
  • Supports innovation in key growth industries—clean technology, digital and agri-food—with new measures that will improve access to financing, encourage investment, support the demonstration of technologies and build the capacity necessary for Canadians to take advantage of growth opportunities and create good, well-paying jobs.
  • Continues the Government's plan to invest in the middle class and in the long-term productive capacity of the Canadian economy.
  • Invests in transformation Canadian communities, create middle class jobs and generate clean, sustained growth.
  • Will takes the next step in the Government's long-term economic plan, understanding that in the face of unprecedented change, a confident Canadian middle class will always be the beating heart of our country and the engine of our economy.
  • Continues its investments in a Strong Middle Class. Budget 2017 builds on the Government's previous actions, and proposes smart, targeted investments that will help create the conditions required to foster economic growth; and, grow the middle class.
  • The total value of new proposals announced in Budget 2016 was $11.6 billion in 2016–17. This investment was expected to boost real economic activity by 0.5 per cent in the first year of implementation, reflecting increased government expenditure on infrastructure, new programs, as well as increased transfers to households.
• The Prime Minister's mandate letter to the Minister of Innovation, Science and Economic Development, states that CanNor's Minister will support and invest in economic development:^{Footnote 19}
  • We made a commitment to invest in growing our economy, strengthening the middle class, and helping those working hard to join it. We committed to provide more direct help to those who need it by giving less to those who do not. We committed to public investment as the best way to spur economic growth, job creation, and broad-based prosperity. We committed to a responsible, transparent fiscal plan for challenging economic times.
  • Work with the Minister of Indigenous and Northern Affairs and the Minister of Employment, Workforce Development and Labour to promote economic development and create jobs for Indigenous Peoples.
  • We made a commitment to Canadians to pursue our goals with a renewed sense of collaboration. Improved partnerships with provincial, territorial, and municipal governments are essential to deliver the real, positive change that we promised Canadians. No relationship is more important to me and to Canada than the one with Indigenous Peoples.
  • As Minister of Innovation, Science and Economic Development, your overarching goal will be to help Canadian businesses grow, innovate and export so that they can create good quality jobs and wealth for Canadians. You will achieve this goal by working with provinces, territories, municipalities, the post-secondary education system, employers and labour to improve the quality and impact of our programs that support innovation, scientific research and entrepreneurship. You will collaborate with provinces, territories and municipalities to align, where possible, your efforts.

10.2 CanNor's Economic Development Priorities [EQ6.2]

CanNor's commitment to economic development has been outlined in two key documents:

• The Minister of Innovation, Science and Economic Development issued The Canadian Northern Economic Development Agency – 2017-2018 Departmental Plan states that:^{Footnote 20}
  • CanNor will continue to help develop a diversified, sustainable, and dynamic economy across Canada's three territories. To do so, CanNor will direct contribution funding programs.
  • CanNor will advance northern economic development by supporting projects that contribute to economic growth and diversification.
  • Through the Northern Aboriginal Economic Opportunities Program (NAEOP), CanNor supports increased participation of northern Indigenous communities and businesses in the economy.
  • CanNor will continue to deepen its relationship with partner organizations that deliver Indigenous business funding and services on behalf of the Agency, ensuring that they can continue to assist Indigenous small and medium-sized enterprises (SMEs) to develop into sustainable companies with long-term growth potential.
• CanNor published its Strategic Framework 2013-18, Building a Strong North Together. Some of the key statements within that document include^{Footnote 21}:
  • Vision: A strong, diversified, sustainable and dynamic economy for Northerners, including Indigenous Peoples, communities and businesses across Canada's three territories, that contributes to Canada's prosperity.
  • Mandate: Fostering regional economic development in Canada's three territories by delivering programs; building partnerships to leverage investments in the North; and advocating for the interests of Northerners, including Indigenous Peoples.
  • CanNor supports and collaborates with Indigenous Peoples and communities so that they can leverage opportunities to develop their economies.
  • CanNor's role as a federal regional development agency for the three territories is particularly important as the federal government plays a more direct role in the territories than in the provinces.
  • This will be accomplished through three building blocks: policy and research; leadership and advocacy; and economic development funding programs.

10.3 NAEOP's Alignment with Canada's and CanNor's Economic Development Priorities [EQ6.3]

NAEOP increases the participation of northern Indigenous communities and businesses in economic opportunities.^{Footnote 22} 

• The CROP fund is intended to improve the economic development capacity of Indigenous communities and increase economic development in the three territories. It achieves this by providing financial support to First Nations and Inuit communities, so they can improve their capacity and prepare to plan for, pursue, and participate in economic opportunities.
• The EBD fund provides financial support to Indigenous entrepreneurs, businesses, and commercial entities to expand their business in the three territories. It helps applicants pursue economic opportunities that benefit Indigenous people. The fund is also intended to provide project-based support for activities that help to establish or grow Indigenous businesses.

NAEOP works with communities, businesses and entrepreneurs and CanNor's partners by supporting and funding projects that could lead to economic development. Both CROP and EBD are directly aligned with the Government of Canada's and CanNor's objectives by providing support and funding for the creation of economic development in Canada's North.

11.1 Measuring Results [EQ7.1]

During the Key Informant interviews and the project file review options for the measurement of program results were identified. They are presented in the following sub-sections.

11.2 Improving Program Performance and Delivery [EQ7.2]

During the conduct of the evaluation CanNor staff and the evaluators identified a few proposed changes that should improve program delivery and prepare for the upcoming summative evaluation of NAEOP. The proposed changes are discussed in the following sub-sections.

11.3 Level of Preparedness for the 2018 Summative Evaluation [EQ7.3]

Summative evaluations should focus on the degree to which desired outcomes specified within the PMS's Logic Model have been achieved; and, the extent to which the policy and program have contributed to the achievement of outcomes.

The project file reviews and interviews, as discussed in the previous sections of this report, have confirmed that the PMS and data needed to measure program performance, effectiveness and efficiency does not exist. Without suitable performance measurement indicators and reliable date to measure results, the summative evaluation will most likely not be able to identify program performance, efficiencies and effectiveness.

If CanNor were to undertake the tasks suggested in this report, it will likely take 18 months to complete the work which needs to be done before the summative evaluation could be undertaken.

11.4 Strategy to Measuring Results and Improve Program Performance [EQ7.4]

During the review of the findings presented in this and previous sections of the report, some common threads were identified with most obvious being that most of the tasks are directly linked to the need to develop measurable performance indicators; primarily for outcomes and to a lesser degree for administrative outputs. The three more obvious are:

• Recipient reporting should be focused, in part, on providing the information to measure program outcomes and performance;
• Administrative forms including the PAR should, in part, gather the information about the projects' expected outcomes; and,
• CanNor reporting on performance should be based on measurable approved performance indicators.

When CanNor staff were asked how to undertake and implement the required changes, the summary of the most common responses is:

• The tasks are all important and they are interlinked; they should be done simultaneously as a single comprehensive package. For example, how can you develop a reporting template to gather performance information when the Performance Measurement Strategy requires a major revision to the performance indicators?
• The work should be undertaken by teams composed of policy and operations staff. Participation and input is required from both sides during the entire process.
• With the fast approaching deadline to start the summative evaluation of NAEOP, external resources and/or dedicated staff should be assigned to the task.

One interviewee stated that true change can only come about if all staff embrace change. An undertaking of this magnitude and importance requires a combined Change Management Plan and a Transition (project management) Plan. This approach will reduce staff uncertainty about change as well as encouraging buy-in and acceptance of the proposed changes.

Evaluation Objective #1: Program Design and Changes

What were NAEOP's original and current management and program objectives and the expected benefits to be achieved from the Harmonization of the four AED programs (ABDP, CEDP, CEOP and CSSP) into NAEOP?

Conclusion: Harmonization of AED into NAEOP was successful; the expected benefits of Harmonization have been addressed; and, most of the operational Transition tasks were addressed. All but one of AED's Eligible Recipients and Eligible Projects were retained and included within NAEOP.

The findings that support this conclusion are:

• Harmonization have been successfully implemented; AED has been merged into NEOAP's two program streams.
• Harmonization made no changes to the organizations and individuals who are Eligible Recipients nor to the types of Eligible Projects that can be funded through NAEOP, with the exception that charitable and religious organizations are now excluded.
• There have been some improvements in management, policy and application processes with additional work required to complete the tasks.
• From a policy perspective Harmonization was successful in implementing a single application process.
• From an operational perspective Harmonization was not fully successful because funding applicants would be better served by revised application forms.
• CanNor has successfully implemented multi-year agreements for both business-based and community-based projects.
• Many of the policy-related Transition tasks have been completed while some key tasks need to be addressed.
• The organizational changes within CanNor, at the time NAEOP was introduced, did not appreciable change the way Indigenous economic development funding and support is delivered.
• Work on updating administrative guidelines and interpretations has been initiated but not completed. Administrative efficiencies and increased consistency in interpreting Ts&Cs will be increased by the completion of the guidelines and interpretations.

Recommendation: Develop an updated Transition Plan to complete the implantation of NAEOP. Section 11 proposes some specific tasks that could be included within an updated transition plan.

Evaluation Objective #2: Effectiveness

Has NAEOP resulted in improved administrative effectiveness of delivery? [Effectiveness]

Conclusion: Harmonization and the transition to NAEOP did not produce any measurable levels of increased effectiveness as some key implementation tasks have not been completed.

The findings that support this conclusion are:

• The evaluation did not identify any areas where there had been a measurable increase in the effectiveness of program delivery
• Work on updating administrative guidelines and interpretations has been initiated but not completed. Administrative effectiveness could be increased with the completion of the administrative guidelines and interpretations.
• Harmonization and the transition to NAEOP did not impact upon CanNor's partnerships.
• The Transition to NAEOP was seen mostly as a process internal to CanNor and that there would have negligible impact day-to-day operations.

Recommendation: Use the updated Transition Plan to guide the completion the implantation of NAEOP from an effectiveness perspective.

Evaluation Objective #3: Efficiency

Has NAEOP resulted in increased administrative and operational efficiencies? [Efficiencies]

Conclusion: Harmonization and the transition to NAEOP did not produce any measurable increases in efficiency as some streamlining implementation tasks have not been completed.

The findings that support this conclusion are:

• The administrative burden on CanNor staff has not been reduced by Harmonization and the introduction of NAEOP.
• Harmonization and the implementation of NAEOP has not streamlined the reporting process nor has it reduced recipient's reporting burden.

Recommendation: Use the updated Transition Plan to complete the implantation of NAEOP from an efficiency perspective.

Evaluation Objective #4: Impact

What are the direct and indirect impacts of the Harmonization of the four Indigenous economic development programs upon program delivery? [Impact]

Conclusion: The impact of Harmonization and the Transition to NAEOP did not have any appreciable impacts upon program delivery except for the positive impacts of multi-year funding and the negative impacts from the cuts to core-funding.

The findings that support this conclusion are:

• There has been an increase from 20% to 30% in the number of business-based projects; however, there are no policies or operational strategies that specify a target for the percentage of NAEOP funding to be allocated to businesses nor a date by which that target should be achieved.
• There is potential to increase the percentage of business-base funded projects; and, those targets should be set on a Territory by Territory basis. The target of 70% discussed in the past is probably too high.
• Other than for the introduction of multi-year project funding and the decrease of funding to core-funding, the Transition to NAEOP did not appreciably change the composition and/or characteristics of the projects funded by CanNor. There is a trend towards larger projects with larger budgets since multi-year funding was introduced with NAEOP.

Recommendation: None required.

Evaluation Objective #5: Core-funding

What has been the impact of reduced core-funding and its planned elimination?

Conclusion: There is a consensus that core-funding and core-like-funding improves the ability of Indigenous Economic Development organizations to support economic development.

Conclusion: The proposal-based approach towards core-like-funding is possible through the development of clear directives, guidelines and templates.

The findings that support this conclusion are:

• CanNor does not have a mutually shared understanding of the differences between core-funding and core-like-funding, and core-funding is often used to identify core-like-funding.
• Many CanNor economic development officers agree that core-like-funding to Indigenous Economic Development organizations is a positive support to economic development in the North; while other CanNor staff describe core-funding in a negative manner
• There is agreement that core-funding and/or core-like-funding is effective in promoting economic development; however, CanNor would have to conduct additional data gathering and analysis before a results-based conclusion could be rendered.
• There is some agreement that a proposal-based funding approach which would allow for core-like-funding would work if clear directives, guidelines and templates would be produced.

Recommendation: Maintain core-funding to the Indigenous Economic Development Organizations until a workable proposal-based funding approach or other alternative is developed.

Evaluation Objective #6: Relevance

Is NAEOP aligned with current government priorities and federal initiatives? [Relevance].

Conclusion: NAEOP is relevant and it is aligned with current government priorities and federal initiative.

The findings that support this conclusion are:

• The Government of Canada is committed to supporting and funding economic development and job creation in Canada.
• CanNor supports and operationalized the Government of Canada's committed to support and fund economic development and job creation in Canada's Territories.
• NAEOP is aligned with the Government of Canada's and CanNor's commitment to support and fund economic development and job creation.

Recommendation: None required.

Evaluation Objective #7: Program Improvements and Measurement

What actions are required to improve program performance and prepare for the summative evaluation? [Program Improvements].

Conclusion: Program improvements and performance measurement can be achieved with the design and implementation of a combined Change Management Plan and Transition Plan.

Conclusion: CanNor does not have the policy tools (PMS etc.) and the results-based data needed to conduct a successful a summative evaluation of NAEOP in 2018.

Conclusion: CanNor's operational procedures, interpretations, guidelines and controls are lacking and/or outdated.

Conclusion: Reducing reporting burden and improving the measurement of results can be achieved through the development of reporting templates that are linked to the performance indicators within an updated PMS.

The findings that support these conclusions are:

• A joint policy and operations staff-led combined augmented by external and/or internal resources to develop and implement a Change Management and Transition Plan would increase program performance and facilitate results measurement.
• CanNor will not have the performance indicators, data and updated PMS needed to conduct a productive summative evaluation before 2019.
• Reporting templates and report examples, adapted to the type, size and complexity of different funded projects, would streamline reporting and reduce the burden of recipient reporting.
• CanNor's operational procedures, interpretations, guidelines and controls are lacking and/or outdated. Also, they do not adequately address the changes brought about by the transition to NAEOP.
• It is necessary to update and approve the administrative guidelines and interpretations. 
• If CanNor decides to increase the percentage of funding directed to business-based projects, then it should identify a target percentage for each Territory, more clearly define what a business-based project is and develop an implementation plan on how to achieve those targets. 
• Replacing the need to sign new contribution agreements with an exchange of letters in the second and following years of a multi-year project. This would increase administrative efficiencies without sacrificing effectiveness. 
• The project application form, the PAR and the end-of-project report should be redesigned and integrated within the Performance Measurement Strategy to produce the information needed to measure program performance. 
• Develop performance indicators for each of the primary types of projects funded by CanNor. 
• Revise the Performance Measurement Strategy to increase the focus on program outcomes.

Recommendation: Develop a PMS that is results-based and which focuses on program results and not administrative outputs.

Recommendation: Develop new and update existing operational procedures, interpretations, guidelines and controls reflecting the transition to NAEOP.

Recommendation: Develop reporting templates linked to the performance indicators in an updated PMS.

001_Renovations in the North

Author: CanNor
Date Published: October 5, 2012
Contents of the Document: Page 8 of the PowerPoint presentation provides an indication of the direction CanNor would be taking with the Harmonization of NAEOP.

002_Summative Evaluation, EcDev, INAC

Author: INAC, Evaluation, Performance Measurement, and Review Branch
Date Published: April 2009
Contents of the Document: A summative evaluation conducted by INAC of their economic development programs being delivered in Canada's provinces and territories. The document predates NAEOP's implementation in 2014. The Aboriginal Economic Development Program (AED) was included within the evaluation.

Observations/Finding: The evaluation found:

• Cost Effectiveness:
  • is cost-effective considering the revenues, jobs and profits generated; (page 37)
• Relevance:
  • consistent with the federal government and departmental priorities; (page 18)
  • literature indicates that there is an ongoing need to support economic development in Indigenous communities; (page 19)
• Program Impact/Success:
  • ABDP and EDP contributed to Indigenous economic development as they increase participation of Indigenous people in the economy through the creation and retention of an estimated 6500 jobs per year; (page 21)
  • Information about jobs created could provide background information for future summative evaluation of NAEOP; (page 22)

003_CEOP, CEDP AND CSSP and Program Guidelines

Author: INAC

Date Published: Unknown. Was produced prior to the program handover is CanNor. Most likely no changes prior to April 2014.

Contents of the Document: Contains the CEOP, CEDP and CSSP program guidelines, embedded into a single document. Each program guideline includes descriptions and examples for: (i) eligible recipients; (ii) eligible projects; (iii) eligible expenditures; (iv) application requirements; (v) project approval criteria; (vi) funding levels; and, (vii) terms and conditions of funding agreements.

004_EBD Program Guidelines

Author: CanNor

Date Published: April 1, 2014

Contents of the Document: Includes descriptions and examples for: (i) eligible recipients; (ii) eligible projects; (iii) eligible expenditures; (iv) application requirements; (v) project approval criteria; (vi) funding levels; and, (vii) terms and conditions of funding agreements.

005_CROP Program Guidelines

Author: CanNor

Date Published: April 1, 2014

Contents of the Document: Includes descriptions and examples for: (i) eligible recipients; (ii) eligible projects; (iii) eligible expenditures; (iv) application requirements; (v) project approval criteria; (vi) funding levels; and, (vii) terms and conditions of funding agreements.

006_CanNor Service Delivery Standards for Program Delivery

Author: CanNor

Date Published: Uncertain. Viewed on CanNor's website January 6, 2017.

Contents of the Document: The Policy on Transfer Payments requires the establishment of departmental Service Standards for federal transfer payment programs. Under Section 6.5.9, the Deputy Head is responsible for establishing reasonable and practical departmental service standards for transfer payment programs. The Canadian Northern Economic Development Agency (herein referred to as "the Agency") has developed Service Standards to support its mandate: to advance economic development in Nunavut, the Northwest Territories, and Yukon.

CanNor's service standard is:

• Within ten (10) business days of receiving an expression of interest, the Agency will send an Acknowledgment – letter or email - to the Proponent.
• Within ten (10) business days of receiving a signed application for funding, the Agency will send an Acknowledgment – letter or email - to the Proponent.
• Within ninety (90) Net Business Days of receiving a signed and complete application, the Agency will render a funding decision (i.e. approval, rejection or withdrawal).
• Proponents will receive a payment from the Agency within ten (10) Net Business Days of a scheduled payment.

007_CanNor Project Assessment Report (PAR)

Author: CanNor

Date Published: Unknown. Updated 18 November 2016. Viewed on CanNor's website: Project Assessment Report PDF Version (168 Kb, 7 pages).

Contents of the Document: Within CanNor's Service Standards for Program Delivery, a copy of CanNor's Project Assessment Report (PAR) is presented. The PAR is a seven-page form used by CanNor to assess funding applications and to decide whether CanNor will or will not seek the Minister's funding approval for the project. The information within a completed PAR is drawn from completed funding applications as well as other information obtained from the applicant and other sources.

008_EBD Application Form

Author: CanNor

Date Published: Modified May 12, 2014

Contents of the Document: The document is the application form used to apply for EBD funding. It also contains basic instructions on how to complete the form as well as providing CanNor contact information where support can be sought to complete the application form.

009_EBD Program Guidelines

Author: CanNor

Date Published: April 1, 2014

Contents of the Document: Includes descriptions and examples for: (i) eligible recipients; (ii) eligible projects; (iii) eligible expenditures; (iv) application requirements; (v) project approval criteria; (vi) funding levels; and, (vii) terms and conditions of funding agreements.

010_2005 EC Program Administration Manual

Author: INAC

Date Published: August 24, 2005

Contents of the Document: The manual was written for the CEOP, CEDP and CSSP programs, but not the ABDP. The manual applies to all of Canada but it is not specific to the Territories.

INAC's economic development authorities, which had been in place since 1990, were updated under Treasury Board's policy on transfer payments in June 2000. The manual is outdated and is no longer used.

011_Federal Framework for Aboriginal Development

Author: INAC

Date Published: April 2009

Contents of the Document: The Framework was developed by the Government of the day and was reportedly used to substantiate the Harmonization of the CEOP, CEDP and CSSP programs and subsequent transition to NAEOP. The Framework replaced the Canadian Aboriginal Economic Development Strategy (CAEDS) that was released in 1989. The reason for the new Framework was stated as:

the conditions, needs, opportunities and relationships associated with aboriginal Canadians and economic development have changed significantly.

Note: The current Government and CanNor no longer use the Framework as a policy development tool.

012_LEDSP Program Guidelines

Author: INAC

Date Published: Updated August 6, 2014

Contents of the Document: The Lands and Economic Development Services Program (LEDSP) is funded and managed by INAC. LEDSP provides core-funding as well as targeted funding to First Nations and Inuit communities, and the organizations that they own and/or manage. Funding is available South of 60.

The Guidelines includes descriptions and examples for: (i) eligible recipients; (ii) eligible projects; (iii) eligible expenditures; (iv) application requirements; (v) project approval criteria; (vi) funding levels; and, (vii) terms and conditions of funding agreements.

013_ABDP Transfer Payment Terms and Conditions

Author: INAC

Date Published: Approved by Treasury Board July 23, 2003 with the most recent update being approved by Treasury Board on October 23, 2008.

Contents of the Document: Describes the objectives, terms and conditions, eligibility criteria, eligible activities, assessment criteria, and special conditions for the Aboriginal Business Development Program.

This document has been superseded NAEOP Transfer Payment Terms and Conditions which is presented on the following pages as document 016_NAEOP Transfer Payment Terms and Conditions.

014_CEDP CSSP CEOP Terms and Conditions

Author: CanNor

Date Published: 2009-10

Contents of the Document: An update to INAC's CEDP CSSP CEOP Terms and Conditions. It is identical to AANDC Authorities 372, 376, and 378. This document is quite like 003_CEOP, CEDP AND CSSP and Program Guidelines which was presented on the preceding pages.

015_AED Program Guidelines

Author: INAC, assumed

Date Published: Unknown

Contents of the Document: A summary of the four AED programs . . . seems to have been summarized for the Program Guidelines and the Ts&Cs.

016_NAEOP Transfer Payment Terms and Conditions

Author: CanNor

Date Published: Unknown

Contents of the Document: Given the uncertainty as to the accuracy of this version of the Ts&Cs, a more recent version was sought. Document 038_NAEOP Transfer Payment Terms and Conditions was obtained from the CanNor website.

017_NAEOP Identified Risks

Author: CanNor

Date Published: Approved by CanNor Policy Committee on August 29, 2013.

Contents of the Document: The risk drivers, risk levels and mitigation practices for the implementation of NAEOP are presented within this document. The risk drivers range from legal considerations; reaction of recipients to the changes; reaching expected outcomes; staff preparedness to implement NAEOP; through to the 11th which expresses concerns about performance measurement.

018_NAEOP Program Control Framework

Author: CanNor

Date Published: Last approved by CanNor Policy Committee on January 14, 2014

Contents of the Document: The document sets out program specific design, to clarify roles and responsibilities and provide administration of CanNor's Northern Aboriginal Economic Opportunities Program (NAEOP). It contains greater detail than is reflected in the NAEOP Ts&Cs document.

Note:   Document 029_Program Management Control Framework is the same as this document except in MS Word.

019_CROP Program Guidelines (CanNor)

Author: CanNor

Date Published: Modified April 1, 2014, published on the Internet.

Contents of the Document: A less technical version of the Ts&Cs for CROP. The document is designed to provide potential funding applicants on the eligibility criteria and the process to apply for funding.

020_NAEOP Program Guide for EBD

Author: CanNor

Date Published: April 14, 2014

Contents of the Document: This is Section 7 of NAEOPs program guide for use by CanNor staff when delivering EBD. CanNor staff in Nunavut explained that the document is outdated, and that work has begun on updating the manual. Nothing has been done in about a year since staff is waiting for the new DGOps before continuing, as review and approvals are required.

021_NAEOP Program Guide for CROP

Author: CanNor

Date Published: Updated March 10, 2014

Contents of the Document: This is Section 7 of NAEOPs program guide for use by CanNor staff when delivering CROP. CanNor staff in Nunavut explained that the document is outdated, and that work has begun on updating the manual. Nothing has been done in about a year since staff is waiting for the new DGOps before continuing as review and approvals are required.

022_EBD Delivery Partners NAEOP Third-Party Delivery

Author: CanNor

Date Published: Unknown

Contents of the Document: Extract of a document (handwritten title is NAEOP Third Party Delivery Agreements) which discusses how a third-party delivery partner is selected and some other basic information.

023_EBD Delivery Partners

Author: CanNor

Date Published: During or after 2015-16

Contents of the Document: Two-page PowerPoint identifying the three ASDPs and the three XDOs. This document was produced in support of an anticipated review of CanNor's suite of policies and programs in support of the Innovation and Growth Agenda.

024_Review of Community Readiness Initiative (CRI) September 2016

Author: DPRA Canada Inc.

Date Published: September 30, 2016

Contents of the Document: A review of CanNor's CRI (pilot 2013-2016) which has similarities to CROP but which is tied to community readiness. Participants highlighted relationship – building and support between communities and CanNor's and NPMO office as essential to supporting the success of the planning process. CanNor staff particularly expressed value of CRI in enabling strategic partnerships across federal and territorial departments. Community involvement in the CRI has also helped, to varying degrees, to raise community awareness and knowledge of resource development projects and the opportunities available through these projects.

025_Strategic Framework 2013-2018

Author: CanNor

Date Published: Uncertain, probably 2012-2013

Contents of the Document: This document is for all CanNor. The framework includes: vision statement; mandate; identifies the stakeholders including the indigenous population as a percentage of total population in the territories; discusses major sectors of involvement; the economy; CanNor strategic focus areas.

026_Minister re Core-funding [Advice to Minister]

Author: CanNor

Date Published: March 2, 2017

Contents of the Document: This is a protected document and a description will not be provided here. The document does not impact on the formative evaluation.

027_AED and NAEOP Contribution Budgets

Author: CanNor

Date Published:

Contents of the Document: Budget information for the four years covered by evaluation.

028_Staff Information Session Harmonization

Author: CanNor

Date Published: March 18-19, 2014

Contents of the Document: High-level five-page PowerPoint presentation explaining the move from AED to NAEOP and the changes that will be occurring.

029_Program Management Control Framework

Author: CanNor

Date Published: Last approved by CanNor Policy Committee on January 14, 2014

Contents of the Document: The document sets out program specific design, to clarify roles and responsibilities and provide administration of CanNor's Northern Aboriginal Economic Opportunities Program (NAEOP). It contains greater detail than is reflected in the NAEOP Ts&Cs document.

Note: Document 018_Program Management Control Framework is the same as this document except it is scanned.

030_Transition Plan Implementation Plan

Author: CanNor

Date Published: updated January 2014

Contents of the Document: The only identified implementation plan. Covers the period January 2013 through to March 2017. While the objectives and' steps' are identified, both from a policy and operational level, there are no updates after January 2014.

Note: This document was received March 27, 2017 and was not available at the time that the evaluation was designed, and the expanded evaluation matrix was created.

031_Harmonization Crosswalk Pre-April 2014

Author: CanNor

Date Published: May 13, 2013

Contents of the Document: The crosswalk compares the Ts&Cs between those that are being proposed and those that existed in May 2013. Where changes are identified there are often notes and rationale as to why the changes were made.

032_NAEOP Budget

Author: CanNor

Date Published: Received from CanNor December 2016.

Contents of the Document:  Multi-year budget for CanNor showing planned and actual disbursements.

033_Federal Budget 2017

Author: Government of Canada

Date Published: March 22, 2017

Contents of the Document: The federal budget that was tabled in the House of Commons.

034_NAEOP Budget Amended

Author: CanNor and ASAinc

Date Published: Updated from 032_NAEOP Budget

Contents of the Document: This document was formatted and restructured a bit so that would fit within the evaluation report.

035_CRI Review November 2016

Author: DPRA Canada Inc.

Date Published: November 2016

Contents of the Document: A review of CanNor's CRI (pilot 2013-2016) which has similarities to CROP but which is tied to community readiness. Participants highlighted relationship – building and support between communities and CanNor's and NPMO office as essential to supporting the success of the planning process. CanNor staff particularly expressed value of CRI in enabling strategic partnerships across federal and territorial departments. Community involvement in the CRI has also helped, to varying degrees, to raise community awareness and knowledge of resource development projects and the opportunities available through these projects.

036_KIA Business Funding Application Guideline

Author: Kitikmeot Inuit Association

Date Published: March 2014

Contents of the Document: The Kitikmeot Inuit Association (KIA) administers two small-business funding programs: (i) Kitikmeot Business Assistance Program (KBAP); and, (ii) Nunavut Sivummut Program (NS). The purpose of these programs is to help provide jobs and income for Kitikmeot Inuit through establishment and expansion of Inuit-owned businesses.

The purpose of these easy to read guidelines is to provide an explanation of the programs, and instructions on preparing applications.  These guidelines are meant to improve the quality of applications submitted, so the KIA can process applications quickly for our clients. The main topics that are covered include: (i) The Application Process; (ii) Timeline for Applications; (iii) Reporting Requirements; and, (iv) a sample of the application forms used by the Kitikmeot Inuit Association.

037_NAEOP Summary of Amendments

Author: CanNor

Date Published: Likely 2013-14

Contents of the Document: This document provides a list of the proposed changes, about three pages in length. Do not know what was the source of this information.

038_NAEOP Transfer Payment Terms and Conditions

Author: CanNor

Date Published: Updated 10 November 2016. Viewed at CanNor website

Contents of the Document: The document contains: (i) Legal and Policy Authority; (ii) Purpose, Objectives and Expected Results; (iii) Eligible Recipients; (iv) Eligible Initiatives and Projects; (v) Eligible Expenditures; (vi) Stacking Limits; (vii) Method for Determining the Amount of Funding; (viii) Maximum Amount Payable; (ix) Basis on Which Payments will be Made; (x) Information (Application) Requirements; (xi) Assessment Criteria; (xii) Monitoring and Reporting Mechanisms; (xiii) Redistribution of Contributions; (xiv) Repayable and Conditionally Repayable; (xv) Contributions; (xvi) Official Languages; and, (xvii) Other Terms and Conditions.

Renovation in the North

Purpose of Harmonization

Program review and Harmonization was undertaken to:

• Align with the strategic priorities of the 2009 Federal Framework for Aboriginal Economic Development.
• Make programs more relevant and effective in supporting the increased participation of northern Aboriginal people in economic opportunities, informed by partner and stakeholder engagement.
• NAEOP is proposal-based, opportunity-driven, focused on results and geared towards maximizing economic opportunities for Aboriginal communities and businesses in the territories.
• NAEOP is expected to result in:
  • Improved program management efficiency through a single-window approach and standardized program management and administration processes.
  • Increased flexibility and simplified process for recipients through, for example, a single application process, multi-year agreements, streamlined reporting.
• NAEOP delivery and implementation is guided by:
  • NAEOP Terms and Conditions
  • NAEOP Management Control Framework
  • NAEOP Performance Measurement Strategy
  • NAEOP Program Risk Assessment and Mitigation Framework
  • NAEOP – Community Readiness and Opportunities Planning Guidelines
  • NAEOP – Entrepreneurship and Business Development Guidelines
• CanNor Policy remains responsible for amendments to and interpretation of the program Terms and Conditions, Management Control Framework and Performance Measurement Strategy. 

Expected Results of Harmonization

NAEOP was presented as a single program composed of two streams; thus, Harmonization was presented as four AED programs rolled into a single NAEOP program.

Ministerial Authority was used to implement Harmonization. Treasury Board approval was not sought.

The following table was presented to CanNor staff to explain Harmonization.

Program Implementation, Key Changes

Harmonization was designed to change and improve program delivery. The following table presents the key changes identified during the staff briefings.

Senior Management Committee

The Senior Management Committee is the Official-level decision-making committee for all CanNor matters and is chaired by the President. It is responsible for the governance and oversight of the Agency. The SMC:

• Sets corporate values, management practice, strategic direction and integrated business plans;
• Approves policies, programs, projects and management undertakings, and ensures overall organizational alignment, integration and awareness;
• Oversees activities and performance in relation to values, management practice, strategy and plans; and
• Approves multi-year monitoring, audit, evaluation and reporting plans for the policy and programs, and tasks for off-plan audits and evaluations as required.

Policy, Planning & Operations Committee

The Policy, Planning & Operations Committee provides a forum to examine and discuss the Agency's strategic issues and priorities for advancing northern economic opportunities with respect to policy development, corporate planning, program management and delivery, and stakeholder engagement. 

The committee provides advice and guidance, and makes recommendations to SMC with respect to policy, planning and operational issues. Specifically, the committee will:

• Identify strategic priorities to help deliver on the Agency's mandate and track progress; 
• Examine trends and issues related to northern economic development and the impact on CanNor's policy and program interventions, including in support of major projects;
• Consider innovative strategies and initiatives to strengthen the Agency's program management functions;
• Discuss trends in actual and forecasted spending for the Agency's contribution funding with a view to determining funding approaches and strategies; and
• Support the integration of strategic communications into Agency activities.

Extended Management Committee

Extended Management Committee will support the work of the Agency at the management level and is chaired by the President. The EMC:

• Provides a forum for key messages from the President to Agency management;
• Provides a forum for management to provide information to the President on the status of work and of new initiatives;
• Promotes meaningful engagement of all executives, resource managers and other key advisors in the direction of the Agency; and
• Strengthen the awareness, capabilities, alignment and teamwork of the leadership cadre on all policies, programs, projects and management undertakings of the Agency.

Resource Management Committee

the Resource Management Committee supports the work and responsibilities of the Agency for sound resource management, stewardship, and human resources management.  RMC assists CanNor senior management in:

• Advancing the Government of Canada's People Management Strategy;
• Addressing workforce and workplace related matters;
• Promoting strong stewardship of financial resources;
• Ensuring transparency in the allocation of funds; and,
• Promoting compliance with procurement policies and objectives.

Program Management Control Framework

In a broad sense, the management control framework for each program includes the full set of processes and guidelines which govern program operations, including the CanNor Management Control Framework for Grants and Contributions.

The purpose of this document is to set out program-specific design, to clarify roles and responsibilities and guide administration of CanNor's Northern Aboriginal Economic Opportunities Program (NAEOP). It contains greater detail than is reflected in the NAEOP Terms and Conditions (Ts&Cs) document Contributions to Support Aboriginal Participation in the Northern Economy. In conjunction with the NAEOP Ts&Cs and the NAEOP Program Performance Measurement Strategy (PMS), it provides the policy framework for implementation of the program, including any design specifics approved at Cabinet level, as well as any areas where CanNor Policy Committee has established further clarification prior to implementation.

Consistent with this Program Management Control Framework (MCF), CanNor's Operations, Finance and Communications units may, through their established processes, identify further tools, processes or standards for delivery of the NAEOP. 

CanNor Aboriginal Economic Development Program Harmonization

Transition Year 2 (April 2014 – March 2015)

Transition Year 3 (April 2015 – March 2016)

Transition Year 4 (April 2016 – March 2017)

Annex: Transition Plan for January – March 2013 (completed)