Evaluation of the Strategic Investments in Northern Economic Development Program

Relevance - Continued need

Primary arguments for the continuation of SINED include the need to address the enormous challenges to economic development in the North, a shortage of funding, the need for further capacity development and diversification, and the importance of a northern approach in the North. Canada's territories have significant natural resource deposits in oil and gas, metals, diamonds and minerals, however, there is a need for economic diversification to protect the economy from the boom-bust cycles of the resource sector. Natural and cultural attractions have contributed to growth in tourism and cultural industries. Renewable energy and fisheries have also advanced.

However, the development of all sectors has been challenged by a lack of transportation and communication infrastructure, skills and housing shortages and high energy costs. The public sector remains the largest employer, private sector investment is difficult to attract and funding from the territorial government is limited. Within this context, the North needs programs like SINED, which integrates northern perspectives and priorities through the Territorial Investment Plans to address the barriers to economic development and contribute to a foundation for economic development to take place.

Relevance - Alignment with federal priorities

There is a very strong alignment between SINED objectives and federal priorities. From 2005 when SINED was created up to now, the North has featured predominantly in federal announcements and investments. Also, Canada's Northern Strategy was introduced in 2007 and since that time, the Government of Canada has committed to the streamlining of major project approvals, housing construction, Aboriginal skills training, adult basic education, mapping of the Arctic seabed, construction of a world-class Arctic research station, completion of the Dempster highway and more. The 2008 and 2011 Speeches from the Throne strengthened the government's commitment to the North and the 2009 SINED renewal complements other federal initiatives (e.g. 2009 Economic Action Plan) in the North.

Relevance - Alignment with federal roles and responsibilities

SINED's role in economic development is embedded in the federal government's jurisdiction in the North, which is clearly mandated in legislation namely, the Canadian Constitution and the Department of Indian Affairs and Northern Development Act. Other entities (e.g. territorial governments) are involved in the economic development of the territories; however, the evaluation found that their roles and activities are complimentary to that of SINED. Along the same lines, while some "province-like" responsibilities for governance and administration have been successfully transferred to the territorial governments, control over land and resource management is in varying stages of devolution from AANDC to the territories with Nunavut only now beginning to negotiate a devolution plan. In the end, economic development in the North remains a shared responsibility between the federal government, territorial governments and self-governing First Nations. Given the continued reliance of the territorial governments on federal transfers, federal involvement in northern economic development will likely continue for some time.

Performance – Effectiveness/success

The evaluation was unable to make any conclusions on the overall impact of SINED investments due to the lack of processes or systems to monitor and aggregate results over time and difficulties accessing project reports through the First Nations and Inuit Transfer Payment System. A Performance Measurement Strategy created for SINED in 2009 was never implemented. In the absence of a formal coordinated approach to monitoring progress, performance information has been collected at the project level and documented in project reports. Performance reporting thus involves the manual review and roll-up of individual reports.

Some positive outcomes and useful indicators for investments in energy, renewable energy, mining, geoscience, fisheries, tourism, information communications technology/broadband and cultural industries were documented through the case studies.

Performance – Efficiency and economy

SINED staff was identified by recipients as a strength due to their dedication, knowledge, and support. However, SINED has experienced difficulties attracting and retaining staff. This has resulted in increased workloads, impacted planning, performance measurement, monitoring and reporting. In turn, these difficulties have led to a reduction in the support that staff provides to project proponents.

Changes to the project approvals process have contributed to delays in project and funding approvals. These delays have disrupted recipient organizations' planning processes, increased project costs and impacted the timely delivery of quality projects. In addition, delays in funding contribute to financial hardship for recipient organizations.

The transition of SINED from Aboriginal Affairs and Northern Development Canada to CanNor has also impacted the effective and efficient delivery of SINED programs. SINED was transferred to CanNor just as the new agency was getting established. Shortly thereafter, SINED staff was tasked with delivering additional programming introduced through the federal government's Economic Action Plans. These actions resulted in increased workloads and impacted SINED delivery. The evaluation also found that the transition has resulted in a "disconnec" between management and operations.

Another key finding is the many similarities between the four SINED component programs. Not only do these components have similar eligible recipients but they also fund similar projects thus calling into question the need for four separate component programs.

Other Issues

An analysis of Canadian and international approaches to economic development in northern and remote regions concluded that SINED is consistent with international best practices for short-term policies to improve economic development. However, without significant, coordinated, long-term investments in the North, SINED is unlikely to result in sustained change in economic conditions. To this end, the literature emphasized the importance of sustained long-term investment in leadership, institutions, education, skills development, infrastructure, entrepreneurship and healthy, vibrant communities.

Conclusions

SINED is consistent with federal government priorities and roles and responsibilities and responds to a continued need for foundational investments supporting economic development in the North. SINED funding complements what is available through territorial governments and other federal departments and should continue given the limed resources and difficulties attracting private-sector investment.

While some quantifiable measures of success were documented for particular sectors, without a process to properly monitor the outcomes of SINED investments it is not possible to assess the impacts of these investments. SINED needs a performance measurement strategy, which identifies outputs, outcomes and indicators and which outlines a process for collecting performance information. Upon implementation, such performance information can be used for ongoing management, reporting on results and the early identification of problems.

The evaluation identified inefficiencies in the project and funding approvals, program design and tracking tools. In addition to negatively impacting internal operations, inefficiencies in these areas have also negatively impacted recipient organizations and the cost, quality and timeliness of projects. SINED should apply lessons learned and best practices from regions and from Regional Development Agencies to address issues impacting efficiency and economy.

The disruption caused by the transition of SINED to CanNor, the additional workload and the disconnection between SINED management and operations must be recognized and addressed immediately. A staffing strategy is needed to identify responsibilities and appropriate resource levels for the effective delivery of SINED and to attract and retain staff. The strategy should target individuals with the right skills (e.g., new business graduates), address barriers to moving north (e.g., housing) and provide incentives for moving north (e.g., training).

Finally, there is a great need for foundational investments in the North to support economic development. SINED provides essential support for capacity building in existing sectors and economic diversification, and has adapted well to northern economic development strategies. However, the need in the North is such that it is greater than one economic development program can possibly fulfill. In order to achieve real change, SINED must be part of a long-term, sustained, coordinated approach to northern development.

1.2.1. Program Context

Canada's North faces unprecedented opportunities: it has impressive economic assets, principally in the form of identified but untapped natural resources. However, on the opposite side of the equation lies a daunting list of challenges to economic development. Beyond the capital cities, the North is composed of remote communities separated by huge distances. The transportation and communications infrastructure is underdeveloped and energy costs are high. Arctic conditions, including permafrost, snow and ice cover, extreme weather, prolonged darkness in winter and light in summer and limited local and regional capacity are a part of life. A large public sector remains the primary employer in the territories. Attracting private sector investment to the North is a challenge and industries struggle to overcome many barriers to business. Opportunities and risks, or barriers specific to key sectors in the North are summarized below:

In 2010-11, tourism contributed over $110 million to the economy in the Northwest Territories. Tourists can take advantage of various activities like sport fishing, hunting, outdoor adventures, aurora viewings, and Aboriginal cultural events across the territories. Long-term trends indicate that the sector has opportunity to grow in adventure and eco-tourism and is expected to benefit from a growing desire among travellers for authentic experiences. Threats to the sector include limited airline capacity, particularly in Nunavut and the Northwest Territories, competition from other tourist destinations, limited connectivity, a lack of capital investments and a slow national and international economic recovery.

The North has a distinct and proud history of Aboriginal culture and traditions and one of the highest forms of Aboriginal cultural expression is through arts and crafts. Cultural industries (such as film, carving, printmaking, weaving, jewellery, clothing) across the North are recognized and respected internationally and have gained an increase in international exposure in recent years, particularly as a result of cultural displays and events at the 2010 Olympics in Vancouver. In Nunavut, the cultural industries employ 20 percent of the local workforce^{Footnote 1} and contribute a total of $33.4 millionannually to the Nunavut economy, including $22.9 million in direct impact, and $10.5 millionin related spinoffs^{Footnote 2}. The cultural sector of the Northwest Territories, including artists, consists of 750 workers, almost three percent of the territorial workforce (2011-12)^{Footnote 3}. New markets in Europe have been steadily growing and there is potential for a Chinese market, but the majority of the market remains in Canada.

Challenges to cultural industries include economic conditions/downturns, barriers to accessing new markets, difficulties accessing materials such as seal skins or carving stone (some quarries can only be accessed once a year), linguistic profiles of artists, a lack of a distinct brand image for the various products and competition from high-paying jobs in other sectors. Obstacles identified by the film industry include lack of tax credits, of the kind offered by many jurisdictions, and limited availability of technicians and equipment.

Fisheries has been identified as an important sector for both the Northwest Territories and Nunavut; however, Nunavut has seen the most promising growth in recent years. In 2010, it was estimated that Nunavut's fisheries contributed between $12 and $14 million annually to the territorial economy ($7.5 to $9.5 million as income)^{Footnote 4}. In 2010, the Standing Senate Committee on Fisheries and Oceans identified an increased harvest in Arctic char as a goal for future development of the industry^{Footnote 5}. Arctic char are a highly priced delicacy, marketed mainly fresh and frozen as whole-dressed fish and steaks^{Footnote 6}. The most important fish species in the Northwest Territories are in Great Slave Lake and include whitefish, pickerel, lake trout, northern pike, and inconnu^{Footnote 7}.

The greatest threat to the fisheries industry is the availability of supply of fish, so the focus of much of the industry is maintaining a stable industry rather than increasing production^{Footnote 8}. In Nunavut, the fisheries industry has also been hampered by the lack of a deep-water port to offload catches. Although Nunavut has Canada's largest coastline, it is the only coastal jurisdiction that does not have a deep-water port, which can accommodate large fishing vessels and the processing of all their catches. As a result, the offloading of catches from large fishing vessels is time consuming and catches that cannot be processed in Pangnirtung are being shipped internationally to Greenland for processing and to other markets for packaging. In turn, these countries label this Canadian product as their own. Investments in infrastructure like port facilities and harbours to support fishing vessels (even of modest size), as well as processing plants and cold-storage operations, are needed for the sector to thrive in Nunavut. In the Northwest Territories, there has been a drop in fish harvests due to challenges with the fish population, however, a recovery strategy is in place to address these challenges^{Footnote 9}.

The United States Geological Survey estimates that the North contains up to 13 percent of the world's undiscovered oil and 30 percent of undiscovered natural gas reserves^{Footnote 10}. However, the oil and gas industry has been impacted by declining international prices and delays to the Mackenzie Gas Pipeline Project. Since peaking in 2001, the Northwest Territories' fuel industry has been in decline with no new wells expected to begin production over the next few years^{Footnote 11}. Additional challenges to the development of the energy sector include the fragility of northern ecosystems, a shortage of skilled labour, a lack of communication and transportation infrastructure and a lack of private sector incentives and investment.

Likewise, there is great renewable energy potential in the North. In the Northwest Territories, there are multiple larger-scale hydro opportunities, creating a foundation for a power grid, which would reduce rates and enhance reliability and diversity to the economy. In the Yukon, hydro and wind comprise 17 percent of energy use^{Footnote 12} and the supply of fuel wood has been valued at approximately $4 million a year in employment and import substitution^{Footnote 13}. While some studies have been conducted to better understand the Yukon's wind, geothermal and solar energy potential, it is largely unknown and undeveloped^{Footnote 14}. Yet, threats such as limited local capacity and funds for renewable energy development in the North and poor economies of scale pose particular challenges to the sector's growth.

Moreover, the potential of mining in the North also looks bright. Many northern regions have a strong history in mining and are highly developed, while others offer underexplored regions with significant potential for resources^{Footnote 15}. It is estimated that northern diamond mining accounts for 8.87 percent of total global diamond production, making Canada the third-largest global diamond producer by value^{Footnote 16}. The global economy is recovering and the long-term global demand for commodities is increasing, making mines in remote northern areas potentially viable^{Footnote 17}. Several new mines are slated to open before the end of the decade in Nunavut, Yukon, and the Northwest Territories and at least five mines in the Yukon are expected to be producing during the mid and later part of the decade^{Footnote 18}. In addition, the overall northern metal and non-metallic mineral output is expected to grow by 91 percent from 2011 to 2020, with a compound annual growth rate of 7.5 percent^{Footnote 19}.

Devolution will likely contribute to increased mining activity. Since 2005, the Yukon has been described as one of the world's best mining jurisdictions because of its very sta­ble political climate, good transportation and energy infra­structure, and demonstrably pro-mining government. As a result, the Fraser Insti­tute's annual mining survey regularly places post-devolution Yukon near the top of its Policy Potential Index for best mining jurisdic­tions within Canada and around the world^{Footnote 20}. Until devolution is finalized in the Northwest Territories and Nunavut, regulatory uncertainty will continue to be a threat to mining in those territories. Additional threats include limited capacity and resources of First Nations and Inuit to engage in consultation and negotiation, recruitment of labour to work on remote construction and mining sites, poor road infrastructure, lack of an energy strategy for large-scale mining, serious shortages of housing^{Footnote 21}, a fragile natural environment, distance to market and climate change.

Within this context, an economic development program, such as SINED, can address many of the barriers identified above and help establish the conditions for future economic growth and stability.

1.2.2 History of SINED

SINED emerged as the federal government's primary response to economic development in the North in March 2004 when the federal budget committed $90 million over a period of five-years for northern economic development. In the spring of 2005, SINED was launched by AANDC. It included three complementary programs: a new Targeted Investment Program was created with incremental funds. The existing Innovation and Knowledge fund and Northwest Territories-specific Partnership and Advisory Forums fund, both A-based programs, were renewed and integrated into the set.

In 2009, SINED was renewed as part of Canada's Economic Action Plan, whichallocated a new $90 million in funding for another five years (April 1, 2009 to March 30, 2014) over and above the ongoing funds, which totaled almost $6 million over the same period. Following review and stakeholder input, the program was renewed with essentially the same design, but with enhanced flexibility to realign funds between program elements, and (consistent with the announcement that it would form the core activity of a new regional economic development agency for the North) increased emphasis on leadership and helping build partnerships. The Partnership Advisory Forums was extended to all territories to help promote multi-stakeholder discussions, and a new Pan-Territorial Fund was established to support projects impacting more than one territory.

1.2.3 SINED Components

Targeted Investment Program

In 2009, Targeted Investment Program was allocated $67 million over five years to be divided equally among the territories. An analysis of the distribution of Targeted Investment Program funding between 2007-08 and 2011-12 revealed a fairly equal allocation of Targeted Investment Program funding across the three territories. Northwest Territories accounted for 36 percent of expenditures and Nunavut and the Yukon each spent 32 percent of Targeted Investment Program funding.

A single project cannot exceed $2 million in annual funding and the amount payable to any one recipient cannot exceed $6 million. Targeted Investment Program projects must align with Territorial Investment Plans (which were developed in advance of the 2009 renewal based on progress in the first round of SINED); economic realities in each territory; and, input from Territorial and Aboriginal governments, federal departments, the private sector and other stakeholders. Investment Plans are reviewed and updated as necessary.

The Investment Plan in each territory may identify a mix of specific activities, recipients and objectives; more broadly defined activities with specific objectives and outputs; and broad developmental areas with activities and outputs to be identified through general calls for proposals. See Section 1.2.8 for more information on Territorial Investment Plans.

Targeted Investment Program projects must also align with the following four thematic areas:

1. Building the knowledge base in key economic sectors to help set the stage for further economic opportunities. Eligible projects include research projects, strategic plans, feasibility studies and other related projects^{Footnote 22}.
2. Enhancing the economic infrastructure base through the development of multi-user physical infrastructure critical to economic growth. Eligible projects include feasibility or planning studies for public or multi-user infrastructure; public/multi-user telecommunications investments that fill gaps not covered by other federal funding; and, small-scale investments in new or existing infrastructure with direct linkages to enhanced economic activity^{Footnote 23}.
3. Capacity development of organizations, associations, small and medium enterprises and individuals to help them take advantage of economic opportunities. Eligible projects include research projects, development of materials, curriculum, standards which support capacity development, and capacity development initiatives^{Footnote 24}.
4. Economic diversification within regions and sectors with a focus on the development of new sectors, products, and markets. Eligible activities include, but are not limited to, the development of sectoral strategies that can be then used to strategically guide investments; support for the creation of technology clusters; support to cultural industries; and, new product development and marketing^{Footnote 25}.

As shown in Graph 1, the majority (43 percent) of Targeted Investment Program funding between 2007-08 and 2011-12 was under the "building the knowledge base" theme. 30 percent of projects were classified as "capacity development" and 16 percent were directed towards "economic diversification". Projects for "enhancing the economic infrastructure base" represented 11 percent of total SINED projects during that time.

Text description of Graph 1

Graph 1 depicts the Targeted Investment Program Expenditures from 2007-08 to 2011-12, broken down by theme. The majority (43 percent) of Targeted Investment Program funding was under the "building the knowledge base" theme. 30 percent of projects were classified as "capacity development" and 16 percent were directed towards "economic diversification". Projects for "enhancing the economic infrastructure base" represented 11 percent of total SINED projects during that time.

Ineligible projects/expenditures include: core funding; ongoing operations and maintenance of organizations; major capital investments in infrastructure (e.g., laboratories); research of knowledge-based activities that are primarily a resource management function; construction activities related to large engineering works (e.g., roads, highways, bridges); investments in new or existing infrastructure which is either owned or operated by a single for-profit enterprise that is not for public use; direct skills training of individuals of the sort normally offered through learning institutions or apprenticeship programs; and wage subsidies for persons receiving training through course work or on the job^{Footnote 26}.

Eligible recipientsinclude organizations, associations or social enterprises whose mandate includes work with respect to northern economic development; individuals and small and medium-sized enterprises; other levels of government; and other non-federal entities, public or private, that have an interest/involvement in economic development in the North^{Footnote 27}. An analysis of recipients for Targeted Investment Program expenditures between 2007-08 and 2011-12 reveals that the territorial government is the primary recipient in all three territories - mainly for geoscience. Between 2007-08 and 2011-12, the Government of Nunavut received $15 million^{Footnote 28}, the Government of Northwest Territories received $10.7 million^{Footnote 29} and the Government of Yukon received $8 million^{Footnote 30}.

Text description of Graph 2

Graph 2 depicts the recipients of Targeted Investment Program funding between 2007-08 and 2011-12, broken down by the following regions: Nunavut, Northwest Territories, and Yukon. The graph shows that the Northwest Territories and Yukon regions support industry associations more frequently than Nunavut. The Yukon supports Aboriginal organizations significantly more than the Northwest Territories and Nunavut. Also, the Nunavut region has the most dealings with the territorial government and non-government organizations.

Graph 2 above shows that the Northwest Territories and Yukon regions support industry associations more frequently than Nunavut. The Yukon supports Aboriginal organizations significantly more than the Northwest Territories and Nunavut. Nunavut region has the most dealings with the territorial government and non-government organizations.

1.2.4 SINED Investments

As shown in Graph 3 below, between 2007-08 and 2011-12, 87 percent of SINED investments were made through Targeted Investment Program, eight percent through Innovation and Knowledge, four percent through Pan-Territorial Fund and one percent through Partnership Advisory Forums.

Text description of Graph 3

Graph 3 depicts the distribution of SINED funding between 2007-08 and 2011-12, broken down by program. The vast majority of SINED investments, at 87 percent, were made through Targeted Investment Program. Eight percent of investments were distributed through Innovation and Knowledge, four percent through Pan-Territorial Fund, and one percent through Partnership Advisory Forums.

For the second round of SINED, 84 percent of the $90 million was expected to be distributed through Targeted Investment Program, eight percent through Innovation and Knowledge, six percent through Pan-Territorial Fund and two percent through Partnership Advisory Forums. Graph 3 shows that the actual investments are within range of planned expenditures.

1.2.5 Objectives and Expected Outcomes

In June 2009, while SINED was still with AANDC, a Performance Measurement Strategy was created to measure, manage and report on the performance of SINED investments. The following outcomes were taken from the Performance Measurement Strategy:

1.2.6 Program Management, Key Stakeholders and Beneficiaries

1.2.7 SINED Budget Allocations for 2007-08 to 2011-12

A comparison of Tables 1 and 2 above reveals that actual Grants and Contributions expenditures are close to planned/forecast spending, but are slightly below expected levels. Since EPMRB did not have access to all planned/forecast and actual spending for Operations and Maintenance, the evaluation was unable to comment on changes in these areas.

1.2.8 Design and Delivery

2.3.1 Data Sources

The evaluation's findings and conclusions are based on the analysis and triangulation of the following lines of evidence:

Literature Review

A literature review of academic and government reports was conducted to examine economic development strategies employed in other circumpolar countries and remote regions and determine which strategies yielded the most promising results. SINED was compared to these recognised best practices.

The literature review also looked at government and industry studies, reports and statistical analysis relating to trends in economic opportunities and challenges in the North. This analysis informed the ongoing need for SINED programming. Current federal initiatives and strategies relating to economic development in the North were also looked at to test SINED alignment with Government of Canada priorities.

Document and File Review

CanNor provided the evaluators with a list of projects funded through SINED between 2007-08 and 2011-12. The list contained project descriptions and projects were classified by recipient, year, project amount, sector (for Targeted Investment Program projects), region and theme (for Targeted Investment Program projects). This list of projects was used to determine the distribution of funding by region and theme. It was also used to better understand which sectors were supported, identify the major recipients, and quantify investments in capacity.

The quantification of investments in capacity is linked to the assessment of the immediate outcome "Increased ability of Northerners to respond to economic development opportunities". The first step to understanding the impact of access to and use of capacity-building products and services is to calculate investments in this area.

This document review also looked at internal SINED documents such as planning and program reports, program manuals, templates, foundational documents, the performance measurement strategy, recipient reports, previous reviews, evaluations, and audits. These documents contributed to the development of the program profile and the assessment of its performance.

Key Informant Interviews

A total of 46 interviews were conducted: 15 were with SINED staff and 31 with recipients. Seven SINED staff members interviewed were from senior management (Director and above) and the remaining were Economic Development Officers and Managers. Recipients selected for interview represented the largest projects (in terms of dollar value) that were funded in each sector. However, some recipients were selected for interview based on the perceived success of their projects.

The interviews were based on standardized, largely open-ended questions that addressed evaluation issues pertaining to program design and delivery and performance. The majority of interviews were conducted in person during site visits, however, some interviews were conducted by phone to accommodate schedules.

Interview notes were recorded and analyzed manually to identify patterns in responses. The final report presents these patterns in the findings, so no individual may be identified.

Case Studies

A total of 12 case studies were conducted. Seven sector case studies were completed to explore the opportunities and threats facing major sectors in the North, review the types of projects supported by SINED in each sector, identify major recipients and document the impacts of SINED investments.

The sector case studies were selected based on the analysis of the list of SINED projects and input from the Working Group that was created for the purposes of this evaluation. The sectors that received the most funding were selected along with some smaller emerging sectors such as fisheries and renewable/alternative energy, which have responded particularly well to SINED investments. The seven sector case studies were: Energy, Alternative Energy, Cultural Industries, Fisheries, Geoscience, Mining and Tourism

The sector case studies incorporated interviews with recipients, a review of program documentation and literature relating to the sector in the North, project products and reports.

Five other case studies were completed. These took a closer look at the impacts of capacity investments, Innovation and Knowledge, Partnership Advisory Forums, Pan-Territorial Fund and broadband/information communication technologies on SINED outcomes. The case studies consisted largely of document reviews, which looked at the types of projects funded and their impacts, however, interviews were also conducted with recipients working on broadband.

Survey

As part of the geoscience case study, a survey was administered to users of geoscience data produced by the Nunavut geoscience office to further supplement information found in project files. Five organizations were surveyed - two representing mining/exploration companies, one Aboriginal organization and two municipalities. The survey was drafted with the assistance of the Nunavut Geological Society which also provided names and contact information for survey recipients. Surveys were e-mailed and reviewed by EPMRB.

2.3.2 Considerations, Strengths and Limitations

Key Considerations

The design and delivery of SINED, lessons learned and best practices would have been greatly informed by a comparative analysis with programs in other Regional Development Agencies. This approach was proposed as an additional line of evidence for this evaluation, but was removed because it was felt that this information could be gathered through an existing Regional Development Agency Working Group. Discussions of the Regional Development Agency Working Group were not documented as part of this evaluation.

Attribution/Contribution

The evaluation acknowledges that other factors may have contributed to SINED's expected results/outcomes making it difficult to isolate SINED's specific influence/impact. As a result, each line of inquiry was designed to best overcome the issue as much as possible. Analysis has been confined to program participants to limit the effects of external variables. Also, immediate impacts have been sought out with the knowledge that the expected results/outcomes of many investments, especially in capacity building, may not be seen for many years.

Strengths

The strengths of this evaluation include the use of multiple lines of evidence, as indicated in the evaluation matrix, in order to triangulate findings and increase reliability. With key informant interviews, a significant sample size incorporated the perspectives of all key SINED stakeholder groups. The methodology for the evaluation, including the data collection instruments, was developed in consultation with CanNor officials and the evaluation Working Group.

Limitations, Analytical Challenges and Mitigation Approaches

The preference in all areas of inquiry is to have quantitative data and documented information complemented where appropriate by the informed views of stakeholders. Several limitations encountered by EPMRB evaluators include differences in how client data and information are reported across jurisdictions and over time, reliance on qualitative data, and differences in delivery across the three territories. Because no formal system is in place to collect performance information, EPMRB had limited access to quantitative data on SINED activities and outcomes. So, SINED's effectiveness, such as improved performance due to enhanced behaviors, was a challenge to ascertain. Other methodological and analytical challenges are listed below by line of evidence.

Document/File Review

Key Informant Interviews

The weakness of all key informant interviews is their reliance on the opinions and perceptions of individuals, which are subject to bias. To mitigate this risk, the evaluation does not reference individual interviews, but reports on trends or themes in responses to questions. In addition, interviewees were asked for supporting evidence for statements and key informant interviews are cross-referenced with other evaluation lines of evidence. Validating findings through other primary and secondary research mitigates this limitation.

Case Studies

The biggest challenge for the case studies was the lack of consistent performance data. Project outcomes were agreed upon with proponents on a project-by-project basis and documented in project reports. However, due to the non-implementation of the performance measurement strategy and the overwhelming variety of projects, there is little consistency in the measurement of outcomes. The case studies sought to overcome this challenge by looking at impacts of major projects in supported sectors, identifying potential indicators for success, and then seeking out evidence that demonstrated progress. As a result, impacts documented through the case studies are project-specific and cannot be rolled up to describe the impact of investments on a territory or sector, nor can they be linked to expected outcomes. Instead, they provide anecdotal evidence of successful outcomes.

a) Impact of SINED investments in energy and renewable energy

Between 2007-08 and 2011-12, SINED invested $4.6 million in energy and renewable energy. The case study covered capacity/knowledge building projects amounting to $2.1 million, which represents 47 percent of the total SINED investment in energy and renewable energy. The projects reviewed in the case study were in the Northwest Territories and Yukon and included studies, conferences, guides, public engagement and workshops on natural gas, pipelines, geothermal from abandoned mines sites, biomass, wind energy and waste to energy. The project proponents examined in the case study were Yukon Energy, Alaska Highway Aboriginal Pipeline Coalition, Department of Environment and Natural Resources of the Government of Northwest Territories and the City of Yellowknife.

Energy projects examined by EPMRB evaluators showed that though these had generated new knowledge, the impact of this new knowledge on the energy sector was not documented and, in addition, the extent to which this knowledge was made available to, and accessed by, the public is also not known. For example, the Eagle Plain Case Study determined and compared costs of different options for electricity generation from natural gas; however, no information is provided about the usage or impact of this new knowledge on the energy sector.

On the other hand, some renewable energy projects that were assessed led to concrete steps to advance biomass energy. For example, EPMRB found that studies in the Northwest Territories led to the identification of a biomass industry, attracted private sector partners and led to the development of Forest Management Agreements with First Nations for harvesting rights^{Footnote 44}. Also, the Fort Smith Métis Council used the findings of a feasibility study to convert the heating systems in four buildings to bio-fuel, which has resulted in estimated savings of $17,000 per year^{Footnote 45}.

The case studies on energy and renewable energy highlight the fact that most northern communities^{Footnote 46} as well as mining developments rely upon diesel generation for electricity. The North contains potentially vast hydro resources, which implies not only an economic opportunity, but also an expedient way to reduce greenhouse gas emissions. Other renewable energy sources (e.g., wind, biomass, geothermal) all have potential to reduce the North's reliance upon imported diesel. To some extent, SINED funds have increased attention on these areas.

b) Impact of SINED investments in mining

Between 2007-08 and 2011-12, SINED invested $5.8 million in mining. The case study covered capacity/knowledge-building projects amounting to $2.5 million, which represents 43 percent of the total SINED investment in mining. The projects reviewed in the case study were in the Northwest Territories and Yukon and were focused on training development, facilities, "asset acquisition" and additions to simulators. Recipients were the Mine Training Societies in the Northwest Territories and Yukon.

The impacts of the above investments documented in the case study are as follows: In the Northwest Territories, $226,000 of SINED funding allowed for the training of 22 participants and purchase of training equipment (which appears to have been donated to communities). Further results, such as the number of additional trainees and employment afterwards, were not provided.

In contrast, performance measures collected by the Yukon Mine Training Association showed a direct impact of SINED investments on employment. The Yukon Mine Training Association trained 852 applicants and 457 graduates obtained jobs between April 2008 and March 2011^{Footnote 47}. In 2010-11, almost 200 trainees in the Yukon were placed in careers^{Footnote 48}.

c) Impact of SINED investments in geoscience

Between 2007-08 and 2011-12, SINED invested $23.6 million in geoscience. The case study covered capacity/knowledge building projects amounting to $20.3 million, which represents 86 percent of the total SINED investment in geoscience. The projects reviewed in the case study were in all three territories and included geophysics (i.e., magnetic property of rocks), geochemical (i.e., analysis of soil samples for mineral composition), and geological maps and geo-technical reports (i.e., for sand, gravel, permafrost). The project proponents examined in the case study were the Canada Nunavut Geoscience Office, the Department of Economic Development and Transportation of the Government of Nunavut, Industry Tourism and Investment from the Government of Northwest Territories and the Yukon Geological Survey.

The case study showed that SINED funding has been instrumental in increasing the availability of geoscience information for the territories' mineral industry and territorial, municipal and community planners. The Canada Nunavut Geoscience Office calculated the extent to which geoscience was made available to the public by looking at the amount of information available and the number of downloads. In Nunavut, 4 terabits^{Footnote 49} of information is available online and approximately 8,000 reports have been downloaded. Similarly, a data mining project undertaken by the Yukon Geological Survey resulted in the scanning of 4,364 documents comprising 2.29 GB of data, which is new information available for use.

Even though SINED geoscience investments are meant to contribute to the immediate outcome "expanded publically-accessible information", the ease of access to geoscience information by the public was not monitored by SINED. In response, the EPMRB survey of geoscience users in Nunavut asked respondents to comment on their experiences identifying and accessing geoscience data through the Canada Nunavut Geoscience Office. The majority of respondents indicated that the information they were looking for was easy to access and the ease of obtaining copies of reports ranged from "easy" to "very easy". Similar information was not available for Yukon and Northwest Territories.

Geoscience information is used by private sector companies and has contributed to leveraging private sector investments in the territories. For instance, preliminary results of the "Yukon Geological Survey" Client Questionnaire for 2012-2013 reveal that geoscience information has resulted in staking of claims, informed investment decisions, helped companies identify new areas of interest and reduced risk. All respondents reported that they "are either planning to, or already have, integrated the data into their corporate datasets". Further, press releases by companies (Arcturus Ventures Inc. and Commander Resources Ltd.) exploring in the Yukon state that "SINED-funded publications provided data that positively influenced exploration".

Similarly, the two exploration companies in Nunavut surveyed by EPMRB agreed that the geoscience information reduced risk, the amount of time required to advance exploration and reduced company costs by identifying areas not viable for exploration. Both companies indicated they have invested as a result of Nunavut geoscience information.

The case study found that geoscience information has been used by federal, territorial, municipal governments and First Nations to inform land use planning. Interviews with territorial government representatives in Nunavut revealed that geoscience data has been used to plan the location of a new airport and deep water harbour. Interviews with territorial government representatives in the Northwest Territories showed that geoscience information has been used in the planning of the route for the extension of the Dempster highway to Tuktoyaktuk. The survey conducted by EPMRB of geoscience users in Nunavut revealed that geoscience information has also been used by Aboriginal organizations to make decisions regarding cultural/heritage sites and has reduced the costs of infrastructure planning and land use planning in communities.

The geoscience case study also shows new discoveries. According to the Canada Nunavut Geoscience Office, there have been discoveries of 11 new potential sites for soapstone and nine undeveloped deposits contain substantial resources of high-quality stone. These discoveries of raw material for high-value-added fine art carvings have been well received by local artists and communities. In the Yukon, a geoscience study led to the identification of a source of aggregate half the distance of the existing site. The local community estimated a potential savings of $34,000 in hauling costs^{Footnote 50}.

As shown above, there are many indicators that SINED investments in geoscience have resulted in positive impacts, but since these impacts are measured or reported on differently in each territory and for each project, the evaluation was unable to conclude on the overall impact of all investments.

d) Impact of SINED investments in fisheries

Between 2007-08 and 2011-12, SINED invested $2 million in fisheries. The case study covered capacity/knowledge-building projects amounting to $1.9 million, which represents 95 percent of the total SINED investment in fisheries. The projects reviewed in the case study were in Nunavut and included research primarily relating to fish stock assessments, however, some funding was also provided for the testing of fishing vessels and a fisheries symposium. The project proponents examined in the case study were Department of Economic Development and Transportation and the Department of the Environment from the Government of Nunavut.

The fisheries case study found that in Nunavut, the annual offshore surveys, supported by SINED, contributed to a 90 percent increase in the turbot quota. The total landed value of turbot in Nunavut increased from $35.4 million in 2001 to $78.8 million in 2012^{Footnote 51}. Stock assessments conducted for shrimp and arctic char also contributed to increases in catches. The total landed value of shrimp in Nunavut increased from $12.1 million in 2001 to $32.6 million in 2012^{Footnote 52}.

The increases in quotas have contributed to growth and private sector investment in the fisheries sector. The case study found there are now four 100 percent Inuit-owned companies, which hold 100 percent of Nunavut's share of offshore shrimp and turbot allocations. When the industry began, these companies did not own a single vessel, however, in 2013 they have two offshore factory freezer trawlers and three large fixed gear vessels^{Footnote 53}.

e) Impact of SINED investments in tourism

Between 2007-08 and 2011-12, SINED invested $16.1 million in tourism. The case study covered projects amounting to $7.1 million, which represents 44 percent of the total SINED investment in tourism. The projects reviewed in the case study were in all three territories and included marketing projects, strategy development, feasibility studies, tourism infrastructure, product development, visitor exit surveys and research. The major recipients in each territory were examined in the case study and included the Department of Economic Development and Transportation from the Government of Nunavut, Nunavut Tourism, Northwest Territories Tourism Association, Department of Industry Tourism and Investment from the Government of Northwest Territories, Carcross/Tagish Development Corporation, Department of Economic Development, Department of Tourism from the Government of Yukon and Tr'ondëk Hwëch'in First Nation.

Sixty-eight percent of the projects looked at in the case study were marketing projects so, naturally, the case study sought to document the impact of SINED investments in this area. Although many measures for marketing campaigns were identified in the case studies, these measures were not uniformly applied to all marketing activities across the three territories. This can be seen in the impacts documented by each of the territories:

1. The Northwest Territories analysis of marketing activities looked at the impact of individual marketing campaigns. For example, during the 2009 "Spectacular Northwest Territories" marketing campaign, the number of Twitter/Facebook followers increased from 4,000 to 19,000^{Footnote 54}. In addition, over 50 percent of visitors recalled having seen "Spectacular Northwest Territories" advertisements. In 2008, 53 percent recalled the marketing campaign for that year compared to 32 percent who recalled the 2007 advertisements. The 2010 Globe and Mail campaign resulted in over 1,200 entries into a contest associated with the campaign, 500 information requests and 191 print guide packages being sent out^{Footnote 55}. Finally, a pavilion at the 2010 Olympics which was funded by SINED had between 6,000 and 8,000 visitors daily. Additionally, it was featured in publications, newspapers and the nightly news. It was calculated that the resulting exposure was the equivalent of purchasing more than $150 million in advertizing^{Footnote 56}.
2. Alternatively, the Yukon reported on the overall impact of tourism initiatives. Outcomes recorded for 2010-11 indicate that the Yukon implemented marketing initiatives with a consumer reach of over 23.6 million; hosted 81 media journalists generating the equivalent of over $31.6 million in advertising; generated 230,505 requests for tourism information; and had 241,879 unique visitors to the tourism website^{Footnote 57}.
3. Nunavut Tourism took a similar approach to the Yukon and linked the success of their marketing campaigns to the following outcomes: 1) Increased requests for tourism materials such as travel planner, CD-ROM, maps, DVD and other promotional materials; 2) Increased media exposure through travel articles and television shows; 3) Increased bookings for market-ready operators; 4) New promotional materials; and 5) Number of images in photo library and hours of high quality video available for tourism promotion^{Footnote 58}.

As shown above, the case study revealed many useful measures for documenting the success of marketing activities. Unfortunately, the lack of consistency between territories does not allow for a comparison of marketing activities nor an analysis of overall impacts of these investments.

In addition to marketing, SINED invested in tourism infrastructure and products such as interpretation centres, signage, beautification and aerial parks with a number of different partners. The impacts of investments to increase the number of visitors to Carcross/Tagish First Nation in the Yukon are particularly well documented. SINED provided funding for the establishment of a carving program, which led to the training of new carvers and the creation of 10 totem poles to attract visitors to the community. SINED also invested in the development of 35 km of bike trails, which employs 13 youth. The Mountain Hero Trail, which is part of this network, has received national and international attention as well as an "Epic Rides" designation from the International Mountain Bike Association^{Footnote 59}. The case study documented an increase in employment and mentoring of over 20 First Nation and non-Aboriginal youth over past seven years^{Footnote 60} and an increase in visitors to the Carcross Tagish First Nation community, from 150 before the projects, to 2,000 in 2011.

SINED projects also attracted private sector investment. $9,000 was leveraged from hotels, companies, and First Air for the annual tourism conference in the Northwest Territories. SINED investments in tourism infrastructure and products at Mt. Sima Alpine Adventure Park, combined with the leveraged funds, led to an increase in season pass sales from $30,000 to $90,000 and the involvement of new sponsors (Royal Bank and Air North). In addition, nine tour operators confirmed packages to bring groups to the attraction^{Footnote 61}.

Not so successful were investments made in the Great River Journey project in the Yukon. Between 2004-05 and 2008-09, SINED invested $3.1 million in this project. Two early independent studies of the project were very positive. However, when the project proponents indicated they needed an additional $1.2 million in operating funding to stay afloat in 2010, AANDC commissioned another third-party review to assess performance to date, the business model and long-term viability of the project. The assessment concluded that the business model was flawed, and even if the best-case scenarios for demand came to pass, the venture would operate at a loss. In response to this assessment, there have been no further investments in this project and according to the Yukon News, the eco-tourism company is bankrupt^{Footnote 62}.

Finally, the impact of SINED investments in tourism can be measured indirectly through visitor surveys. The 2001-02 Northwest Territories visitor survey recorded 53,917 visitors that spent $103.8 million. The 2010-11 visitor survey recorded 69,830 visitors that spent $110 million. The 2009 Northwest Territories visitor survey notes that 63 percent of Northwest Territories visitors can be considered Northwest Territories "promoters" of tourism, based on their willingness to recommend the Northwest Territories to others (up from 54 percent in 2008). While the visitor surveys are excellent tools for assessing the health of the tourism industry, their contents are inconsistent between territories and vary in year and frequency, so cannot provide a bigger picture of the impact of SINED investments in tourism.

f) Impact of SINED investments in information communication technologies and broadband

Between 2007-08 and 2011-12, SINED invested $3.4 million in information communication technologies and broadband. The case study covered projects amounting to $1.2 million, which represents 35 percent of the total SINED investment in information communication technologies and broadband. The projects reviewed in the case study were in all three territories and included training, infrastructure planning, feasibility studies and investments in information communication technologies/broadband infrastructure. The recipients examined in the case study included the Nunavut Broadband Development Corporation, the Department of Finance from the Government of Northwest Territories, Northern Native Broadcast Corporation and Katlodeeche First Nation in the Northwest Territories.

In 2010, SINED funded the Arctic Communications Infrastructure Assessment Report, which provides arguments and recommendations for the development of a reliable communications network in the North to establish sovereignty, service communities and support emergency response services. The final report is accessible through a companion website and, according to key informant interviews, has been referenced in Canadian Radio-television Telecommunications Commission hearings, newspapers and magazine articles. It has also been used as a source document for other organizations to support the prioritization of communications projects and has guided other agreements like the development of a Mackenzie Valley fibre link. In addition, the ten recommendations that came out of the report guide the Northern Communications and Information Systems Working Group, which is looking at how to address the recommendations.

The Nunavut Broadband Development Corporation received SINED support for two studies in 2011: one looked into quantifying the social and economic benefits of high-speed internet infrastructure to build a case for continued investment. The second was a study of the feasibility and cost of fibre-optic connectivity in Nunavut. Both studies had findings and recommendations, but the extent to which the studies have been made available to, and accessed by, the public is not known. Information on the extent to which recommendations have been implemented was not provided.

In contrast, the impact of infrastructure investments in information communication technologies are well documented by the Katlodeeche First Nation in the Northwest Territories. With SINED funding, Katlodeeche installed a mixed network of fiber-optic cable and Wi-Fi to provide internet access to the band office, day care, group home and community members. This investment has resulted in cost savings of $35,000 per year in licensing fees for the band office. They have also saved $5,000 per year in external technical services because they hired and trained someone from the community to provide technical support. This technician has held the position for four years. The network also allowed them to introduce video-conferencing, which has reduced travel expenses. Prior to the availability of video conferencing, Chief and Council were spending approximately $135,000 on travel. In 2011, travel expenses were $37,000 and, in 2012, they were under $18,000^{Footnote 63}.

The infrastructure project has benefitted Katlodeeche First Nation in other ways. The new network provides a secondary communications system if phone lines go down, as happened after a 2007 fire. It improved safety in the community and also allowed for the delivery of better education through online education programs. The project has also opened up some business/income earning opportunities. The band office is currently looking into investing in 12 km of fibre-optic line to connect their network to Northwest Tel. This would provide better download speeds, but they could also potentially lease their infrastructure back to telecommunication companies wishing to move north.

g) Impact of SINED investments in cultural industries

Between 2007-08 and 2011-12, SINED invested $4.7 million in cultural industries. The case study covered projects amounting to $2.4 million, which represents 51 percent of the total SINED investment in cultural industries. The projects reviewed in the case study were in Nunavut and included marketing initiatives, strategy implementation, training and planning. The recipients examined in the case study included the Department of Economic Development and Transportation from the Government of Nunavut, Nunavut Film and the Nunavut Arts and Crafts Association.

SINED investments in the film industry in Nunavut supported training, global marketing and producer tours, however, beyond anecdotal evidence of success, information on the impact of these investments could not be found. The key informant interviews also revealed that many social benefits of the film industry are impossible to measure such as the pride film makers have in their work and the hope and self esteem that comes with it.

The impact of SINED investment in the arts and crafts industry was also not well documented. While key informants identified many successes, these were not captured in available reports. Outcomes identified in the interviews included the construction of studio space and the showcasing of Nunavut art in Paris, which led to the purchase of $20,000 worth of art by a gallery. An art auction that led to the $44,000 in sales where artists got 100 percent of their profits could not be confirmed, nor could the leveraging of funds.

It was mentioned that with $200,000 of SINED funding, the Nunavut Arts and Crafts Association was able to leverage an additional $600,000 from other sources, including the Government of Nunavut, the Department of Canadian Heritage and Inuit organizations. However, leveraged funds were not documented in available reports. Essentially, year-end reports focused on activities and not on the impact of those activities.

On a positive note, under the period covered by the evaluation, SINED funds have contributed to upgrades to the Nunavut Arts and Crafts Association's website, which has resulted in an increase in new users. For example, in 2011-12, the website received 2,657 visits of which 74 percent were new visits. Most visits were from Canada (2,058), followed by the United States of America (160), France (136), Germany (90), United Kingdom (74), Bangladesh (13), and India (11)^{Footnote 64}. The evaluation found that for a territory that is so undeveloped, these numbers are significant and encouraging.

h) Impact of SINED investments in capacity

If a broad definition of capacity, as outlined in Section 2.3.2, is consistently applied to the entire list of SINED projects, the total SINED investment in capacity was calculated at 79 percent. In contrast, CanNor classified 30 percent of projects under the "capacity development" theme.

As shown above, SINED capacity investments totalled $66.9 million from 2007-08 to 2011-12. Eighty percent of Targeted Investment Program, seventy-four percent of Innovation and Knowledge, fifty-nine percent of Partnership Advisory Forums and fifty-seven percent of Pan-Territorial Fund investments have been directed towards capacity building. One hundred percent of SINED investments in fisheries, forestry, geoscience and small and medium-sized enterprises have been for capacity building. The breakdown of the types of capacity investments are as follows:

As shown in Table 5, overall, the most frequent type of capacity investment was research projects. In fact, fifty percent of SINED capacity expenditures went towards research projects largely in the area of geoscience. Training/skills development and loan top-ups were the second and third most frequently supported capacity projects overall.

Also interesting to note is the difference between the type of capacity projects supported by the separate funding streams. Targeted Investment Program and Pan-Territorial Fund capacity expenditures went towards research projects, while Innovation and Knowledge capacity projects were mostly in training/skills development and conferences/workshops/etc. Most of Partnership Advisory Forums investments in capacity went towards consultation/expert knowledge and conferences/workshops/etc.

Given that a large proportion of projects covered by the sector case studies were capacity projects, some of the impacts of capacity projects are discussed in sections a) to g) above.