Evaluation of the Northern Adult Basic Education Program (NABEP) from 2015-2016 to 2019-2020 (JULY 2021 REPORT)

Introduction

The Northern Adult Basic Education Program (NABEP) was established in 2011-12 as an education support program designed to help Northerners get targeted training so they can participate more fully in the labour market. The NABEP objective is to expand the territorial post-secondary educational institutions' basic workplace skills programs, particularly in remote communities and improving access to adult basic education.^{Footnote 1} The program is delivered through the three territorial colleges in Canada's North: Aurora College, Yukon College (now University), and Nunavut Arctic College (NAC).

The purpose of this evaluation was to review NABEP's relevance to federal government priorities and goals; achievement of expected short-, medium-, and long-term outcomes; NABEP's cost-effectiveness and efficiency; and recommendations for future programming. The scope of the evaluation covered the five years since the last evaluation of the Northern Adult Basic Education Program (NABEP), from 2015-16 to 2019-20.

Methodology and limitations

The evaluation deployed three lines of evidence as outlined in the Evaluation Plan: document review, interviews, and case study. For interviews, the evaluation team completed 28 interviews with program stakeholders. For document review, a total of 66 documents were submitted or otherwise identified by the evaluators for review. For the case study, the evaluation team focused on Curriculum Development.

Three risks that could impact on the evaluation were identified: 1) Limited Time (5 weeks) for Data Collection; 2) Data Consistency; and 3) Indigenous Interviewees in Remote Locations. All risks materialized, with limited success in the mitigation strategies employed. CanNor's overall readiness for the evaluation was a fourth risk that emerged.

Conclusions and Recommendations

Regarding Relevance, the evaluation concludes that there is a continued and ongoing need for Adult Basic Education (ABE) services in the North, and the NABEP fills that gap. While skills development can be a component of ABE, it should not replace ABE, which is at the core of needs and NABEP. Improved literacy should be the core outcome of the Program. There are differences across the three territories, and changes in socio-economic conditions in each territory that require flexibility in design and delivery.

The program is appropriate for both the federal government and CanNor, but it is not optimal. A longer-term scenario would be the Federal Government helping territorial governments taking responsibility for future ABE services. However, in lieu of such a mechanism and given precedent, it is incumbent upon CanNor to fill the gap to the best of its ability.

Regarding Effectiveness, NABEP is delivered in a complex operating environment in all three territories, each presenting unique challenges but sharing a target audience that is remote, often vulnerable and facing a range of barriers to their accessing and completing learning opportunities. It requires stability in funding, active and effective monitoring and evaluation for continuous improvement, and the flexibility to adapt to a changing landscape.

The evaluation shows that progress has been made on all expected outcomes. The Delivery Partners are experienced educational institutions with the expertise to deliver the program. Program management does vary across institutions, and there is opportunity to better share experiences and best practices in program management. Opportunities for improvement have been identified for timeliness of funding decisions, sharing of best practices and overall program improvements, and data collection and analysis for decision making. The complex operating environment requires a clear results framework, aligned to the scope of the Program, that does not overreach in terms of intended results from an ABE program.

Regarding Efficiency, the Program has expended funding as per budgets (-6% variance overall), although NAC had significant under-expenditure in two of the five years under review. On the part of CanNor, the program has been operated at a low cost, but with limited program oversight and accountability, which can impact Delivery Partners in delivering services and achieving results. The timing and length of the funding agreements could be better aligned to the operational realities of delivering ABE services in the North.

Regarding Best Practices, Delivery Partners are identifying lessons learned and best practices and adopting them in the continuous improvement of the program, although this has at times been interrupted by uncertain funding. Caution should be exercised when moving to online learning, as some data indicates that completion rates are lower than community-based, facetoface learnings. Online learning may not be appropriate for general application across the territories.

1.1 Background

Created in 2009, the Canadian Northern Economic Development Agency (CanNor) is one of Canada's six Regional Development Agency (RDA) under the leadership of the Minister of Economic Development and Official Languages, supported by the Innovation, Science and Economic Development (ISED).

The Northern Adult Basic Education Program (NABEP) was established in 2011-12 as an education support program designed to help Northerners get targeted training so that they can participate more fully in the labour market. The program is delivered through the three territorial colleges in Canada's North: Aurora College, Yukon College (now Yukon University), and Nunavut Arctic College.

The objective of NABEP is to expand the territorial post-secondary educational institutions' basic workplace skills programs, particularly in remote communities and improving access to adult basic education.^{Footnote 3}

1.2 Purpose and scope

This evaluation was required to meet a commitment made in the last NABEP funding renewal Treasury Board submission (2020) to complete an evaluation of this initiative in 2020-2021; to review and analyze NABEP performance, opportunities for collaboration with northern partners; and to inform decisions on the future of adult basic education in the territories. The last evaluation of NABEP was completed in 2016.

More specifically, the purpose of this evaluation was to review:

• NABEP's relevance to federal government priorities and goals;
• Achievement of expected short-, medium-, and long-term outcomes;
• NABEP's cost-effectiveness and efficiency; and
• Recommendations for future programming.

The scope of the evaluation covered the five years since the last evaluation of the Northern Adult Basic Education Program (NABEP), from 2015-16 to 2019-20. The evaluation is based on the following questions (the full Evaluation Matrix is found in Annex A):

2.1 Program description

The Northern Adult Basic Education Program (NABEP) was established to fulfill the Budget 2011 commitment to expand territorial colleges' literacy and numeracy programs, particularly in remote communities, to target working-age Northerners and assist them in accessing the basic skills they need to join the workforce and take advantage of emerging economic opportunities.

NABEP supports skills development and innovation by helping Northerners adapt to the changing nature of work and encouraging a culture of lifelong learning. The program extends educational opportunities to remote communities in the territories, particularly in Nunavut. The program's Delivery Partners, namely the Aurora College, Yukon University, and Nunavut Arctic College, engage communities to assess demand so that student progress is supported without the need for relocation. The program is tailored to the needs of each territory, and in general, remote communities are further supported where qualified community members and Elders are contracted to deliver course material. With differences across the different regions, in general the development of skills curriculum demonstrates flexibility by relying on resource industry partnerships to facilitate co-op learning and by incorporating Indigenous language and knowledge to enhance delivery. Online course delivery has also been developed to function in areas with limited connectivity and bandwidth.

2.2 Results framework for the Program

The objective of NABEP is to expand the territorial post-secondary educational institutions' basic workplace skills programs, particularly in remote communities.

The outcomes and activities identified in the Statement of Work are as follows and are represented in a Logic Model in Annex B:

• Long-term Outcome: Improved employment, occupational training, post-secondary training (non-occupational) and readiness for employment or occupational training for working-age Northerners.
• Intermediate Outcome: Maintained use of and increased completion of services by target cohort of adults.
• Immediate Outcome: Increase in availability of adult basic education services.

The NABEP activities are undertaken by the Delivery Partners – the Arctic College in Nunavut, the Aurora College in the Northwest Territories and Yukon University. The funding also provides one full-time equivalent (FTE) for CanNor oversight of the Program.

2.3 Partners and stakeholders

The main stakeholders are categorized as per the following table:

2.4 Financial information

Please see Section 6.1 where financial budget and expenditure information is presented.

3.1 Methodology

The evaluation deployed three lines of evidence as outlined in the Evaluation Plan: document review, interviews, and case study.

For interviews, the evaluation team planned to conduct between 30 and 44 interviews with program stakeholders, and completed 28 as per the following breakdown with actual numbers being as of April 6th, 2021:

A further breakdown of the interviewee groupings is found in Annex C: Interviewee List.

For document review, a total of 66 documents were submitted or otherwise identified by the evaluators for review. A detailed breakdown of the evaluation documents is found in Annex D: Master Document List.

For the case study, a focus on Curriculum Development was identified in the Evaluation Plan. The evaluation team expected to complete 2-3 interviews per territory plus a document review. A total of 7 interviews were conducted.

3.2 Limitations

Section 3.4 of the Evaluation Plan (dated February 11, 2021) identified three risks that could impact on the evaluation: 1) Limited Time (5 weeks) for Data Collection; 2) Data Consistency; and 3) Indigenous Interviewees in Remote Locations. All risks materialized, with limited success in the mitigation strategies employed. CanNor's overall readiness for the evaluation was a fourth risk that emerged.

An evaluation best practice is to have potential interviewees identified and a near complete set of program documentation ready for review prior to the data collection phase. Given the compressed data collection period of five weeks, this evaluation readiness was important. Neither the evaluation interviewee list nor a complete set of program documents were ready in advance of the data collection period. As a result, the data collection period needed to be extended by three weeks to ensure an adequate number of interviews were completed (with still some gaps remaining), and to collect fundamental program documentation (e.g., annual performance reports, etc.).

The most significant limitation is gaps in fundamental program documents. The initial set of documents supplied by the program contained 5 of an expected 15 annual performance reports. Requests were made through the Project Authority, and directly to the CanNor Regional Offices, which did not result in further documents. Eventually the evaluators requested them directly from the Delivery Partners. In the case of Yukon University, neither CanNor nor the University could provide the requested reports covering the entire period of the evaluation (2015 to 2020). This has resulted in a gap in the performance data required for analysis. In a similar nature, not all Contribution Agreements covering the entire period have been provided. The lack of performance data impacts the assessment of program effectiveness.

Regarding interviews, the relatively short timeframe and limited understanding or engagement of the Regional Offices in identifying territorial representatives, meant that no territorial representatives were interviewed for the evaluation. This data collection gap impacts the Relevance review questions, and the "necessary" role of the federal government in adult literacy in the North.

4.1 Is there a continued need for the Program?

This question was further broken down into three sub-questions, finding for which are presented in the following section:

• Who are the stakeholders and what are their needs?
• Have those needs changed in the last five years? Has the program changed to address them?
• How will those needs change in the future?

4.1.1 Who are the stakeholders and what are their needs?

The following table presents the NABEP stakeholders as identified by all lines of evidence.

Table 1: NABEP Stakeholders

Target beneficiaries	Adult Learners/Course participants in the North (with below secondary school literacy and numeracy proficiency)
Primary stakeholders	Delivery Partners, namely Nunavut Arctic College (NAC), Aurora College and Yukon University, that are comprised of management, staff, and Adult Educators
Secondary stakeholders	Territorial government departments Industry First Nation (FN) self-governing bodies Communities Like-minded organizations and contracted experts

Regarding stakeholder needs, according to document review, a significant proportion of adults in the North do not have the proficiency level for most literacy tasks needed in society and in the workplace, with the proficiency levels of Indigenous peoples on average being lower than non-Indigenous people. People with lower literacy levels are much less likely to be employed, work fewer weeks per year on average, are more likely to go through periods of unemployment, and remain unemployed for much longer periods.^{Footnote 4}

Table 2: Proportion of the population with literacy proficiency Level 2 or below, aged 16 to 65, 2012^{Footnote 5}

	Total population	Non-Indigenous	Indigenous
Yukon	44%	37%	69%
Northwest Territories	64%	44%	82%
Nunavut	84%	34%	92%
Canada	49%	-	-

The gap in proficiency levels between Indigenous and non-Indigenous is supported by other data. For example, there is a significant difference in educational outcomes between Indigenous and non-Indigenous peoples in the North, where 50% of the Indigenous adult population has not completed high school, compared to the 11% of the non-Indigenous population. In June 2020, 18.6% of the territorial adult Indigenous population was unemployed compared to 3.4% of the non-Indigenous population.

Text alternative for Table 3: Population aged 25-64 without Secondary School Completion 2014-19

Description of Chart: The line graphs presents the portion of off-reserve Indigenous, non-Indigenous, and total population aged 25 to 64 without secondary school completion from 2014 to 2019 for each territory as well as Canada. The proportion of the off-reserve Indigenous population without secondary school completion is approximately 20 percent higher than the total population in each territory, and only 10 percent in Canada.

The Nunavut, NWT, Yukon, and Canada graphs show that the number of off-reserve Indigenous population without secondary school completion is higher than the total population in each territory.

Territory	2015-16	2016-17	2017-18	2018-19	2019-20
NWT	620	671	552	529	526
Nunavut	343	348	328	272	306
Yukon	272	298	265	265	228

While the need for skill development support is consistent across the territories, the extent of the gap in literacy and numeracy skills does vary, with Nunavut having the greatest gap. In addition, caution should be exercised when directly linking the program's results to employment, as there are several external factors that influence employment, such as the economic prospects in each territory.

Interviews also supported the findings from document review in terms of literacy proficiency levels in the North, and the gap between Indigenous and non-Indigenous. For the Delivery Partners, the need identified in interviews is for resources to deliver programs centred on the Adult Learner, addressing their learning needs with trained teachers and appropriate methodologies, culturally appropriate content and materials in a manner that maximizes availability, access, and completion (retention).

4.2 Is the Program consistent with government priorities and CanNor strategic objectives?

The alignment to federal priorities is based on NABEP's ties to skills development, more than its core program mandate of adult literacy, which would fall under provincial and territorial responsibilities. NABEP's skills development component is consistent with government priorities as evidenced by announcements in Budget 2017 and Budget 2019 and that the Program has undergone three renewals during the evaluation period. As an example, the 2017 program renewal was also announced in Budget 2017, under "Creating more opportunities for Indigenous Peoples" under Part 1 – Equipping Canadians with the Skills They Need to Get Good Jobs, under Chapter 1 – Skills, Innovation and Middle Class Jobs. NABEP is also aligned to the education, ongoing learning, and skills development aspects of the Arctic and Northern Policy Framework.

Similar to the above, alignment of NABEP to CanNor's priorities and objectives is based on its skills development component. NABEP contributes to CanNor's Strategic Outcome "Developed and diversified territorial economies that support prosperity for all Northerners" in the Agency's results framework, falling under Economic Development Program and the Community Development Sub-Program. NABEP also aligns to the CanNor Pan Territorial Growth Strategy (PTGS) under a skilled workforce, and the proposed action of increasing the availability of ongoing skills development opportunities, including Indigenous-based knowledge and skills, through investments in training infrastructure as well as the development of certification standards, curriculum, or other materials.^{Footnote 7}

The Program is consistent with government priorities and CanNor strategic objectives. An individual cannot participate in federal skills training programs without the pre-requisite literacy and numeracy proficiency, which is part of the continuum of learning. The challenge is for federal and provincial/territorial partners to work collaboratively within this continuum to avoid jurisdictional issues or siloed programs that can work against achieving the desired result.

4.3 Is there a legitimate, appropriate and necessary role for the federal government in the Program?

Regarding legitimate role, NABEP has been announced in federal budgets, has undergone renewal three times through Treasury Board submission processes, and is aligned to federal priorities and Ministerial mandate letters (i.e., Indigenous Services Canada (ISC), Crown-Indigenous Relations and Northern Affairs Canada (CIRNAC), Employment and Skills Development Canada (ESDC), and Innovation Science and Economic Development Canada (ISED)). All interviewees indicated that the program is a necessary, appropriate, and necessary role for the federal government.

Regarding appropriate role, federal and territorial responsibilities for both education and skills training are evidenced by the Budget 2017 and 2019 and announcements, the Aboriginal Skills and Employment Training Strategy, and the establishment of the Task Force on Northern Post-Secondary Education led by CIRNAC and including a range of territorial stakeholders. The NABEP falls between secondary and post-secondary education and skills training, both of which are shared federal and territorial responsibilities. The NABEP deals with an underserviced group of adults who have left and are outside the formal educational system and lack the pre-requisite literacy and numeracy proficiency levels to undertake skills training or pursue employment opportunities.

A Horizontal Skills Review as reported in Budget 2019 indicated that there are 30 government departments delivering over 106 skill development programs at approximately $7.5 billion annually.^{Footnote 8} Amongst this landscape it is difficult to justify one more skills development program, and NABEP should retain a focus on adult literacy to remain relevant.

Regarding necessary role, there was less certainty with interviewees on the role of CanNor with the design of the program. The original program was established as an interim measure that has been renewed for over 10 years and a Grant and Contribution Program may not be the most appropriate delivery approach. In addition, the same Delivery Partners are selected each year, despite there being alternatives available in all territories. An alternative viewpoint was articulated by one Yukon interviewee that is included here as it encapsulates what could be a longer-term approach to addressing adult literacy in the North:

No, there is not a role for CanNor in adult basic education in the North. This should not be a proposal or project driven initiative. It should be an ongoing responsibility on the territorial government funded by the Government of Canada. I look at this from a different perspective. The ongoing idea that adult basic education programs would be run on a proposal basis by a service delivery organization is inappropriate. I do not believe that CanNor should have a responsibility to adult basic education. I believe that the link between economic activity and adult basic education should be broken.

5.1 To what extent did Program activities result in planned outputs?

There was considerable variance in the quality of the reporting by the Delivery Partners, making comparison of planned versus actual outputs difficult in the case of Nunavut and Yukon University. The Aurora College provided the highest quality annual reports in terms of adhering to the reporting requirements stipulated in the Contribution Agreement, as well as content and comprehensiveness. Aurora College reported a 95% completion rate (155 of 163) of planned versus actual outputs.

The Nunavut Arctic College reports identified outputs, but no comparison to work plans was made in the reporting. The reporting did however indicate outputs in areas such as program delivery (i.e., courses), ABE Instructors, Elder programs, capacity building of staff, curriculum development, assessment tools and research. As noted under limitations, the analysis of Yukon University's actual versus planned outputs was not possible.

The logic model (see Annex B) identified several categories of outputs. Under service delivery capacity/quality, the outputs include activities such as teacher training, staffing and tools. Some select examples from all partners in this area (illustrative only):

• Nunavut Wide NABEP Training in Iqaluit (August 2016): Community Adult Educators and Coordinators from Qikiqtaaluk, Kivalliq, and Kitikmeot regions participated in a 3-day professional development program in Iqaluit at the end of August.
• An Educational Leadership Framework (Aurora, 2019), which supports an Indigenous approach to leadership, was created to build a culture of leadership and to support the development of leadership capacities of Instructors and Adult Educators.

Also, under service delivery capacity/quality, there are clear indications on the part of Nunavut Arctic College and Aurora College that NABEP funding assisted in staffing ABE positions, mostly at community level. In the case of Nunavut, the funding was also used to contract elder instructors to participate in course delivery. As funding varied so did the positions funded, but as an example:

• In 2016-17, Nunavut Arctic College reported 14 additional ABE support instructors were hired and 56 Elder instructors and cultural expert instructors were contracted.
• In 2016-17, Aurora College reported an increase of 4.6 FTEs in ABE instructors, sometimes as new positions, and other times making part-time community positions into full-time positions.
• In 2016-17, Yukon University funding went to support program management in the form of the ABE Coordinator, Budget Officer and Researcher and one Instructor.

Partnerships with communities and others are evident in all three territories:

• For example, the Nunavut Arctic College has partnered with industry for delivery of courses in specific communities, such as: a Pre-trades program in Rankin Inlet, in partnership with Kivalliq Mine Training Society and Agnico-Eagle (2016-17); a Pre-Trades Math & Science Program at Mary River in partnership with Baffinland Iron Mines Corporation to deliver from January to April 2018.
• Partnership was also evidenced by Aurora College's consultative approach on program development and delivery through a structured and regular annual partners meeting. As an example, in 2018, two meetings were held with partners such as Aboriginal Skills and Employment Training Strategy, representatives of the Akaitcho Territory Government, Dehcho First Nations, Gwich'in Tribal Council, Inuvialuit Regional Corporation, Northwest Territory Metis Nation, Sahtu Dene Council, and Tlicho Government, NWT Literacy Council and the Department of Justice.
• Yukon University reports provided some examples of partnerships, including an Industrial Skills Program delivered in 2016-17 to prepare students for work at a mine site. Companies provided the equipment for practical experience (Victoria Gold, Alexco Resources, CNIM, Cobalt Construction, Nacho Hyak Dun First Nation). Of the 10 students, five found fulltime work either in the community or with one of the mines.

Regarding access to services, culturally appropriate materials are most evident in Nunavut, where inclusion of elders and adapting materials and curriculum to the Inuit peoples is clear. As a few examples:

• A custom Inuktitut resource binder for use in the ABE program was developed that gathered Inuktitut traditional knowledge and oral history for various topics. This was supported by two instructor guides to assist the instructors in using this local content in their existing ABE program.
• Resources for contracting Elders along with learning materials. NAC contracts Elders to lead practicum sessions in some of their LES courses. It has been very popular with learners and Adult Educators:

  Perhaps the most innovative contribution to ABE program success has been the investment in collaborating with Elders in the teaching and learning process. Elder instructors bring critically important traditional Inuit values and knowledge to the ABE curriculum. Both instructors and students say having Elders in the classroom is the top motivator for engaging Inuit students in their learning leading to better retention and graduation rates.^{Footnote 9}

• Yukon University developed and delivered a Yukon First Nations Art Certificate Program to support the Yukon's indigenous visual art forms while also enhancing academic skills of learners with the goal to pursue further academics or to engage in the Yukon's arts and cultural industries.

Interviewees supported that document review findings. CanNor interviewees indicated the Delivery Partners had specialized knowledge and were delivering on their plans. This was despite a lack of detailed knowledge on the part of CanNor interviewees of NABEP. In general, CanNor staff viewed the relationship as being transactional in nature, meaning they receive reports and transfer funding. There were comments about the quality of reporting but no indication that any reporting instructions were provided to the Delivery Partners to better address the needs of CanNor. One interviewee noted that results reporting is difficult given that the skills development and education sector is very fragmented at all levels – federal, territorial, and even within Delivery Partners own programs.

Other interviewees (i.e., Delivery Partners, Adult Educators, students) supported the finding that Delivery Partners complete work as planned. It is recognized that NAC had some difficulty due to changes within the College as well as the impact of COVID-19. There have been challenges in the last year (2020-2021) due to COVID-19, with a need to move to more remote and online teaching, and NABEP has provided Delivery Partners with the flexibility to be able to do that.

5.2 To what extent did the Program achieve (progress on) intended outcomes?

The following table presents a summary assessment of the achievement of NABEP's intended outcomes. The following sections provide the evidence and findings for each outcome.

5.2.2 To what extent is there maintained use of and increased completion of services by target cohort of adults?

Key findings

• Over the five years, NABEP has provided ABE services to 5,823 adult learners which is slightly higher than target (103%). There has been however some decline in enrollments in recent years as compared to 2015-16. Based on NAC and Aurora College numbers, the breakdown is:

  	% Female	% Indigenous
  Aurora College	58%	79%
  NAC	64%	99%

• In terms of overall completion rates, NAC completion rates only met targets once over the five year period, sometimes with significant variances. Aurora College completion rates were on target for four of five years, in some cases exceeding targets. There is no data for Yukon University.
• Completion rates can vary depending on the course, the community, and the cohort. In general, there has been better retention in the shorter LES type of courses. Factors for not completing have been identified under preceding questions.
• There are indications that online courses have lower completion rates (Yukon, PASS in Nunavut)

There were specific performance indicators for the assessment of use and completion. The following table provides a summary assessment of each indicator with the evidence presented in the following section:

Performance indicator	Assessment
i. Maintain annual number of unique individuals enrolled in adult basic education courses or programs (M/F, Indigenous, Youth).	Met targets, although there are variances with a general decline noted. Yukon University figures are not validated.
ii. Increase in the total proportion of unique individuals, and proportion of women and Indigenous peoples, enrolled in an adult basic education program who successfully completed their program.	Data incomplete. Aurora College met targets four out of the five years.

Maintain annual number of unique individuals enrolled in adult basic education courses or programs (M/F, Indigenous, Youth).

Table 9: (i.a) Number of individuals served vs target:

Partner	Target	2015-16	2016-17	2017-18	2018-19	2019-20	Total (%)[1]Footnote 13
Aurora	529	620	671	552	529	526	2,898 (109%)
NACFootnote 14	329	343	348	328	272	306	1,597 (97%)
YUFootnote 15	265	272	298	265	265	228	1,328 (100%)
Total	1,123						5,823 (103%)

Overall, the program has met its targets (103%) in terms of individuals enrolled, with each territory meeting, or exceeding targets: NWT 109%, Nunavut 97%, Yukon 100%). There have been variances from year to year, with a peak in 2016-17 for all three territories and a notable decline since then.

Table 10: (i.b) Number of ABE students served (disaggregated):

	Indigenous		Female		Others	Total
	Total	%	Total	%
NAC 2015-16	340	99%	202	59%		343
NAC 2016-17	344	99%	222	64%		348
NAC 2017-18	328	98%	211	63%		335
NAC 2018-19	272	100%	194	71%	9	272
NAC 2019-20	NR		NR			306
NAC sub-total	1284	99%	829	64%	9	1604
Aurora 2015-16	496	77%	359	55%	27	647
Aurora 2016-17	554	83%	392	58%		671
Aurora 2017-18	434	79%	334	61%		552
Aurora 2018-19	432	82%	307	58%		529
Aurora 2019-20	397	75%	310	59%	25	526
Aurora sub-total	2313	79%	1702	58%	52	2925
Yukon U 2015-16	234	86%	NR		38	272
Yukon U 2016-17	269	90%	NR		29	298
Yukon U 2017-18	227	86%	NR		38	265
Yukon U 2018-19	238	90%	NR		27	265
Yukon U 2019-20	NR		NR			228
Yukon U sub-total	968	-	-	-	-	1328
Totals	-	-	-	-	-	5857

The number of Indigenous peoples enrolled in ABE course as a percentage of total individuals was reflective of the demographics of each territory. It was consistently high in Nunavut (98%-100%), a territory where 86% of the population is Indigenous. Female participation fluctuated between a low of 59% to a high of 71%. In the NWT percentage of Indigenous people versus total students was also high, between 77% to 83% in a territory where approximately 50% of the population is Indigenous. Female participation was between 55% and 61%. In the Yukon, the percentage of Indigenous students varied from 86% to 90% in a territory where approximately 23% of the population is Indigenous.

Overall, there are variances each year with some decline noted in NWT in recent years, as well as Yukon. Nunavut figures are also down for community courses.

Text alternative for Figure 1: Course participants by territory 2015-16 to 2019-20

Description of Graph: The line graphs presents the total amount of course participants by territory from 2015-16 to 2019-20. The number of the course participants from Northwest Territories is approximately 300 participants higher than participants in Nunavut, and 400 participants higher than participants in the Yukon in 2015-16, and over the span of five years, the result changes to the number of participants in the Northwest Territories is approximately 200 participants higher than Nunavut, and 300 participants higher than Northwest Territories.

Partner	Target	2015-16	2016-17	2017-18	2018-19	2019-20
NWT	529	620	671	552	529	526
Nunavut	329	343	348	328	272	306
Yukon	265	272	298	265	265	228

ii.a Increase in the total proportion of unique individuals, and proportion of women and Indigenous peoples, enrolled in an adult basic education program who successfully completed their program.

The target for this indicator was set at 0.5% growth from a baseline that included completion rate for all students in each territory, as well as by Indigenous and Female. The baseline completion rates are found in the following table:

%	Total	Female	Ind.
YU	83	83	85
Aurora	46	49	45
NAC	46	42	46

In terms of overall completion, NAC completion rates only met targets once over the five year period, sometimes with significant variances. Aurora College completion rates were on target for four of five years, in some cases exceeding targets. The information submitted by Yukon University is not complete and thus it is not reliable.

Table 11: Completion rates

Partner and year	Number enrolled	Number completed	Percent actual	Percent target	Percent of target
NAC 2015-16	343	140	41%	46%	89%
NAC 2016-17	348	157	45%	46%	98%
NAC 2017-18	335	120	36%	46%	78%
NAC 2018-19	272	91	33%	46%	73%
NAC 2019-20	309	85	28%	46%	60%
NAC sub-total	1607	593	37%	46%	80%
Aurora 15-16	647	305	47%	46%	102%
Aurora 16-17	671	352	52%	46%	114%
Aurora 17-18	552	259	47%	46%	102%
Aurora 18-19	529	256	48%	46%	105%
Aurora 19-20	526	231	44%	46%	95%
Aurora sub-total	2925	1403	48%	46%	104%
Totals	4532	1996	44%	46%	96%

When further broken down by gender and Indigenous, the completion rates are (below overall average completion rate is marked as yellow). It is more likely that female participants will not complete a course than their male counterparts. Indigenous calculations are more difficult, as almost all participants are Indigenous in Nunavut, but in general, they meet or exceed the average.

Table 12: Disaggregated completion rates

	Overall	Indigenous completion rate	Female completion rate
NAC 2015-16	41%	41%	39%*
NAC 2016-17	45%	46%	41%*
NAC 2017-18	36%	37%	32%*
NAC 2018-19	33%	33%	33%
NAC 2019-20	38%	NR	NR
Aurora 2015-16	47%	49%	47%
Aurora 2016-17	52%	52%	52%
Aurora 2017-18	47%	47%	46%*
Aurora 2018-19	48%	45%*	33%*
Aurora 2019-20	44%	44%	44%
* Below overall average completion rate

Some information was found in Yukon University quarterly reports^{Footnote 16} on completion rates where they reported ABE courses and online course completion rates (for that quarter) as follows:

	Indigenous female	Indigenous male
ABE	45%	69%
Online	39%	54%

Lower completion rates for online courses were also noted in Nunavut PASS courses. For example, in 2019-20, 22 out of 150 participants completed PASS with a further 17 in progress, imply a maximum of 39 out of 150 participants would complete, or a 26% completion rate. This compares to community based courses during the same period, with 156 participants and 87 either completed the program/course or are still in progress, or a 56% completion rate.

Most interviewees indicate that completion rates can vary depending on the course, the community, and the cohort. In general, there has been better retention in the shorter LES type of courses. Factors for not completing have been identified under preceding questions.

5.3 What factors (internal and external) impacted the achievement of outcomes?

6.1 To what extent was the Program delivered as planned (budget versus expenditure)?

Overall, the program expended funds as per budget with a minus 6% variation overall. There were some significant variances from year to year with Nunavut Arctic College with overall expenditure 17% less than budget. Major variances in NAC expenditure were noted in 2017-18 (minus 32%) and 2018-19 (30%). Rationales provided included significant turnover at the executive levels of the College that disrupted decision making on program activities.

There is a slight over-expenditure related to the Yukon University of 9%, or approximately $342,000 over the five years. No contribution agreements covering these increases were reviewed by the evaluation.

6.2 To what extent was the Program delivered efficiently?

There is a lack of document management on the part of CanNor, and in some cases the Delivery Partners. There were significant delays in receiving annual reports and complete and correct contribution agreements. CanNor (headquarters and regional offices) were only able to provide 5 of 15 annual reports. In the case of the Yukon University, only one full annual report was received from the University.

Reporting quality also varies amongst Delivery Partners with Aurora College providing the most consistent and comprehensive reporting, often going beyond the requirements stipulated in the contribution agreements. Nunavut Arctic College reporting was also acceptable, with additional performance information added when it was collected.

The reporting requirements as per the contribution agreements, changed from the period 2015-17 to the new funding cycle of 2017-20. The 2015-17 period has 12 performance indicators, of which the data for 4 indicators was difficult to collect due to the need to follow up with students several months after completing a course. For the 2017-20 period, the performance indicators were reduced to 5, then subsequently to 3. Two of the three Delivery Partners retained the practice of reporting on the original 12 indicators to allow for some time series analysis over the evaluation period.

In some cases, quarterly progress reporting was a requirement. In some cases, in the period 2017-20, this was reduced to twice a year reporting. Finally, contribution agreements were often finalized late in the fiscal year, which can hinder program planning and delivery.

Most interviewees indicate Delivery Partners are efficient in maximizing resources to address community needs. According to Delivery Partner interviewee, late announcements, late agreements or changing decisions can negatively impact program delivery. Also, the complex operational environment can be challenging.

As an example, housing for Adult Educators in remote communities can be limited (Nunavut). Prompted by COVID-19 pandemic, there has been a move to online learning, which can be challenging given participants limited (computer) literacy, equipment, and bandwidth.

The move to more community focused and tailored short LES courses can be difficult to achieve. As much as possible, Delivery Partners try to apply tailored courses in another community if possible and requested. As an example, Yukon develops scalable course frameworks which can be applied in other communities with some adjustments, assuming other communities identify the same need (e.g., House Maintenance, etc.).

6.3 Are there alternative approaches that would be more efficient or effective at achieving expected outcomes.

7.1 What are the lessons learned, best practices and areas for improvement?

Best practices that have stood out from the review of documents:

• In Nunavut, the inclusion of elders as instructors in some courses has been seen as a positive development by both AEs and students. The allocation of resources for contract elders and supply necessary materials can be considered a best practice.

  "Both instructors and students say having Elders in the classroom is the top motivator for engaging Inuit students in their learning leading to better retention and graduation rates."^{Footnote 17}

• In the NWT, the contracting of local monitoring and evaluation expertise has resulted in high quality, consistent reporting and tracking that can be used by the College for decision making. The expert was first hired in 2012 and has continued, including an organized and structured handover to new consultants in 2018-19. This also permitted NWT to conduct their own evaluation of the program (in 2016). This provides the colleges with a more focused and tailor made evaluation than can be conducted when looking at the Program as whole.

The previous NABEP evaluation in 2016 identified lessons learned. A review of these lessons indicate that they were not fully incorporated by CanNor and remain relevant specifically, "Changes should be made to the program's performance measurement framework to include outcomes that are achievable during the program term and more focused on literacy and essential skills acquisition".

The following were the best practices, lessons learned and areas for improvement identified by interviewees. Some are considered general to the program, while others are identified as regionally specific.

Recommendations for CanNor

Recommendations for Delivery Partners

Recommendations for CanNor

Recommendations for Delivery Partners

Recommendations for CanNor

Recommendations for CanNor

Relevance

Performance – Effectiveness

Efficiency

Lessons Learned

Introduction

The case study will involve the three territories and as such the evaluation is limited to one topic given time and resources available. It is expected that 2-3 interviews will be conducted per territory; therefore, the case study represents 6-9 interviews, plus document review.

After proposing two case study topics to the Project Authority and Internal Advisory Committee, the decision was made to conduct a case study on curriculum development.

Rationale and scope

The rationale for selection is based on the program having a history of collaboration with the colleges over the last ten years, but needs in each territory are different, from basic education needs to other types of education and training for job readiness. The curriculum development process and content will allow the evaluation to look deeply on how the program is addressing needs and is it doing so in an appropriate manner (participative, culturally appropriate, innovative, etc.), and how partnerships and relationship play a role in shaping curriculum, for example, with industry.

Methodology

The protocol for conduct of the Case Study will be to:

• hold an initial meeting with program officials to collect preliminary evidence and identify interviewees;
• review documents and files;
• conduct interviews from a high of 9 to a low of 6;
• perform analysis that will assemble the data by territory, by evaluation issues and their indicators and develop findings; and
• document the Case Study.

Case Study Interview Guide