Re-blogging…review of LBS evaluation report by Christine Pinsent-Johnson

The LBS evaluation report and distinction between design and implementation problems

by ChristinePJ

On April 12, Ontario’s Ministry of Advanced Education and Skills Development (MAESD) released a report detailing the findings of a program evaluation completed November 2016. The report is here, and here is an accompanying executive summary.

The findings are hard-hitting. Those in the field will finally see their concerns acknowledged, and meticulously documented in a ministry sponsored effort. Cathexis, the evaluation consultants, must be commended for producing such a transparent and accessible report, filled with powerful findings and useful data. This is a compelling wake-up call and, frankly, a revealing case study of the ineffective and damaging use of a high-stakes pay-for- performance management system—although that was not the explicit message, it’s pretty hard to ignore.

More amazing, is the tone and voice conveyed in the report. It’s not a distant, authoritative voice. The authors do not gloss over people’s concerns or attempt to reformulate those concerns using abstract categories or heavy-handed theoretical analysis. The case studies at the end of the report, particularly the one of an Indigenous literacy program, are astounding. I read example after example of disregard, managerial arrogance and the over-riding message that compliance to a faulty system must be upheld no matter what the cost to people and programs.

The report was released four months after a finalized version was completed, soon after a representative group from support organizations lobbied those in high-level leadership roles within the ministry. The release sends the message that the ministry has finally decided to respond to concerns that began to be documented soon after key elements of the pay-for-performance model were introduced in 2012.  Accompanying the report and its executive summary are memos from the deputy minister and assistant deputy minister, stating that the ministry is ready to work with the field to address the report’s recommendations.

However, my worry is that these conversations could be curtailed, leading to some short-term solutions that fail to get at the under-pinning problems. My concern stems from reading a prominent statement appears in the introduction of the executive summary:

The LBS program is providing a vital, valued, and effective service to Ontarians. Its key components—the OALCF and the Performance Management Framework (PMF)—are well designed. However, serious problems have arisen in the implementation of these components.

It’s a divisive and troubling statement.  I immediately thought, ‘Here we go again; shift all blame for the problems on to the field.’ Who else implements what policy analysts and others within the ministry have designed so “well”? Only after reading the report did I feel a bit better. There is no indication of blame in the report

I’m trying to figure out where the statement came from and why it was made. The mandate of the evaluation was to examine implementation and not design. So why include a prominent, comprehensive and evaluative statement about design?  Despite the mandate, a couple of design issues are directly examined and one is the focus of a recommendation. In addition, design related problems are alluded to, although not fully explored, throughout the report, demonstrating it’s impossible to separate implementation from design. It’s unclear if the statement represents the ministry’s perspective—and was perhaps a suggested inclusion—or if this is the perspective of the evaluators. I have requested some context and possible references or sources from the ministry.

For those of you have followed the blog, my concern and recent projects with LBS partners, have been aimed squarely at design issues, and the misuse of international literacy testing levels, skills criteria and its information-processing pedagogy in the LBS performance management system. The design-implementation divide could be used to dismiss this work and limit discussions going forward. I see this happening already in a few of the recommendations. While most of the report’s recommendations are a sound response to identified problems, those related to the performance management framework, including its testing mechanisms need to be reconsidered.

Design is a fundamental problem that has a direct impact on implementation. For the most part, the field seems to be implementing exactly what has been designed. The report didn’t suggest that programs have gone rogue, making up their own tests (Milestones and Culminating Tasks) or creating their own learner satisfaction surveys (two ideas worth thinking about in future discussions) or ignoring mandatory performance target reporting (Service Quality Standards or SQS). Even on-the-ground confusions and mixed messages are not simply an implementation problem, since the dispersed organizational structure within MAESD and the multiple roles of 130 field consultants (Employment Training Consultants or ETCs) who have responsibility for LBS and other Employment Ontario programs contribute directly to the confusion and poor communication. Overall, it seems, the field struggles to implement what has been designed and deal with how the ministry is designed.

The field struggles to implement what has been designed and deal with how the ministry is designed

What has been designed is a high-stakes PMF that ties program performance directly to funding using a series of performance measurement mechanisms, including three high-stakes tests (Milestones, Culminating Tasks and the still-to-be-determined Learner Gains tool), a not so useful learner satisfaction survey and a listing of suitability criteria that doesn’t reflect the reality of many programs. The mechanisms have been designed either by the ministry or by consultants hired to do the work.

The evaluators articulate the field’s issues with two of the test systems—Milestones and Culminating Tasks—and recommend that their “merit” be reviewed. This is encouraging. However, they don’t make a direct connection between the problems and test design, even though a recent AlphaPlus study on the Milestones came to this conclusion. They do however recommend that the suitability mechanism be redesigned, an indication that the evaluators are willing to extend the mandate of the evaluation past implementation and into design issues in some circumstances but not others. Overall, there is also some confusion in the way that the OALCF is examined in the report, which I will look at in a future post.

According to the memo from the assistant deputy minister, the ministry will be open to “ensuring the design and management of the LBS program” so that it “supports all learners and all service providers” and acknowledges that not all tools are “available and/or appropriate for all learners and/or service providers.” To facilitate this discussion it will be important to fully understand inherent design issues, including the use of a pay-for performance PMF in a under-resourced program that sees an incredibly diverse range of adult learners who attend a range of programs for anywhere from a few weeks to a few years. There are nearly no homogeneous elements in LBS such as time spent in the program, age and background of learners, funding per learner, class/learning group sizes, educator training, and curricula. But the minisitry insists on the use of the same measures and targets for all.

How the ministry is designed also has direct impacts on programs and even the design of the mechanisms used in the PMF. MAESD’s dispersed and siloed organizational structure parses responsibility for LBS across several organizational units with no apparent and sustained connection between those units, and, more importantly, no sustained and meaningful connection with the field. Since LBS was integrated into Employment Ontario, I’ve seen a constantly revolving rotation of new names in various positions, and I’m never quite sure who I should even contact about a particular question. Directly related to the siloed structure is a preference for public administration and managerial expertise over content expertise among policy analysts and the majority of ETCs.

This has serious consequences. Policy analysts are in charge of developing a complex high-stakes assessment system without any assessment, curriculum, educational, literacy development and adult learning expertise. This then leads to an over-reliance on a small pool of consultants, promoting particular products and appraoches, and a limited ability to fully evaluate the relevance, rigour and usefulness of their products. The ETCs, in turn, must oversee an educational program without any knowledge of education and learning.

Then, the siloed and parsed structure within the ministry must somehow connect with a rather complex structure within LBS that includes regional networks, stream organizations and sector organizations. Of course, upending the organizational structure of Employment Ontario is not likely on the table, but understanding the multiple design challenges related to both MAESD’s internal organizational structures and the LBS structure are important when discussing possible responses and solutions moving forward

Let’s have a truly collaborative discussion and put all problems on the table, whether framed as design or implementation issues, in order to think about and better articulate more comprehensive and lasting solutions.

Posted in Adult Literacy and Basic Skills Policy, Literacy and Essential Skills, Provincial/Territorial Governments and Literacy | Leave a comment

Evaluation of Ontario’s Literacy and Basic Skills (LBS) Program released

The Ontario Ministry of Advanced Education and Skills Development (AESD) released today the report of the evaluation of its Literacy and Basic Skills (LBS) Program prepared by Cathexis Consulting Inc.*

The key message of the evaluation is described as:

The LBS program is providing a vital, valued, and effective service to Ontarians. Its key components—the OALCF and the Performance Management Framework (PMF)—are well designed. However, serious problems have arisen in the implementation of these components. These problems stem from fragmented leadership, poor relations between the Ministry and the field, threats to sustainability (including inadequate funding) and a lack of a clear vision for whom the program is intended to serve. The Ministry and the field must work collaboratively to rebuild cooperative relations and clarify LBS’s mission, so that the program can address Ontario’s literacy needs with greater efficacy, efficiency, and accountability.

I’ve attached both the Executive Summary (Cathexis – LBS Executive Summary – 2016) and the full report (Cathexis – LBS Evaluation Report – 2016). Take a look at the last two pages of the Executive Summary for a description of the key recommendations. The full set of recommendations are in the final report beginning on page 165.

AESD’s posting of this report is a positive first step towards responding to the evaluation findings. The evaluation reinforces the value of LBS and the work of the practitioners and organizations across the province. For those outside Ontario, the evaluation provides an insight to how a large jurisdiction like Ontario organizes and manages its literacy delivery system.

You can access the full report and the Executive Summary at:

*Full disclosure – I was a member of the Cathexis Consulting team that performed the evaluation.

Posted in Literacy and Essential Skills, Provincial/Territorial Governments and Literacy | Tagged , , | 3 Comments

Budget 2017 – Skills and Training a Priority

Today the Minister of Finance tabled the 2017 federal budget ( While there is no mention of additional funds for adult literacy, the budget contains a number of items for those interested in adult learning and literacy as well as workplace training, workforce development and skills training.

I must say that this budget is chock full of social policy initiatives, the like I’ve not seen in 10 years. I was invited to the budget lockup with over 100 stakeholders. From the conversations around me, people seemed to like what they saw, felt they had been heard, but, of course, thought more could be done on their issue. The last chapter of the budget is a gender-based analysis of the budget measures – a welcome innovation. The budget contains so many placeholders for positive and progressive action.

Canada Job Fund replaced

The government is proposing to create Workforce Development Agreements. These agreements, to be negotiated with the provinces and territories, will consolidate into one agreement the Canada Job Fund Agreements (CJFA), the Labour Market Agreements for Persons with Disabilities (LMAPD) and the Targeted Initiative for Older Workers (TIOW). The idea is to give the provinces and territories greater flexibility and make everything simpler. $900 million will fund these new agreements over the next 6 years starting in 2017-18.

The budget does not indicate the final total amount available for the new Workforce Development Agreements. While the budget indicates that the $900 million is additional funds, the TIOW was scheduled to end March 31, 2017 and the CJF on March 31, 2020. Below is my best ‘guesstimate’ of how much money will be available for the Workforce Development Agreements.

$ millions

2017-18 2018-19 2019-20 2020-21 2021-22 2022-23
LMAPD 222 222 222 222 222 222
CJF 550 550 550
Budget 2017 200 300 400 550 625 Not provided
TOTAL 972 1072 1172 772 847 222

The new agreements will allow the provinces to provide labour market programming as they see fit. Funding like the Canada Job Grant could continue but there will be no mandatory targets according to officials. If the provinces and territories see value in a program like the TIOW, I’m certain they’ll advocate for that funding to continue along with the funds original budgeted for the Canada Job Fund.

As you know, I have been highly critical of the Canada Job Fund. It has resulted in fewer vulnerable people being served and became a vehicle for employers to pay for training their existing workforce. I’m optimistic that these new Workforce Development Agreements will better meet the needs of the unemployed and the low skilled, but of course, the devil is in the detail.

Labour Market Development Agreements

The government indicated its intent to undertake a serious reform of the Labour Market Development Agreements (LMDAs) following a consultation last year. The budget however does not give details about the nature of the reform. The budget proposes an additional $1.8 billion over the next six years. The LMDAs provide support to those who are EI eligible. As the number of EI eligible is far less than the total of unemployed, the Workforce Development Agreements are designed to fill that gap.

New Labour Market Stakeholder Organization

A new organization will be created to:

  • Identify the skills sought and required by Canadian workers
  • Explore new and innovative approaches to skills development
  • Share information and analysis to help inform future skills investment and programming

The organization will work in partnership with “willing” provinces and territories, the private sector, educational institutions and not-for-profit organizations. $225 million over four years will kick-start the organization with $75 million after that.

This is welcome news. After the demise of the Canadian Labour and Business Centre which did much of this work (and where I worked), the opportunity for stakeholder involvement in skills development and training was lacking. Having the labour market partners at the table will result in a better system. I encourage the government to ensure all stakeholders are there including labour unions, adult learning and literacy organizations, and indigenous organizations. And of course, I would look to this organization to take a leadership role on literacy and adult learning.

Support for Adult Learning and Training

A number of measures support adult learning and training:

  • The Northern Adult Basic Education program (NABE) has been extended for the next three years with $14.7 million.
  • Amendments would allow EI recipients to pursue self-funded training and maintain their EI status (today, you must be actively looking for work to maintain your benefits). ($132.4 million over four years, and $37.9 for each year thereafter).
  • Today, adults who take occupational skills courses below the post-secondary level (e.g. second language training, basic literacy and numeracy) at a college or university cannot claim the Tuition Tax Credit while those who go to a non-post-secondary institution can. The budget proposes to allow everyone to claim the credit regardless of where they take their programming.
  • A three-year pilot project will test new approaches to make it easier for adult learners to qualify for Canada Student Loans and Grants ($287.2 million over 3 years starting in 2018-19).
  • Support is also being provided for Pathways to Education Canada. This longstanding innovative program works in low income communities to help youth graduate from high school.

Support for Skills Development and Training for Indigenous Peoples

The Aboriginal Skills and Employment Training Strategy (ASETS) is a parallel program to the LMDAs, managed by indigenous organizations. The program will receive $50 million in the coming year while work is underway to renew and improve the program.

Other Items of Interest

  • The budget has funds set aside to improve the recognition of foreign credentials.
  • Work continues to provide high-speed internet for all Canadians no matter where they live. This is critical for ensuring access to online and distance learning in northern, remote, and rural communities.
  • Commitments are made for early learning and childcare which will help efforts to build strong literacy skills in the early years.
Posted in Adult Literacy and Basic Skills Policy, Canada Job Fund, Canada Job Grant, Federal Budget, Federal Government and Literacy, Labour Market Agreements, Literacy and Essential Skills, Skills | 2 Comments

Broken Link…Matchup report

The link to the full report mentioned in my last post was broken. I’ve fixed the link on the post. Here is the new link: canada-west-matchup-a-case-for-pan-canadian-competency-based-frameworks-feb-2017

Posted in Adult Literacy and Basic Skills Policy, Literacy and Essential Skills, Skills | Leave a comment

Matchup: A case for pan-Canadian competency frameworks | Canada West Foundation

Janet Lane who many of you will remember from her time as Executive Director of Literacy Alberta, has co-authored a report advocating for a competency based approach to skills shortages. The report contends that the skills mis-match experienced across the country is due to:

“01  Formal education and apprenticeships do not teach all, or even the right, skills and competencies to the right levels needed by employers.

02  Unrecognized skills: Many people have skills and competencies that their official credentials do not address. They also may not be able to articulate the skills that go beyond official credentials that they possess, or their value. Therefore, unsurprisingly, employers may not be aware of the varied and specific skills that people have.

03  Employers are not sure what skills and competencies they need, or are unable to articulate what they are.

04  Foreign credentials, which are otherwise adequate in terms of skills and competencies, are not recognized.

One ambitious solution – a competency-based, pan-Canadian qualifications framework – would help to eliminate the mismatch problem by addressing all of these issues.”

You can read the Executive Summary here: Matchup: A case for pan-Canadian competency frameworks | Canada West Foundation

I’ve also uploaded the full report: canada-west-matchup-a-case-for-pan-canadian-competency-based-frameworks-feb-2017

I’ve always liked the idea of hiring based on what people can do rather than on a credential earned years ago. Such an approach would definitely improve outcomes for those whose skills have been developed over time through a combination of education, training and on-the-job experience. How one moves to such a system without it becoming very bureaucratic, i.e. requiring testing or other proxies to determine skill competencies, is one consideration.

Let me know what you think…

Posted in Adult Literacy and Basic Skills Policy, Literacy and Essential Skills, Skills | Tagged , , | 7 Comments

Citizenship, social cohesion and literacy

I ran across an article from some time ago making the case for a link between literacy levels and active citizenship. Given some of the conditions being faced in the world today, I thought it would be helpful to share some of the findings.

“Active citizenship and non-work related aspects of PIAAC” was published in Elm, European Lifelong Learning Magazine, on March 28, 2014 (

While much of the focus placed on Programme for the International Assessment of Adult Competencies (PIAAC) related to information about labour market and employment outcomes, this article outlines the impacts of lifelong long learning. First, it reminds us of what PIAAC said about citizenship and democracy:

Trust is the bedrock of democracy. Without trust in others and in the rule of law, all relationships, whether business, political or social, function less efficiently. The foundations of trust are established on three complementary levels: trust as an individual trait, trust as a relationship, and trust as a cultural rule (Sztompka, 1999). For an individual, certain skills may lead to trust in others. For example, key information-processing skills may enable people to understand better the motives and aspirations of others and the conditions under which these may be shown. Skills may also enable people to forge trust by fostering lasting relationships with the aim of accomplishing mutually rewarding outcomes” (OECD; 2013, p. 237 f)

PIAAC found that those with lower proficiency in literacy are less likely to trust others. In addition, those with lower proficiency levels say they have a low level of ‘political efficacy’ or the belief that they can make a difference.

Second, there appears to be a relationship between PIAAC skills and social cohesion. The authors state, “proficiency in skills may influence social (in)equality, social in- or exclusion and social cohesion, through the role of building trust in others who are or who are not like them (such as highly- or low-skilled person). “

Third, higher literacy skills are associated with civic engagement such as volunteering. The authors reference other studies that relate an increase in racial tolerance and a greater likelihood of voting with participation in adult education.

The authors conclude, “Participating in learning activities and increasing skills can provide a stable time framework, a community, a chance for re-orientation, a safe place, a new challenge, social recognition, and end up being an important tool for empowerment. Especially in times of crisis, literacy skills are necessary for tackling economic and societal challenges.”

This article makes the case for more, not less investment in adult education and lifelong learning, not just for economic or labour market objectives but to help foster a strong sense of social cohesion, community engagement, and active citizenship. Certainly a theme worth raising with public officials.

Posted in Adult Literacy and Basic Skills Policy, Federal Government and Literacy, International Literacy Surveys, Literacy and Essential Skills, PIAAC, Provincial/Territorial Governments and Literacy | Leave a comment

More PIAAC research documents

Thanks to Suzanne Smythe, Simon Fraser University and Gabriela Lopez, RESDAC for bringing two more PIAAC related research papers to my attention.

RESDAC, COFA, Statistics Canada and Employment Ontario sponsored further research into the situation of Ontario’s francophones.

Posted in Federal Government and Literacy, International Literacy Surveys, Literacy and Essential Skills, PIAAC, Provincial/Territorial Governments and Literacy | Leave a comment