“Learning analytics”: a red herring?

A key message from my recent Kirkpatrick program was to start with the “top of the mountain”.  In this metaphor the top, the peak, the target to reach, is the organizational vision.  Strategic learning programs are, therefore, helping the org reach this vision and should be evaluated as such.

My reflection during the program was that this, of course, is common sense.  We should be working to support the organizational goals.  The challenge then becomes prioritizing multiple needs – so only by forecasting potential impact up front can prioritization be done correctly.   And this is one of the areas where there is misconception with KirkP – it should be about starting with the end in mind and working backwards (not just dealing with level 1 in a standard way and then carrying on from there).

In terms of evaluation of success, LEO have recently discussed the role of learning analytics (LA).  Now, like a lot of things in L&D, I would say the problem with LA is that it has meant multiple things to different people.  One of the earliest examples I saw, sold as LA, was Starfish Solutions (SS) who had a clear solution and goal – use existing LMS/VLE data to improve student retention.  SS makes perfect sense for organizations where educational outcomes and student retention are the organization’s objectives.  I liked SS’s solution (in part discussed with them at the BBWorld Conference back in ’09) but it also faced the challenge that, for many university courses, there was/is less need for ‘big’ data solutions – lecturers know their students in traditional models.  It only made real sense when talking about large scale education – the problem then, again, is that ‘large scale’ means multiple things to different people 😉

The LEO article does a good job at articulating the problems I have always had with L&D impact – especially how to assess when there are so many other uncontrolled variables.  As mentioned in my previous post on the KirkP Cert, this was my main challenge I wanted clarity on from the course.  The recommended KirkP approach of identifying multiple ‘indicators’ (suggesting behaviors are on track for the desired result[s]) that can show a correlation – a key learning point for me.  In this model, therefore, we are building a ‘chain of evidence’ akin to a court of law – “data, information and testimonies at each of the four levels that, when presented in sequence, demonstrate the value obtained from a business partnership initiative”.

What I really liked about this is the clarity of the evidence/steps up the ladder/mountain, from bottom to top:

  1. Individual Outcomes
  2. Team Outcomes
  3. Departmental/Divisional/Unit Outcomes
  4. Customer/Client Response
  5. Organizational Outcomes
  6. Customer Satisfaction
  7. Market/Industry Response

It is this breakdown, of benefit chain, that I will likely adopt in my existing project planning documents.

Let L&D then be clear, as the KirkP course made clear: stop trying to tackle issues through limited/impossible direct causation but instead correlations.  I would say this is a much better approach than simply seeing “measuring learning impact as a waste of time and money” as the LEO article mentions many people argue.

Therefore, I would argue, let us (i.e. L&D professionals) not worry about learning analytics but instead organizational analytics (i.e. goals and metrics) that can be seen as trending over time and aim to see where our investments have an impact.   As recommended in the KirkP programme, do not reinvent the wheel, borrow metrics from elsewhere as they will already be used by stakeholders and those same stakeholders (should) understand the logic.  This should then allow us to, as I’ve hinted at before, not worry about ROI but instead (as recommended by KirkP) Return on Expectations.

So what do I think of KirkP following the course and certification?  Well I’d have to agree with the LEO suggestions:

  1. It’s better than doing nothing… which is what most organisations are doing.

  2. Think about what level of evidence will be good enough for you. As the Kirkpatricks have pointed out, a chain of evidence that would not stand up to scientific rigour may be enough to convict someone in a court of law. If a level of evidence is good enough for a court then it’s probably good enough to convince your board to invest in L&D.

  3. Good enough develops into great.

Apprenticeships: more following the October guidance

Having continued to reflect since my last post on this topic as well as taking in the October guidance (both through reading and a couple of related events).  It’s starting to look a lot clearer now…my current view on the three main options for employers:

Ignore it all together…

A lot of companies will continue to ignore apprenticeships as the 20% off-the-job and new division between providers and assessment organisations will not be as efficient as what can be done via other development approaches.

It’s not just about the levy – company’s existing training will have some level of value and quality.  I’ve always felt workplace learning, FE and HE need to be much more joined up and its good that the levy is starting to make people look wider than their existing silo, for example the OU working with people consultants from KPMG for a wider solution.

There remains though a lot of snobbery in learning, including:

  • from apprenticeship providers about the quality of non-accredited workplace learning
  • about the lack of skills in HE from FE and employers
  • the HE snobbery around degrees being of most value.

That the levy seems to be breaking down at least the last of these, via degree apprenticeships, and getting some cross-sector conversations going can only be a good thing.  However, as mentioned in this article, if the model is to be employer led why force funding for apprenticeships only?  And will degree apprenticeships get very far if even the BBC refers to them as ‘degree apprenticeships‘ as if a non-recognized qualification.

…or sub-contract…

There’s a logic in presuming subcontracting will be the most popular route with companies who have existing L&D teams but little/no experience of apprenticeships.

You would expect few will have met the short Skills Funding Agency deadlines at this enrollment window and even fewer will attempt the full employer-provider model this time around.  The October guidance suggests sub-contracting is a valuable (upto £500k per annum) way for L&D teams to save their companies from some of the levy ‘hit’ whilst putting existing learning into more formal structures. Indeed its also become clearer in October that the SFA sees investment in management information systems as essential for employer-providers.  This and other logistics may be a big ask for all but the biggest employers and you suspect sub-contracting well allow many employers to deliver the training they deem appropriate but leverage a provider’s economies of scale for systems, standards management, Ofsted requirements, etc.

…or wait and see.

The deadlines of late November for registrations were challenging (when SFA employer engagement events were fully booked in the run up) so the ‘big bang’ of the levy introduction (the event I went to said 500 companies had attended/booked nationwide) may well become a whimper for a year or two.

That the SFA needed to send the below note out on the day of the registration deadlines shows that there’s interest – even if organisations have failed to be totally clear on who is responsible for what in this new world!

The SFA has noticed that some organisations have submitted multiple PQQs despite clear guidance.

Organisations are reminded that only one PQQ route must be submitted. Please check that this is the case.

 

Time for a rethink on ‘support’ services?

Starting my London-based career in an ‘Information Services’ team has led to me always having quite an interest in the semantics of support departments. Information has become a hugely overused term since then, closely associated with the ‘Knowledge Economy’, as business and academia have worried about the growth of technology and overload of web content in the last 20 years.  However, whilst information teams have often dwindled in the face of ‘free’ material on the open web other support services continue along.

At a CILIP event about 5 years back there was an agitated former Institute of Information Scientists member who was furious at perceived continued dumbing down, in other words, a focus on libraries rather than information science. The challenge, of course, is that ‘information’ is a term largely usurped by technology, either as IT or ICT. In this respect the BCS and other groups have usurped the second I in CILIP and there were valid opinions expressed that the CILIP renaming debacle could of done worse than to embrace the old ‘Library Association’ moniker. However, the risk with this would be to alienate members, such as myself, who have long moved on from physical spaces whilst still using an information orientated mindset (I tend to avoid ‘skills’ here as I fear that might be over-egging my pudding!).

So if “information” has become synonymous with technological solutions and support departments what for those with an information mindset?  In many cases they will be found embedded within another traditional department such as research, HR, marketing or training/learning. They may (like me) or may not have formal academic credentials in these areas but do have the option of engaging with professional bodies and potentially seeking professional status such as MCILIP, CMALT, etc.

Of course the challenge is that in ‘knowledge’ (and many other) roles ICT solutions are essential, and I include the C to recognize the role of communication and collaboration tools.

So what of all these support teams? Well, whilst ICT and the currently vogue ESN have tried to break silos they are often exist.  This often is not helped by the splits to C-Suite reporting across various groups including COOs, Heads of People and (of course) the CIO.

What I’m going to suggest today though is that disciplinary focus doesn’t help. Instead let’s pick the best elements to create a single support structure. But what to call it? Well how much your (and I’m largely talking office based support here) support make up of your workforce will impact.

However let’s adopt “Productivity and Performance”. In this model, Ullrich-esque HRBPs can become performance consultants and help identify where things need to improve and have full scope of measures (finance and other data) versus solutions (digital solutions for marketing and learning), etc.

Obviously organisations will vary but its starting to feel like claiming the ‘productivity’ name is a solution – as recognised by Microsoft, Apple and others who recognise software by that name. Indeed if one looks at the latest top 100 tools for learning many are not ‘learning’ specific but productivity/office focused. Many on this list would appear for a lawyer, finance, marketing and other pros.  Let’s recognize the value in the tech and bring together the support staff with different mindsets, strengths and expertise.

Apprenticeship changes: reflections thus far

I’ve drafted a few brief reflective pieces for the site of late – only for referendums, snippets of policy and other articles to make me pause and reflect again.  This is an attempt to capture some of the excellence points being made about apprenticeships as the future for UK (well English) skills development (as well as linking back to old articles on the topics).  I’m well aware that elements will be made out-of-date between typing and publication!

In summary, it is hard not to feel we are in the middle of a seismic shift, akin to the Blair years’ promise of 50% Higher Ed attendance and the earlier end of polytechnics.

Whilst partly being driven by (and carried along in the tsunami of) Brexit news – the potential implications and the importance of the changes to Apprenticeships are also being drowned out by the huge stories in Europe, Syria, America and elsewhere.  This is a shame as whilst the potential for new Grammar Schools has managed to grab headlines, the apprentice changes have struggled to.  This is particularly striking considering the levy policy largely amounts to Government intervention in workplace people development and, therefore, a potentially more important change.  This is especially so for a Conservative government that continues to talk of free trade opportunities whilst having changed minimum wages, pension laws and other cost of business regulations.

Reflections on the emergent policy

The Apprenticeship Levy and related policies (or at least information on them) have been delayed at virtually every stage.  This is not helped, as anyone who has tried to work through the process will find out, that the related gov.uk pages have no real help, support contacts or feedback (beyond formal consultations).  I know it is better than it used to be, as they aim to move to the one site, but there are still plenty of errors on existing sites (see image from an example link on the Skills Funding Agency registration process).

Ultimately, there seems to be a shift from the government or individuals deciding ‘whose education is it anyway?’ to what business and employers determine is appropriate.  However this is within a government structure of shifting the financial cost to business in return.  Yet discussion continues over if some form of central linking of growth strategy and apprenticeships is needed, independent of the market forces of employer led levy policy.

screen-shot-2016-10-13-at-21-04-12

Dead-links from the Skills Funding Agency website

Reflections on the impact on Higher Ed

As this opinion piece from the OU on CIPD.co.uk and a previous post from me suggested, the levy offers L&D departments a huge opportunity.

The challenge may be more for universities.  With a clear challenge on how to keep relevant when charging huge fees.  To massively simplify – why would anyone want to study for 3 years for £40,000+ of debt when you could get an equivalent qualification whilst working?  Whilst the American Dream has been challenged as unattainable (as you can no longer work your way through college – but the universities and other interested parties dont want you to know that), apprenticeships are effectively challenging the ‘traditional’ UK model that started to take shape pre-Blair and was accelerated by his administrations.  Universities are going to have to become more responsive and imaginative in the design of degree-level apprenticeships.

If we accept there will be a (re)balance in the future between apprenticeships and degrees, then we also have to consider the nature of the labour market.  HE fees have discouraged mobility (encouraging people to stay and study at home) and social mobility in applications to ‘top end’ unis from lower income families.  Yet similar mobility challenges are being raised with apprenticeship changes.

Whilst a university education allows mobility, albeit I am an example of someone who chose a location for my undergraduate university (in part) as it was relatively close to home and cheap, there is the question of social value versus gaining efficiency through competition.  Of course, UK HE has some approaches that make entry to the market difficult (and better use of economies of scale, perhaps through mergers, could be made).  JISC could, for example, support degree apprenticeships through technology in a way that would be difficult for employer providers.

Fundamentally, there is also the question over why universities have failed to adopt wider competency models.  The US focus on competency based education in HE has shifted the model somewhat towards a more broken down skill profile assessment, akin in some ways to our apprenticeships.  Inevitably UK HE will have to move away from the vagueness of 1st, 2nd and 3rd class degrees.  Of course a lot of this is not new, I even presented on related issues at a conference in 2010.  There were also sessions on competency based learning beyond the standard curriculum at the same conference.  You have to presume that changes to traditional degree models will be accelerated by the ‘competition’ with new apprenticeships.

And to the future…

The arguments over graduate skills (or lack there of) will continue.  The shift to the Levy means employers paying to the levy have three real options it seems to me:

  1. do nothing – accept the loss as a tax
  2. work with vendors – accept apprenticeships as your career development pathways but also accept the “loss” of the levy money
  3. employer provider route – try and claim back the money whilst doing apprenticeship programmes yourself.  Comes with the highest risk (including dealing with Ofsted) but potentially highest rewards.  Risk is also associated with continuing lack of clarity, not least over assessment standards.

Another area where questions remain is if the target of three million apprenticeships will bring down quality.

Option 3: Employer providers

A key feature to the planned changes with the apprenticeship policy, throughout the planning, has been the promise of a shift to them becoming (more) ’employer led’.  There will be less of an education-sector focus in their organisation, especially in the change to allow people to be an apprentice at a lower or equal level to past qualifications (a big change).

The summer’s raft of announcements, typically made when I was on holiday, seemed to suggest (yet further) that becoming an ’employer-provider’ is a Good Thing for employers.  Questions remain though and I was personally turned down for the pilot of the apprenticeship funding website – again you could probably argue that giving some companies the early access and others not is, again, interventionist and potentially a help to some companies over others in competitive environments.  Yet confidence with the levy seems to only be possible if you think the current government will be in power for a while at least, opposition being in place from across the benches of the commons albeit sometimes ignoring the levy altogether.  Yet the government has continued to tinker with the planned new structures in the face of some of this opposition.

The presumption seems to be that larger employers, who already have L&D resources, can consider the provider route.  This makes sense, not least as apprenticeships focus on learning across the 70/20/10 spectrum.  However, the enforcement of L&D teams working within rules they might not have worked on themselves (depending on if they worked on the standard or not) will be a challenge to many – not least when there has been such a focus on informal elements and recognising the value of learning achieved away from credentialing (this also being part of the challenge for HE). There have also been lots of questions over supporting smaller companies in supply chains (aka ‘extended enterprise’) and the need to apply as a full provider to train other companies’ staff may be too much of a logistical burden for many organisations.  However, again, there is potential here for L&D departments to really transform their position within their organisations and their relation to partner companies.

Indeed the options for what levy money can be used for, by employer providers, are quite comprehensive:

“Employer-providers will be able to use funds in their digital account to pay for the following: • training to achieve the apprenticeship, which could include qualifications, elearning (as part of a broader training package), vendor qualifications • registration, assessment, materials and examination • administration related to the delivery of the apprenticeships. • accommodation for residential trips if necessary for all apprentices to achieve • costs for use of premises where these are used for the apprenticeship • wages and associated costs (such as pension and National Insurance contributions) for employees directly involved in the delivery of the apprenticeship.”

and it is difficult to see quite how the government will regulate and audit against ‘misuse’, even if their attempts are mostly logical:

“To claim eligible costs, employer-providers are required to input the price of these costs on the digital apprenticeship service and through the Individualised Learner Record. As detailed below, employers must retain evidence of these costs. This evidence may be requested as part of the SFA’s audit and assurance checks.

They won’t be able to use it for: • costs that are the employer’s responsibility, for example health and safety requirements, wages, travel, commercial choices (e.g. CSCS cards) • wages for line managers or other colleagues supporting the apprentice 16 • wages of the apprentice • profit or employee bonuses • capital purchases • more than one apprenticeship at a time for an individual apprentice • re-taking qualifications or assessment where no additional learning takes place • apprentice recruitment • anything that has received other government funding (for example European Social Fund)”.

The 20% rule

Historically the opposition to apprenticeships has often focused on the 20% rule:

“All apprenticeships must include 20% off the job training. It is up to you to decide at what point during the apprenticeship the training is best delivered (for example, one day a week throughout, 1 week out of every 5, a proportion at the beginning, middle and end). This will depend on what is best for your organisation and the apprentice. You need to ensure that all apprentices receive all of the training.”

To myself, as an L&D person, this sounds awful and in opposition to the wider potential of workplace learning and on-the-job ‘training’.  Indeed I chose to focus on this point when commenting on the Learning and Skills Group Forum a while back.  The noise seems to be that this can be more flexible than it has perhaps been in the past.

 

Overall, there remain many grey areas over how the FY17 model will work but the support being given to apprenticeships does seem to be hugely important and something for all L&D departments to consider in their strategies – even if to say the ‘brand’ of apprenticeships (one of the outstanding issues according to the CIPD) still isn’t strong enough to work for them.  Many useful links still haven’t made it into this reflective piece so I may well add to the comments.

Why I do not like learning awards

I have been asked a few times lately why I have never personally submitted anything to awards such as the e-Learning Awards.

There is first, of course, the question of if any of my work would be a valid entry, or at least good enough to warrant some of the judges’ time.  However, there are more fundamental issues with such awards in my opinion.

My primary complaint is that they still encourage a ‘solutioneering’, static, approach to learning.  If you can package something in a way that can be submitted for external review it seems, to me, to miss the point of being embedded in a wider learning/knowledge framework – never-mind being embedded in workplace practice.  Even if we take it as being a demonstration of quality content, as presented to an audience via a Learning Management System (LMS), it suggests a chunking of content into large components – rather than breaking things down into a way that allows people to pick and choose what is relevant to them.

Whilst I appreciate award shows offer an opportunity to celebrate success and foster team engagement/morale I am still to be sold on the benefits outweighing the sense of verging into Hollywood-esque hubris.