Evaluating my impact: preparing for Kirkpatrick certification

Ever since I started attending workplace learning related events, it has been clear there is an undercurrent in the industry of criticizing the Kirkpatrick model. There are the views it is too old, simplistic, outdated, etc.

Personally I try and avoid criticism for anything I am not in a position of authority on – for example I feel I can criticize politicians for lying, their interpersonal skills, etc. but not necessarily their skill at the practicalities of being an MP such as passing laws.

Whilst I have worked with different evaluation models I have opted to pursue Kirkpatrick certification (as mentioned previously) to really try and crack evaluation/impact for my team and my wider organization. Thus, I will be in a more authoritative position to consider the pros and cons of Kirkpatrick.

Unlike some L&D teams, I would say we are closely aligned to business objectives and do a reasonable job of challenging top down and bottom up requests for programs, performance improvement, etc. However, evaluating success in tackling the issues is tricky and, as I inherited an L&D policy specifically saying we will use the Kirkpatrick levels to evaluate, then we are following that route.

I’m hoping the two-day program will empower me around evaluation in general but specifically being able to make use of the appropriate models. Or it may well make me a more vocal opponent.

The pre-work includes some nice detail behind the levels. Indeed there are references to deal with some of those regular topics of criticism. However, whilst ‘the 70’ is referenced as covered in the ‘new’ Kirkpatrick model, the template forms still reference evaluation of “training”.

A particular challenge for myself in the past has been evaluating solutions’ real impact on the bottom line; in a world of multiple factors (KSME) how can we (L&D or a wider project team) claim an impact. Indeed I raised this at the Kineo/Boost evaluation session. The pre-read refers to using ‘indicators’ and this will be particularly interesting.

Expect a blog later in the month once I’ve been on and reflected upon the certification program!

Reflections on #LT17uk

I only had around 6 hours in the Learning Technologies exhibition this year (for those who’ve never been – that’s not a lot!) and it helped me stay focused – mostly just walking the hall and meeting people (I stuck to the plan and spent much less time in the free seminars).

Your experience of such shows is, obviously, influenced by current personal/workplace concerns – but here are some of my takeaways…


Listen to the fortune cookie!

  1. Authoring tools – is it now time to move away from Articulate (see my previous posts on the trial of 360 and problems with Rise for more)?
    1. The ramifications of the changes with Articulate have been identified as one of ten eLearning trends and this infographic actually does a fair job of summarizing the big topics that were evident at the wider LT show.
    2. There remain a vast number of options and its perhaps now the case that one has to look at multiple tools – splitting out software simulation, VR, presentations, etc?
  2. LMS or HR platform – after almost a year in role I feel in a better position to consider the pros and cons of the current setup and possible alternatives.  Stick or twist?
    1. Interesting to hear Harold Jarche’s views that whilst there is rebranding and changes to the companies involved there remains not “a whole lot of new” – I would agree.  VR/AR/MR was more prevalent but still lots of stalls with content, platforms, etc.  An in the platform world there remain the big players (like Cornerstone) and lighter touch options – as well as multiple Totara partners.
  3. L&D capability – I’ve got a Kirkpatrick certification course with DPG coming up (I’ll blog in more detail why as I appreciate that’s a controversial choice!) and it was interesting to consider where we’re strong and where we may have gaps beyond that topic.  Should we all do DPG’s new 70-20-10 programme?
    1. I wouldn’t really think about 70-20-10 approach as something to be workshopped/developed but it looks an interesting approach to rolling up performance consulting, modern design and evaluation all in one development package.  The challenge, I guess, is if this should be topics new for people or is really certifying existing knowledge and behaviors.  Also if this is of value over the existing 70-20-10 Forum, etc.?
  4. Future of learning and performance – I mention above about VR/AR/MR but there was still a lot of content and blended learning talks.  What is the future?  How do we balance individual knowledge/skills growth with shorter term engagement/empowerment?  What should be the balance of individual career development, long term wider workforce planning and short term performance?
    1. There still seems to be a lot of push on concepts that have been around longer than I have.  However, it is difficult to challenge this considering I am more than aware that even my own organization is still very face-to-face orientated!
    2. There were more interesting bits though – including TTS and their performance support system (that offers some interesting possibilities for exposing knowledge out to the point of need – not buried in an LMS or Intranet).
  5. Future of Learning within organizations.  The Learning Tech show always involves some navel-gazing and there continues to be the feeling of a split between the “performance” element and more of the “corporate university” type model.  Are they mutually exclusive?
    1. To do my own bit of gazing, I’d say not.  An approach for, say, leadership development can incorporate formal learning and certifications.  Indeed there’s the option to revenue generate if you open your doors.  The apprenticeship levy will also influence how L&D teams tackle these issues and it was interesting to see the likes of bksb and Tribal on the exhibition floor.
    2. Curation remained a theme but it is one that remains a concern for me.  Kate Graham puts it on her blog “In L&D we trust”, however, I fear this may be wishful thinking considering what has happened to library/information teams.  This comes back to my own view on the need for merging/bleeding of skills between marketing, learning, knowledge and more.  Curation is of value but how this works in practice must add value and capture the needs of the organization.  Again, the TTS style approach may be a useful way to break from the “LMS first” dynamics of too many L&D teams.

Reflections on BETT2017

Have to say I enjoyed my day at BETT this year.  I made a couple of changes to my approach which helped:

  • limited session attendance
  • limited myself to one piece of A4 for notes – rather than streams of content on OneNote [yes, ironically a ‘detech’ move but it meant I was more focused]

Some particular thoughts…


As predicted in my preview post, ‘reality’ jumped out  as the big theme from a number of vendors and sessions (be it virtual, augmented or mixed).  The argument given by Avantis in one of the BETT Arena sessions being that this is something “fundamentally different” due to emotive immersion.

As well as a variety of headsets and software providers there were what, I guess, we can call “supportive” tech.  I would categorize in here things such as 360 degree cameras for creating content.  The Arena talk I attended also pointed a lot of content is available on YouTube – indeed the BBC have got involved recently.

Alas I didn’t get a chance to see a Hololens as hoped – but there was an “invitation only” ‘lens lounge and some talks (which I missed).  There were plenty of other examples including paragliding with Intel, Avantis (£2000 for 8 units) and more.  In terms of healthcare, it was interesting to see a few different stalls have similar 3d model of the heart VR content (presumably as it is in the biology curriculum).  The Arena talked of other possible scenarios from day trips to Mars to work that has been done to give school children and understanding of the refugee crisis through VR experiences of Syria.

Other devices

Whilst I didn’t see a Hololens there were opportunities to see a range of Surface devices (including Hubs) in action.  There seemed a lack of much innovation in the devices I saw though (away from Microsoft) and tablets/phones seemed underrepresented in face of Chromebooks and laptops.

Perhaps the hybrid nature of the dock-able Surface-like devices are the future?  Alternatively its a battle between that style and Chromebooks?  Indeed a talk I saw on Chromebooks stressed their key advantage over tablets for schools – a physical keyboard.  The presentation stressing demonstrable data of student improvement and they had some good tips for maintaining handwriting as well as typing skills.  I particularly liked their ban on immediate use of spellchecker correction: instead pupils have to retry the spelling, then check and write it down, then delete it and type it correctly.  This is effectively a hybrid approach to the old spelling books my school had when I was a nipper!


A lot of the show is now Microsoft – as the main sponsor they seem to be slowly drowning out a lot of the Apple and other players.  It was particularly interesting to see the focus on Office365 with a lot of people offering additional functionality – including different takes on LMS/VLE functionality.

An email received on the same day from the Digital Workforce Group (DWG) highlighted the growing importance of Office365:

Office 365 is reportedly Microsoft’s fastest growing commercial product ever...Up to 40% of DWG members are planning to introduce Office 365 in the next 18 monthsOffice 365 isn’t just SharePoint, and the digital workplace isn’t just the intranet.

Now those points are very DWG membership focused but it highlights an issue that is common with schools – Office365 has the potential to be the single platform that then combines with other things.

A talk I attended from Microsoft (“Enhance your classroom with the latest intelligent business applications across CRM and ERP form Microsoft”) stressed that digital transformation can be driven by Microsoft’s combining of CRM, ERP and productivity suites in the cloud (via the new Dynamics365).  Unfortunately the talk then just focused on how education can pick up licenses for Dynamics to use in the curriculum (such as for accountancy, HR, etc courses) rather than anything more strategic about the organization or the academic really enhancing strategy [as the presenter mentioned, D365 is breaking organizational silos so perhaps there are bigger cross-subject area opportunities that could have been talked about: as they said it offers “holistic” views of business].  Instead the message really was just that you would be enhancing your curriculum by training students in sought after skills [although of course technology changes rapidly] and to do that best is to join the Dynamics Academic Alliance (1436 members worldwide).

Of course technology is a complex picture, recently described well by the Fosway Group (looking at it from the eLearning market perspective):

“Providers now offer broader and broader solutions that include much more than e-learning alone. The impact of the growth in workplace, social and collaborative learning and focus on learner engagement mean that more and more buyers are looking for a complete digital learning experience.”

One outcome of this is that they will now look to research, review and compare “learning systems” – thankfully dropping the “m” word which, as we all know, has been a misnomer.  You can support, facilitate and scaffold learning – you can not manage (all of) it.

Google were also presenting on how their Classroom product combines with Apps, etc .and it will be interesting to see how this all continues to develop – devices in part driving software choices and vice-versa between the big ecosystems of Apple, Google and Microsoft.

Elsewhere, in relation to Microsoft, it was interesting to see Slomer who, among other things, are the first to offer SIMS on Azure.  Pottering past the Microsoft stall to see Delve and a few other pieces being highlighted showed quite how much they are doing but also how it is a moveable feast and almost feels like – if you are to go ‘all in’ with Microsoft you would have to be very systematic in saying what people should be using and how.  For example, as last year, there were sessions on running your classroom through OneNote and how Minecraft can be used – whilst you could easily run Microsoft platforms and tech but ignore such software altogether.

Where were the crowds?

Unusually there was only one part of the day (around 12ish) when I got stuck among a throng of people.  I’m presuming this may well be a sign of the challenges schools are facing with budgets.  Talking to a couple of stalls about it they clearly felt it was quiet for the Thursday and were hoping Friday would pick up and be its usual mass of people.

That said, perhaps such a big tradeshow (with everything from beanbags to servers) is just too much?


A talk from South Denmark University, in the Higher Ed section, showed that eAssessment remains a challenge.  They’ve developed what sounds a very comprehensive solution.  A monitoring application to check BYODs during exam situations – with screenshots taken regularly throughout the period, capturing of open applications and other functionality.  I liked the approach of not blocking certain sites but instead flagging them in the logs as possible issues – for example if a student access Dropbox you can then check the screenshot log that their essay hasn’t suddenly grown in length – which would indicate copy and paste of prepared content.  The responsibility for device management is with the students but other policies and considerations need to be place as well as clarity over how they should be setup within Blackboard – with papers still going through to SafeAssign for plagiarism checking.  The impact on pedagogy is clear though – it makes exams more like coursework in needing questions that test how the student thinks rather than just what they know (as that can be easily searched for).

Digiexam also looked like an innovative eAssessment platform.

Elsewhere it was interesting to see BKSB have a big stall considering the looming importance of apprenticeships and, perhaps, one solution for struggling schools is to cut teachers for more off-the-shelf functional skills and GCSE content?

(Cognitive) Health

The SEN section of the show is always intriguing, not least as it is an area I have limited experience in, and there seems to be a growth in looking at some of the issues from a ‘health’ perspective.  Indeed it felt like some of this was comparable to ‘wellness’ in the corporate world – what can we do to help people for whom the normal environment (Schools in BETT’s case or offices/workplace in employee programmes) is challenging.

Products/stalls of interest in this area included:

  • http://www.mycognition.com/ who do have a business focused product that “specializes in identifying the cognitive health of your entire workforce”.  This CQ being very interesting compared to the more traditional IQ and EQ.
  • There were, as in previous years, sensory booths for assisting with different issues.  Presumably an example of far too much cost for a single school and the kind of thing that the old buying powers of LEAs were helpful with and are, in part, replaced by the academy ‘chains’.
  • Atmybest.com – a positive psychology platform seemingly similar to Strengthscope.

Other stand out points

iTrent and other software that can be used in education and corporate environments were in attendance.

Open Campus and other tools combined learning elements with admin features, their corporate offer being to combine: “a constantly growing number of standard business applications such as corporate health programs, internal surveys, compliance training and more”.

There were of course plenty of other tools in place for different solutions, such as: video conferencing, publishing/ebooks, iPad equipment for video recording, Nuance Dragon was present and it was perhaps a little surprising to not see more on voice control and recognition and also some nice ideas like Modern Muse for inspiring the next generation.


Finally, another product to catch the eye were iStay backpacks – and I’ve in fact ordered one since the show!

Avoiding cynicism during tradeshow season

This week and next I’ll be doing my usual pilgrimages to the BETT and Learning Technologies exhibitions.  Some of my notes from previous years are on this site (BETT16; BETT14; LT16; LT15; LT14).

On reflecting about what I want to get out of these shows this year I realised the growing levels of cynicism in my previous reviews.

I then read Laura Overton’s mind shifts for 2017 piece and realized that it is perhaps as I’ve just now just been around a while!  Obviously I don’t hold Laura’s level of experience but her fourth point resonated:

I’ve been going since it started years ago and it is great to catch up with old friends. However it’s a time when cynicism can abound, particularly with those who have been around a bit.

After all, we’ve been talking about how the latest tech will change the world, about performance, the need to get managers on board, to communicate more for decades.

These are not new ideas but our cynicism and fixed mindset thinking kicks in when we believe we see no change.

Whilst the concept of disruption is all the rage right now in L&D, fundamentally we need to apply the idea of radically transforming the norm to our own thinking, especially our cynicism. A growth mindset will ask – why no change? Was it that the time wasn’t right? Is the time right now? What can I do to make a difference now?

One weakness I have is that I perhaps do not take enough positive feeling from the change and impact I’m having – instead frustrated by the slow pace (or lack of) clear change.  Rather than just thinking “yeah, heard this before” I’m keen, this year, to get more out of the events:

  1. Get a feel for what may be possible from BETT – 3D/VR/AR are of real interest to me at the time being as the practical skills in my workplace would lend themselves to these types of technologies – making virtual learning far more real whilst maintaining a safe learning environment (well VR would at least).  If Microsoft don’t have a Hololens, to try their take on AR, I’m going to be disappointed.
  2. More generally BETT should be a good opportunity to get a feel for what is happening in schools – as I’ve argued before: an understanding of this is essential for workplace learning pros who want to build suitable inductions and optimize future readiness.
  3. At Learning Tech I’m going to be more of a ‘guide’ as I’m taking my head of department to the show and she will be a ‘newbie’.  As we have a meeting in the morning we’re likely to be restricted to about half a day in the exhibition so I suspect it will be something of a whistle-stop tour – that should be good as it will stop me listening to as many of the free presentations.  I think last year, in-particular, a number of those sounded like broken records.  It should be a great opportunity to drive home some key messages and some of the things I keep banging on about…

and on that last point – Laura’s 6th point (“believe in yourself”) is another worth reflecting on.  This time last year I was looking for work and was increasingly frustrated – both with (only some it has to be said) agencies (due to lack of contact/followups/etc) and with conversations/interviews where I felt I could contribute to an organisation but the recruitment processes were frustrating or annoying.  Thankfully I’ve found an organisation where my skills and experience are a good fit – now just to avoid the cynicism and keep on driving on performance improvement and employee empowerment!

Further thoughts on Articulate360

Following on from my post on Rise

You are first prompted to install 360.  However, you quickly realize this is just the new installer/aggregated menu.  Once installed, this then prompts you to install or launch other tools – if you have the trial banner then “Preso” is shunted off the bottom.

Articulate360 menu with open, install and launch options

articulate/Articulate 360 menu

Installation of the local content was a bit tricky – I had my local IT team to tinker with my admin permissions to allow the Storyline and Replay installs whilst Studio 360 would not install due to a “conflicting version of Studio is currently installed”.  Fair enough as presumably they interact with PowerPoint in the same way.

The feel (as well as the 360 name) of the new Storyline is very Microsoft – perhaps predictably considering the close alignment with PowerPoint in the past.  This all feels very Office still, the purple theme giving a warm OneNote kinda feel to proceedings.

Once into Storyline360 things look even more familiar – indeed it looks and acts much like Storyline 2.  The differences for now are really what is highlighted on the first/home screen (see image).

The additional media libraries are fine but many people will have an image bank or corporate access to Thinkstock anyway and Review is probably presumptive that IDs are getting more time with their SMEs (or more of their SMEs’ time) than they are!


Feeling at Home in OneNote Storyline

Then the real value would be left in coming from the responsive player.  It would be lovely if the product comparisons (web / PDF) linked to videos or pages showing the specific change to the Storyline UI and UX.

Bigger differences are clear in Replay where it is now a more genuine alternative to Camtasia Studio thanks to the “New editing tools”.

The editing tools offer the simple Cut/Delete/Split/Silence that you tend to need and would remove my tendancy to switch between iMovie, Replay, Screencastify [which is more the mark ‘Peek’ is after] and other things.

Replay home page

One major change highlighted here

Plenty of good reviews elsewhere (for example here) so I’ll stop at that point.  Needless to say it will be interesting to look into this and other authoring tools again at BETT and Learning Tech this year and try to get a handle of what is the best way forward.