Evaluating your e-Learning Design

Evaluation

After I finish developing an eLearning module I am always wondering if the final product represents a good design. So far project stakeholders are really happy with what I have created but I wanted to measure with a checklist or some sort of evaluation tool if the design was good, and I found what I consider a “wonderful tool” (like everything Cathy Moore creates 🙂 ) called “Checklist for a strong learning design” (Moore, 2015).

I took her tool and I wrote it in excel and I added a point system in the spectrum section to get at the end a score, where the maximum possible score is 70.

Checklist for strong learning design
Module:
No. Action-oriented materials Spectrum Information dump
5 4 3 2 1
1 The goal of the project is to change performance in a visible, measurable way. The goal of the project is to transfer information into people’s brains.
2 Objectives used to design the materials describe visible, on-the-job behaviours that are necessary to reach the project goal (“sell”, “lead”, “encrypt”, “schedule”, “design”). Objectives describe knowledge (“understand”). If behaviours are described, they are behaviours that happen during a test (“identify”, “explain”, “define”).
3 The format of the materials (webinar, PDF, etc.) is determined by the type of activities and users’ needs. The format of the materials is determined by tradition, the LMS, or what’s most convenient for the client.
4 The materials feel like one immersive, challenging activity or a series of activities with little interruption. The materials feel like a presentation that’s occasionally interrupted by a quiz.
5 The authors appear to respect the learners’ intelligence and previous experience. The authors appear to doubt the learners’ ability to draw conclusions and assume they have no experience.
6 Activities make people practice making decisions like the ones they make on the job. Activities are quizzes, trivia games, or other knowledge checks that don’t happen on the job.
7 Activity feedback shows people what happens as a result of their choice; they draw conclusions from the result. Activity feedback explicitly tells people “correct” or “incorrect”; they aren’t allowed to draw conclusions.
8 People can prove that they already know material and skip it. Everyone is required to view every bit of information regardless of their existing knowledge or performance on activities.
9 Reference information is supplied outside the activity in job aids; people practice using the job aids in activities. Reference information is delivered through the course or training; people are expected to memorize it or come back to the course for review.
10 Characters are believable; they face complex, realistic challenges with emotionally compelling consequences. Characters seem fake (e.g.’ preachy or clueless); their challenges are minor and are presented as intellectual exercises.
11 Visuals are used to convey meaning. Visuals are used as “spice”.
12 Photos of people show humans with realistic expressions. Illustrations appear intended for grownups. Visuals of people are stock photo models who are over-acting or childish cartoons.
13 In eLearning, audio narration is used only for:
> Dramatic realism (e.g. characters’ voices in a scenario).
> Explanations of complex or rapidly-changing graphics.
> Motivational messages and explanations from people who really exist (e.g. CEO, subject matter expert).
Audio narration is used to:
> Deliver information while displaying simple, static screens.
> Redundantly read text on the screen.
> Lecture people about what they should or shouldn’t do.
14 The writing is concise, uses contractions, and sounds like a magazine (Flesch Reading Ease score of 50 or higher in Word). The writing is wordy and stiff; it sounds like a textbook or insurance policy (Flesch Reading Ease score of 49 or lower in Word).
SCORE 0 0 0 0 0
TOTAL (Out of 70) 0

With this checklist I can go through different aspects of the course design, give a score and then determine if the design is more an information dump or an action-oriented course. The idea is to build better eLearning products that are more in the spectrum of action-oriented courses. Of course, I am still learning and I think my first products unfortunately are in the “information dump” side but one of the last project I created got a score of 54 out 70 which is not too bad.

This evaluating exercise made me think on new design strategies I should consider in my next projects and how I can build better and stronger courses to achieve the maximum of 70 points.

Thanks Cathy Moore for all the work you do and for making the work of instructional designers a lot easier 😀

Reference: Moore, C. (2011). Checklist for strong learning design. Retrieved from http://blog.cathy-moore.com/2011/07/checklist-for-strong-elearning/

My excel version of the checklist here

Evaluation and Distance Education

Evaluation

After developing and implementing several eLearning solutions in my workplace we are now at a point where we want to take some time to analyse the effectiveness of the training. To do this, we decided to develop a progress report to the stakeholders to gather the results from the course survey and insights on course participation, engagement, issues found, and next steps to conduct the training to a completion or closure process.

Remember that the idea for some of the courses is to have a refresher module later on, or to get to a closure so we can measure if the learning solution was effective and allowed us to achieve the learning objectives identified at the beginning of the consultation process with the stakeholders.

This evaluation phase is relevant because it will bring critical information about how learners perceived the learning tool and how we can improve future projects. From this evaluation, we can get valuable information that will create the design criteria for the next eLearning solutions. So it is important to have an organised approach to collate the data and get the best possible information for our own learning as instructional designers.

An organised and practical approach to do program evaluations in the corporate sector is to use Kirkpatrick’s levels of evaluation (Simonson, 2008). I have prepared the table below to summarise Kirkpatrick’s evaluation model and I have added levels 5 and 6 to adapt it more to my work. This tool helps me in understanding the instruments that can be used to gather data on each level.

 

 

Training Evaluation
 Level of Evaluation  Instruments to Gather Data
 Level 1 – Reactions or Learner Satisfaction

(How users perceive the module, what did they like or not about the training? This can be measured with the course survey)

Survey with questions to identify what people like and didn’t like about the training module, questions to determine how learners perceived the training.Examples:

Rating type of questions (Strongly disagree, disagree, neutral, agree, strongly agree):

  • The information in the module is relevant.
  • The presentation of the module was engaging.
  • Overall, I liked the module (content, layout, interactivity, presentation).

How likely are you to use the information of the course?

How likely are you to share the information of the course?

What did you enjoy most and least about the module?

What would you change (add, remove, improve) in the module?

Would you like to add any other comment?

 Level 2 – Learning

(Have learners advanced in skills, knowledge or attitude? This can be measured with case scenarios, quizzes, completion rate, post-test scenario with questions in a refresher type of training, check if learning objectives defined during consulting process were achieved)

  • Case scenarios with multiple-choice questions.
  • Quizzes.
  • Pre-tests.
  • Post-tests.
  • Completion rate of the course.

Check if learning objectives were achieved.

Level 3 – Transfer or Application of Knowledge

(Are learners using the knowledge in the workplace? )

Are learners:

  • Using the policies?
  • Visiting the intranet?
  • Having discussions about the topic discussed in the training?
  • Following procedures, skills, knowledge and attitudes explained in the training?
Level 4 – Results / Impact of Training on the Organisation

(This is the direct and indirect impact of training on the success of the organisation. This can be measured with the business KPIs, comparing results of pre-test and post-test. Impact most be measurable)

  • Pre-tests.
  • Post-tests.
  • Measuring business KPIs and how the training helped achieve those KPIs.
Level 5 – Usability, Technical issues, Accessibility

(This can be measured with the course survey)

Survey about content, layout, interactivity, presentation, length of course, resources, etc.
 Level 6 – Return on Investment (ROI)
  1. This is measured collecting data from level 4.
  2. Can we convert result of training into monetary value? How much was the saving, or process improvement translated into a monetary value, etc.
  3. Determine cost of training (developing time/cost, time users spent doing training).
  4. Calculate ROI by comparing the monetary benefits to the costs.

 

 

Reference:

Simonson, M. (2007). Evaluation and distance education, five steps. The quarterly review of distance education, Vol 8(3). Retrieved from Kirkpatrick – Evaluation and Distance Education.

Brain-Based, Active, and Self-Directed Learning

Learning Theories and Instruction

The fact that the brain is a living organism connected to each part of the body, making it actively involved in the learning process is the main focus of the brain-based learning instruction (Houff, Klinger, & Coffman, 2013). This type of instruction is learner centered and understands the individual learning process and the ability of each person to incorporate new knowledge into pre-existing concepts.

To achieve successful learning, the main objective of the brain-based instruction is to stimulate the brain of the learner through a rich environment where the learner will find interactive activities that will maintain learners’ engagement during the learning process.

According to Houff, Klinger, and Coffman (2013), it is student-centered learning that utilizes the whole brain and recognizes that not all students learn in the same way. It is also an active process where students are actively engaged in constructing their own knowledge in a variety of learning situations and contexts.

The brain-based learning puts in the table a different approach to learning that is not what I used to see in the typical classroom format, where the strategies are focused on content and how to present it but not on what activities could be more stimulating and engaging for the students. And this is of serious consideration if we want to ensure all individuals make the most out of any learning experience through their lifespan.

To understand more the brain-based instruction I searched articles about the subject in EdITLib, which is a digital library dedicated to education and information technology (http://www.editlib.org/). In this library I found an interesting research called “using Brain-Based Learning Strategies in the Classroom” (Houff, Klinger, & Coffman, 2013). And one aspect from this research captured my attention which is the explanation in easy terms of specific chemical and functional processes in the brain that are relevant information to understand learning processes. Below I cite some of the passages that I considered relevant from this article:

“Through recall, chemicals and neurotransmitters are released into the endocrine system which is connected to synapses, altering and intensifying our conscious experience of the situation. Thus, emotions aid in memory retention or learning. When the threat is decreased through cooperation and a safe learning environment, motivation is increased and positive emotions flourish which helps learning retention (Jensen, as cited in Houff, Klinger, & Coffman, 2013, p.2062).”

“Through extensive brain research, neuroscientists have found that neurons continue to grow throughout adulthood. Thus, if we continually exercise our brains with complex cognitive processes, such as active engagement with content, our existing neural dendrite branches will continually grow and cultivate.” (Houff, Klinger, & Coffman, 2013, p.2061).

“The brain is unique. It is continually developed through each interaction within our environment that we are immersed in at a particular place and time. Neuroscientists have identified this phenomenon as plasticity. A process where our neural pathways are created each time we use our brains to think through a complex problem. If we do not think through complex problems, we lose these neural pathways through the process of pruning (Carter, 2006; Marzano, Pickering, & Pollock, 2001, as cited in Houff, Klinger, & Coffman, 2013, p.2062).”

To implement brain-based strategies in learning experiences I considered that the “Active and Self-directed Learning” (ASDL) model would be advantageous because it includes self-regulation activities such as self-paced learning, critical evaluation, and internal control that could be strengthened during the adolescence when the brain is still developing and is adaptable. And this is why I wanted to read more about ASDL.

ASDL is split into a four part learning cycle:

1) Sensitisation: in this phase, the new topic is presented to active students’ prior knowledge and here is where the process stimulates student’s motivation,

2) Exploration: offering students possibilities to understand subject matter in a meaningful way.

3) Integration: helping students integrate various topics from multidisciplinary perspectives, and

 4) Application: offering students possibilities to show their abilities to apply and reflect on the subject studied.

The cycle supports problem-based learning and the developing minds of students through the integration of information, critical thinking and self-evaluation, while also teaching self-responsibility and team management skills (Czabanowska, Moust, Meijer, Schröder-Bäck, and Reobertsen, 2012).

I consider is worth reading the ASDL model presented by Czabanowska, Moust, Meijer, Schröder-Bäck, and Reobertsen due to its offering of different steps to enhance learning in students with a rich environment combining different activities that stimulate the brain and help students create meaningful connections of the new content. This research can be found in the Journal of university Teaching & Learning Practice (http://ro.uow.edu.au/jutlp/).

References:

Czabanowska, K., Moust, J., Meijer, A., Schröder-Bäck, P., and Reobertsen, H. (2012). Problem-based learning revisited, introduction of active and self-directed learning to reduce fatigue among students. Journal of University Teaching & Learning Practice. Vol. 9 (1). Retrieved from, http://ro.uow.edu.au/jutlp/vol9/iss1/6

Houff, S., Klinger, M.B. & Coffman, T. (2013). Using brain-based learning strategies in the classroom. In. Jan Herrington et al. (Eds.), Proceedings of World Conference on Educational Media and Technology 2013 (pp. 2060-2069). Association for the Advancement of Computing in Education (AACE). Retrieved from, http://www.editlib.org.ezp.waldenulibrary.org/p/112258/share/

 

Mapping my Learning Connections

Learning Theories and Instruction

When I started to think about a career change from engineering to the educational field I did not know where to start or what steps to take. In order to organize my ideas I went online and started researching about the different fields in education and what could be more suitable to me. In this research I found an incredible amount of information and I realized that I could start creating projects and building my own professional experiences with the guidance of all the tools available. I found blogs with valuable information such as practical tips that I could start putting into practice in my current work environment.

But one of the most fantastic tools that I found was the communities in the Learning and Development field, with people sharing their knowledge. Some of the members of these communities have relevant experience in the field and share information through podcasts, videos on Youtube, webinars, blogs, and articles in websites. It is even possible to establish a more direct connection with Twitter, Facebook, and LinkedIn.

My learning network has changed the way I learn, because traditionally I was enrolling in a face-to-face course and receiving the information passively from the instructor. Now, with the different elements in my learning network I have more time to reflect and post my own comments on a particular topic. I have more flexibility in exploring the content that is relevant to me and I take control of my learning process.

Another advantage I encounter in my learning network is the different methods used to present content, which provides a rich diversity to learn from one topic in different ways. For example, if I want to investigate about adult learning theories, I start consulting the different sources in my learning network and I find videos explaining the principles of adult learning, I find also posts in blogs with comments from the readers, diagrams in websites, and podcasts. After visiting those resources, I start making my own opinion and ideas about the topic, I start commenting in the forums, and I can even reflect on the subject and make my own contribution in my blog.

From all those resources available in my learning network I learn easier with videos and diagrams. I can read extensive texts, but as soon as I see a picture, a diagram, or a video, I immediately get the idea the author is trying to transmit. However, when it comes to writing my own reflection about a topic, I like to consult educational journals to read more studies in the topic that can be based on deeper research and then can support my idea with more evidence.

And those educational journals are my main source of information when I have a question about a subject. I start exploring answers to my questions in the different articles available in the educational journals and databases. However, access to those journals are most of the time subject to subscription, then if it is not possible to have access I consult the blogs, or using Twitter I send my question directly to the groups, or forums.

Finally, I would like to mention that my learning network supports the principles of connectivism due to the diversity of opinions, the process of learning where relevant sources of information are connected and there is a continual learning process through the connections with the resources. Also, the currency of the information and learning activities, and the decision-making process that involves managing information that can be correct right now but it could be modified in the future (Siemens, 2014).

Reference:

Siemens, G. (2014). Connectivism: a learning theory for the digital age. Retrieved from http://www.elearnspace.org/Articles/connectivism.htm

Mind Map

Mind Map