‘Is utilizing a mobile application to track student competency achievement efficient and effective?’  Thus began my presentation at the 2011 CNIE Conference in Hamilton, Ontario which was themed ‘Cascades of Innovation’.

 

First – a bit of background.  Over 8 years ago, the company I now work for in Edmonton, Alberta, called Great Big Solutions Ltd., launched and now continually develops and supports a web and mobile competency tracking solution called CompTracker®.  The solution has so far been tailored to post-secondary health care students tracking many different skill and competency profiles in their studies.  

 

Though many people still haven’t heard of us, personal experience and market information tells us demand is going to continue for sound, mobile applications like ours that deliver on the promise of making our work and home lives easier.  

 

In her 2005 article, Enabling Mobile Learning, Ellen Wagner said “The use of PDA-based performance tools to support classroom instruction and on-the-job training alike has been well under way for a number of years, particularly in the fields of medicine and allied health, business, and journalism…”  Further, “…whether we like it or not, whether we are ready for it or not, mobile learning represents the next step in a long tradition of technology-mediated learning.”1
 

So we know there’s demand, and we see it increasing, but, is this a good thing?  This question really stems from my personal need to know, and to question, almost daily, ‘Am I making a difference and contributing to a greater good through the work that I do?’ 

 

Indulge me for a moment as I relay a story.

 

A 20-something student on the last day of her practicum, just before graduation.  She’s a province away from her home university, and only a few junior staff are still onsite before the weekend.

 

Carefully taking 20 plus pages of her reflective journals, and detailed self and preceptor evaluations, she approaches the fax machine.  Fanning the pages, she places them face down on the fax, then double and triple checks the number.  Pressing ‘Send,’ she holds her breath for a moment, but relaxes as the pages begin to scan easily.  She leaves the fax humming away, and goes to bid farewell to staff and her mentors of the past four months.

 

15 minutes later, the student returns to the fax and her heart leaps into her throat!  She looks at the crumpled pages and jammed fax machine in utter disbelief.  How was it possible to make such teeny, tiny accordion folds in paper?! With some pages now illegible and torn – she wishes she’d made copies of her evaluations first – but too late!! 

 

Utterly dejected, she cleans the fax as best as she can and places a desperate call to her practicum coordinator hundreds of kilometres away.  (True story about the author!)

 

So experience, then and now, leads me to believe that mobile competency tracking is a viable solution.  But what do educators and our customers believe?

 

An article review on the ‘efficacy of mobile competency tracking’ revealed there was limited study on this specific topic.  I was able to find a number of publications, however, that discussed various facets of mobile technology incorporated as a learning/teaching tool both in and outside the classroom.  

 

Given the lack of formal studies available through an online search, the time also seemed right to survey our customers about electronic competency tracking to get to the heart of what they found beneficial – or not. Bearing in mind that some of our students may use our website, a device and sometimes a blend of both to submit competency information, some questions remain.  The purpose of this article is to present some food for thought and potential points for future study.

 

The CompTracker® Benefits Assessment Survey was designed to be brief and anonymous, and was provided to customers who had been utilizing our application for at least one year or more.  It included a mix of 16 demographic, open-ended, multiple choice, rating scale, and modified semantic differential questions. 

 

Following the article review and survey, the following four areas surfaced as the most significant.  That is, through utilizing mobile technology:

 

If you as an educator or parent have sensed some differences in terms of how quickly your students or children have glommed onto mobile technology as a learning tool versus your own speed of adaptation, there may be good cause.  As early as 2001, Mark Prensky coined the term ‘Digital Natives’3 to refer to the Kindergarten through College age group who not only behave differently in interacting with technology–their brains may physically be wired differently.4  So if you sometimes cannot relate to your kids or students in this aspect it may be because they’ve embraced mobile technology–literally! body and soul.  As ‘Digital Immigrants’3 we may lag behind in adopting this technology to the same comfort level or extent, but it appears here to stay and there are good reasons for us to jump on the bandwagon with this more technically adept demographic.

 

A Norris and Soloway article talks about using handheld computers to bring K-12 classrooms into the 21st century, but central to success, they say, is that students be sole users of a given device.5  As sole users, students don’t have to share a bank of laptops or mobile devices, let alone, computer lab time, and teachers then seem more likely to incorporate learning activities in the classroom that are centered around the use of technology.  In turn, incorporating technology into learning activities may then aid student focus and organization.  From this same Norris and Soloway article: 

From Ellen Wagner’s article mentioned earlier, “Technology in and of itself may not guarantee better learning, but when effectively deployed, technology can help focus attention while attracting and maintaining a learner’s interest.”1  

 

In post-secondary circles, educators may not run into as many behavior or distractibility issues as K-12, but some of our CompTracker® customer survey comments were similar in theme to those above: 

In the Internet Journal of Internal Medicine, 2010; Ferenchick, Sneed, Solomon and Mohmand evaluated “The effect of a grading incentive and a problem-specific mobile electronic clinical evaluation tool (eCEX) in the direct observation of medical student’s clinical competencies: A pilot study.”6  

In the study, authors say, “Direct observation of medical students’ clinical skills by faculty is uncommon.”  This is also the case for many of our customers’ health care students who are often evaluated by field or clinical site preceptors.  Regardless of who is evaluating the student, the study suggests that transparency of information is important, and goes on to conclude, “Students and the faculty both agreed that the CEX helped them understand which problem-specific competencies were targeted for the assessment… Faculty generally agreed that eCEX improved their ability to provide feedback to the students.”  Which ties in neatly to the next point...

 

I think we can agree that timely feedback serves a very important purpose for students.  It provides an immediate benchmark of their performance.  They can also refer to feedback given to them over time to gauge their progress and relating back to point number one in this article, it can help them determine where to focus their efforts and improve.

 

From the Educause Learning Initiative site, under M-Learning and Mobility:

Using portable computing devices (such as laptops, tablet PCs, PDAs, and smart phones) with wireless networks enables mobility and mobile learning, allowing teaching and learning to extend to spaces beyond the traditional classroom. Within the classroom, mobile learning gives instructors and learners increased flexibility and new opportunities for interaction. Mobile technologies support learning experiences that are collaborative, accessible, and integrated with the world beyond the classroom.7 

 

With CompTracker®, there are numerous opportunities to provide feedback and to collaborate through the application.  Students are generally the ‘drivers’ of the system, and their submissions may be almost real time: as soon as they submit information on the website or sync their device via wireless, the information may be reviewed, feedback given, and where appropriate, graded.  The entire submission, including feedback and rating then becomes part of the students’ educational profile and achievement record.

 

In the case of a significant skills deficit, the practicum coordinator or preceptor may even initiate an Action or Learning plan through CompTracker® to address the skill shortfall.  This shortfall will be evident to all parties with the appropriate permissions to view the students’ information and may be verified and audited later.  This near real-time process means that issues may be identified sooner rather than later and appropriate measures put in place before it may be too late.

 

From our CompTracker® Benefits Assessment Survey, these were some comments received about feedback and collaboration:

The CompTracker® website also supports asynchronous messaging which can be utilized by faculty, students or preceptors to initiate record specific or general queries.  These message notifications are sent to the users’ usual email account, and with email capability on their cell phone, they will be notified of this new message which may also be flagged as urgent and requiring their immediate attention.

 

Whether utilizing a device or the website, the transparent evaluation and feedback process helps to keep the lines of communication open, and parties up to date as well as accountable.  And rather than detracting from collaboration, I am coming to accept that in this new digital age that face-to-face interaction isn’t always necessary to support teamwork on a given endeavour such as when we’re preparing students for competency in entry-to-practice skills.  

 

With textbooks, references, calendars, email, mobile web browsers, document storage, and communication all in one, a mobile device seems to deliver a good ‘bang for your buck’ whether you’re a student buying your own or a school district administrator making a bulk purchase.  Now I’m not sure I would want to read a 500 page text on my iPod Touch if I was that 20 something student again, but given the choice to read it on an iPad or other type of tablet and have the benefits of mobility – no question – I would have saved the minimum wage earnings from my student summer job to purchase one.  
 

There are currently more than 350,000 applications for free or a fee from the Apple® App Store:
http://www.apple.com/iphone/apps-for-iphone/.

Apple® also has a site specifically outlining education applications:  
http://www.apple.com/education/apps/.

 

Norris and Soloway refer to a New York Times article from May 2007 and the one-to-one laptop initiatives being terminated by some school districts due to the total cost of ownership.  Back in 2004, Beverly Walker, deputy superintendent, Alvin (Texas) Intermediate School District commented, “If Alvin was going to prepare its children for the 21st century, I had to find a way to provide each child with a computer.”  She goes on to say, “I didn't see the laptop costs would come down enough to accomplish the goal. But handheld computers seemed to have the right price-performance ratio.”5 

In this same article, Norris and Soloway comment, “Cell phones or cell phone use is banned from many school districts now, but savvy administrators will realize they can avoid buying computers since the students’ own devices will be sufficient for most learning tasks.”5

Though return on investment depends on many factors, strongest being the type of device and the extent to which the device is used in the educational setting, when compared to the cost of purchasing a laptop it’s hard to argue with the return.  A laptop will also have limited mobility.  With a truly mobile device such as an iPod Touch, students can be behind the bulkhead in the X-Ray department, or on wards observing rounds, and when the time is appropriate they can still track their competencies, obtain written feedback and get a valid electronic signature from their preceptor.  As soon as they sync their device via wireless, their placement coordinator from any Internet-connected browser will know what they’ve been working on, how well they’re doing, and if any intervention is required.  Oh – and no jammed fax machines to contend with!
 

 

My informal research indicates workload and job satisfaction are positively impacted through the use of mobile technology.  With my customers and on a personal note, with many of my extended family working, or having worked as educators, I was very pleased with these findings.

 

From our CompTracker® Benefits Assessment Survey: 

In our survey, we specifically questioned customers about the impact to their workload.  We asked, “How has CompTracker Reporting affected the time you spent previously monitoring or calculating student progress when it was done on paper?” Refer to Figure 1 for the results.


  

 

To summarize this chart, 58% of survey respondents experienced a workload reduction of at least 30% (the orange, pale blue, pink and pale green pie pieces).  An equal number of respondents (14%) saved either 10-20% or 20-30%.  Only a small minority of 15% of respondents said that they saw “No noticeable reduction” or only up to 5%.  
 

0-5% is a very low rate of workload savings, and even 10-20% seems low.  Based on comments made in other parts of the survey, there may be some reasons to account for this.  
 

First, some customers, due to their role, use only one component of our system, and they find no time savings when electronically marking attendance, for example.  One customer commented it took them more time as compared to doing it on paper.  This is understandable when nothing further needs to happen with that attendance record.  If that same paper record needs to be filed or entered by someone electronically in another system, however, and they are not one in the same person gathering and entering the information, they may not experience the time savings.

Some of our customers are also still quite dependent on paper tracking methods.  They may even duplicate some of their efforts by validating CompTracker® data against paper records.  As comfort and confidence utilizing in an electronic competency tracking system increases, the amount of paper that needs to be utilized or validated should decrease and result in lower workloads.
 

From the Blog, Reducing Teacher Workload by Mark Bethelemy, “Talk to most teachers in England... about what would make the biggest improvement to their professional lives and they will say ‘reduce the paperwork.’” 8

 

The question, “Has CompTracker affected your personal job satisfaction?” produced the following results; see Figure 2 below.



Figure 2

 

To summarize, a convincing 59% of respondents rated their job satisfaction at least a 4 out of 5 with 5 being “Significant positive impact.”  At the other end of the scale, 15% of customers reported they were “Not noticeably” affected; a rating of 1.

 
 

Mobile competency tracking is not without its challenges, but utilizing what this technology has to offer both in and outside the classroom may lead to improved student behavior, and can assist learners as well as educators to stay better organized and focused on requirements and outcomes. 
 

Mobile technology readily facilitates feedback.  For ‘Digital Immigrants,’ it presents us with an alternative method to face-to-face collaboration which we may want to consider embracing to the same extent as younger generations for the convenience and benefits it can offer.
 

With its many capabilities, its portability and accessibility, mobile technology is cost effective.  Depending, perhaps, on your role and also the extent to which you need to manage paper and then extrapolate information from it in the future, it can be quite time and workload effective as well.  
 

With 78% of our survey respondents indicating their job satisfaction rated at least 3 out of 5 as a result of using our solution, and taking the other factors in this discussion into account, I can reflect  - ‘Am I making a difference and is electronic competency tracking efficient and effective?’ the answer is a Great Big Solutions – YES!

 

 

===================
 

 

1  Wagner, Ellen D., “Enabling Mobile Learning.” Educause Review, vol. 40, no. 3, (May/June 2005): 40-53. http://www.educause.edu/EDUCAUSE+Review/EDUCAUSEReviewMagazineVolume40/EnablingMobileLearning/157976

 

2  Livingston, A., “Smartphones and Other Mobile Devices: The Swiss Army Knives of the 21st Century.”  Educause Quarterly, Volume 27, Number 2, 2004.  

http://www.educause.edu/EDUCAUSE+Quarterly/EDUCAUSEQuarterlyMagazineVolum/SmartphonesandOtherMobileDevic/157292

 

3  Prensky, M., “Digital Natives, Digital Immigrants.” On the Horizon, Vol. 9, No. 5, 2001. Pp. 1-2. http://www.marcprensky.com/writing/default.asp

 

4  Prensky, M., “Digital Natives, Digital Immigrants, Part II: Do They Really Think Differently?”  On the Horizon, Vol. 9, No. 6, 2001, p. 5. http://www.marcprensky.com/writing/default.asp

 

5  Norris, C., & Soloway, E., “Handhelds: Getting Mobile.” (Cover story), District Administration, 44(8), 20-24, 2008, July. http://www.districtadministration.com/viewarticle.aspx?articleid=1638

 

6  G. Ferenchick, S. Sneed, D. Solomon & A. Mohmand. “The effect of a grading incentive and a problem-specific mobile electronic clinical evaluation tool (eCEX) in the direct observation of medical student’s clinical competencies: A pilot study.” The Internet Journal of Internal Medicine. 2010 Volume 8 Number 2. 
http://www.ispub.com/journal/the_internet_journal_of_internal_medicine/volume_8_number_2_18/article_printable/the-effect-of-a-grading-incentive-and-a-problem-specific-mobile-electronic-clinical-evaluation-tool-ecex-in-the-direct-observation-of-medical-student-s-clinical-competencies-a-pilot-study.html

 

7  Educause Learning Initiative, M-Learning and Mobility, 2006, http://www.educause.edu/M-LearningandMobility/12397

 

8  Berthelemy, Mark.  “Reducing teacher workload,” Learning Conversations (blog).  November, 2007.  

http://www.learningconversations.co.uk/main/index.php/2009/11/07/reducing-teacher-workload?blog=5