Skip to main content
STUDENTS! Join our social media pages to stay informed of new features, versions, and upcoming contests!
Like us on Facebook!

Follow us on Google+!




 CompTracker Blog 
Wednesday, July 06 2011
Author: Wendy Kostiuk
Presented at The Canadian Network for Innovation in Education Conference, 2011 as a practice-based report on emerging trends in educational technology.
‘Is utilizing a mobile application to track student competency achievement efficient and effective?’  Thus began my presentation at the 2011 CNIE Conference in Hamilton, Ontario which was themed ‘Cascades of Innovation’.
First – a bit of background.  Over 8 years ago, the company I now work for in Edmonton, Alberta, called Great Big Solutions Ltd., launched and now continually develops and supports a web and mobile competency tracking solution called CompTracker®.  The solution has so far been tailored to post-secondary health care students tracking many different skill and competency profiles in their studies.  
Though many people still haven’t heard of us, personal experience and market information tells us demand is going to continue for sound, mobile applications like ours that deliver on the promise of making our work and home lives easier.  
In her 2005 article, Enabling Mobile Learning, Ellen Wagner said “The use of PDA-based performance tools to support classroom instruction and on-the-job training alike has been well under way for a number of years, particularly in the fields of medicine and allied health, business, and journalism…”  Further, “…whether we like it or not, whether we are ready for it or not, mobile learning represents the next step in a long tradition of technology-mediated learning.”1
So we know there’s demand, and we see it increasing, but, is this a good thing?  This question really stems from my personal need to know, and to question, almost daily, ‘Am I making a difference and contributing to a greater good through the work that I do?’ 
Indulge me for a moment as I relay a story.
Picture this:
A 20-something student on the last day of her practicum, just before graduation.  She’s a province away from her home university, and only a few junior staff are still onsite before the weekend.
Carefully taking 20 plus pages of her reflective journals, and detailed self and preceptor evaluations, she approaches the fax machine.  Fanning the pages, she places them face down on the fax, then double and triple checks the number.  Pressing ‘Send,’ she holds her breath for a moment, but relaxes as the pages begin to scan easily.  She leaves the fax humming away, and goes to bid farewell to staff and her mentors of the past four months.
15 minutes later, the student returns to the fax and her heart leaps into her throat!  She looks at the crumpled pages and jammed fax machine in utter disbelief.  How was it possible to make such teeny, tiny accordion folds in paper?! With some pages now illegible and torn – she wishes she’d made copies of her evaluations first – but too late!! 
Utterly dejected, she cleans the fax as best as she can and places a desperate call to her practicum coordinator hundreds of kilometres away.  (True story about the author!)
So experience, then and now, leads me to believe that mobile competency tracking is a viable solution.  But what do educators and our customers believe?
A Survey is born
An article review on the ‘efficacy of mobile competency tracking’ revealed there was limited study on this specific topic.  I was able to find a number of publications, however, that discussed various facets of mobile technology incorporated as a learning/teaching tool both in and outside the classroom.  
Given the lack of formal studies available through an online search, the time also seemed right to survey our customers about electronic competency tracking to get to the heart of what they found beneficial – or not. Bearing in mind that some of our students may use our website, a device and sometimes a blend of both to submit competency information, some questions remain.  The purpose of this article is to present some food for thought and potential points for future study.
The CompTracker® Benefits Assessment Survey was designed to be brief and anonymous, and was provided to customers who had been utilizing our application for at least one year or more.  It included a mix of 16 demographic, open-ended, multiple choice, rating scale, and modified semantic differential questions. 
Following the article review and survey, the following four areas surfaced as the most significant.  That is, through utilizing mobile technology:
  1. Students may be more focused, organized and better ‘behaved’
  2. Feedback and collaboration may be improved
  3. It provides a good return on investment due to its Swiss Army Knife capabilities2
  4. It results in decreased workload and increased job satisfaction for educators
Students wired for technology
If you as an educator or parent have sensed some differences in terms of how quickly your students or children have glommed onto mobile technology as a learning tool versus your own speed of adaptation, there may be good cause.  As early as 2001, Mark Prensky coined the term ‘Digital Natives’3 to refer to the Kindergarten through College age group who not only behave differently in interacting with technology–their brains may physically be wired differently.4  So if you sometimes cannot relate to your kids or students in this aspect it may be because they’ve embraced mobile technology–literally! body and soul.  As ‘Digital Immigrants’3 we may lag behind in adopting this technology to the same comfort level or extent, but it appears here to stay and there are good reasons for us to jump on the bandwagon with this more technically adept demographic.
1. Students more focused, organized and better ‘behaved’ 
A Norris and Soloway article talks about using handheld computers to bring K-12 classrooms into the 21st century, but central to success, they say, is that students be sole users of a given device.5  As sole users, students don’t have to share a bank of laptops or mobile devices, let alone, computer lab time, and teachers then seem more likely to incorporate learning activities in the classroom that are centered around the use of technology.  In turn, incorporating technology into learning activities may then aid student focus and organization.  From this same Norris and Soloway article: 
  • A fourth-grade student said “I don't have to hunt for pieces of paper anymore...  Everything is in one place.”
  • Kerri Neubauer, instructional technology coordinator, Alvin School District in Texas said “...And we see that the use of the devices definitely results in increased student motivation. We are seeing significantly fewer behavior problems in classes where the mobile computers are being used.”
  • Monique Shorr, sixth-grade science and social studies teacher at Hartland Farms Intermediate Schools in Michigan: “…Over the semester my students become autonomous learners who can take charge of their own learning.”
From Ellen Wagner’s article mentioned earlier, “Technology in and of itself may not guarantee better learning, but when effectively deployed, technology can help focus attention while attracting and maintaining a learner’s interest.”1  
In post-secondary circles, educators may not run into as many behavior or distractibility issues as K-12, but some of our CompTracker® customer survey comments were similar in theme to those above: 
  • Students are able to track their progress and [have] been more time management efficient.
  • Greater clarity by students as to program expectations.
  • Students are happy not to have to drag a paper binder with them everywhere they go.
  • No outstanding question from faculty as to which students complete the competencies.
  • THE simplest method of competency tracking we have utilized in the last 11 years.
In the Internet Journal of Internal Medicine, 2010; Ferenchick, Sneed, Solomon and Mohmand evaluated “The effect of a grading incentive and a problem-specific mobile electronic clinical evaluation tool (eCEX) in the direct observation of medical student’s clinical competencies: A pilot study.”6  

In the study, authors say, “Direct observation of medical students’ clinical skills by faculty is uncommon.”  This is also the case for many of our customers’ health care students who are often evaluated by field or clinical site preceptors.  Regardless of who is evaluating the student, the study suggests that transparency of information is important, and goes on to conclude, “Students and the faculty both agreed that the CEX helped them understand which problem-specific competencies were targeted for the assessment… Faculty generally agreed that eCEX improved their ability to provide feedback to the students.”  Which ties in neatly to the next point...
2. Improved collaboration and feedback
I think we can agree that timely feedback serves a very important purpose for students.  It provides an immediate benchmark of their performance.  They can also refer to feedback given to them over time to gauge their progress and relating back to point number one in this article, it can help them determine where to focus their efforts and improve.
From the Educause Learning Initiative site, under M-Learning and Mobility:
Using portable computing devices (such as laptops, tablet PCs, PDAs, and smart phones) with wireless networks enables mobility and mobile learning, allowing teaching and learning to extend to spaces beyond the traditional classroom. Within the classroom, mobile learning gives instructors and learners increased flexibility and new opportunities for interaction. Mobile technologies support learning experiences that are collaborative, accessible, and integrated with the world beyond the classroom.7 
With CompTracker®, there are numerous opportunities to provide feedback and to collaborate through the application.  Students are generally the ‘drivers’ of the system, and their submissions may be almost real time: as soon as they submit information on the website or sync their device via wireless, the information may be reviewed, feedback given, and where appropriate, graded.  The entire submission, including feedback and rating then becomes part of the students’ educational profile and achievement record.
In the case of a significant skills deficit, the practicum coordinator or preceptor may even initiate an Action or Learning plan through CompTracker® to address the skill shortfall.  This shortfall will be evident to all parties with the appropriate permissions to view the students’ information and may be verified and audited later.  This near real-time process means that issues may be identified sooner rather than later and appropriate measures put in place before it may be too late.
From our CompTracker® Benefits Assessment Survey, these were some comments received about feedback and collaboration:
  • ... more transparent for the student; they can access the information at any time; easier to follow the students from one course to another...
  • ... students can plan their clinical activities according to their reports.
  • Use of CompTracker has allowed for more timely and accurate evaluation of our students in the Clinical and Practicum setting; we would not switch back to paper.  
The CompTracker® website also supports asynchronous messaging which can be utilized by faculty, students or preceptors to initiate record specific or general queries.  These message notifications are sent to the users’ usual email account, and with email capability on their cell phone, they will be notified of this new message which may also be flagged as urgent and requiring their immediate attention.
Whether utilizing a device or the website, the transparent evaluation and feedback process helps to keep the lines of communication open, and parties up to date as well as accountable.  And rather than detracting from collaboration, I am coming to accept that in this new digital age that face-to-face interaction isn’t always necessary to support teamwork on a given endeavour such as when we’re preparing students for competency in entry-to-practice skills.  
3. Good ROI with ‘Swiss Army Knife’ capabilities2
With textbooks, references, calendars, email, mobile web browsers, document storage, and communication all in one, a mobile device seems to deliver a good ‘bang for your buck’ whether you’re a student buying your own or a school district administrator making a bulk purchase.  Now I’m not sure I would want to read a 500 page text on my iPod Touch if I was that 20 something student again, but given the choice to read it on an iPad or other type of tablet and have the benefits of mobility – no question – I would have saved the minimum wage earnings from my student summer job to purchase one.  
There are currently more than 350,000 applications for free or a fee from the Apple® App Store:
Apple® also has a site specifically outlining education applications:
Norris and Soloway refer to a New York Times article from May 2007 and the one-to-one laptop initiatives being terminated by some school districts due to the total cost of ownership.  Back in 2004, Beverly Walker, deputy superintendent, Alvin (Texas) Intermediate School District commented, “If Alvin was going to prepare its children for the 21st century, I had to find a way to provide each child with a computer.”  She goes on to say, “I didn't see the laptop costs would come down enough to accomplish the goal. But handheld computers seemed to have the right price-performance ratio.”

In this same article, Norris and Soloway comment, “Cell phones or cell phone use is banned from many school districts now, but savvy administrators will realize they can avoid buying computers since the students’ own devices will be sufficient for most learning tasks.”5

Though return on investment depends on many factors, strongest being the type of device and the extent to which the device is used in the educational setting, when compared to the cost of purchasing a laptop it’s hard to argue with the return.  A laptop will also have limited mobility.  With a truly mobile device such as an iPod Touch, students can be behind the bulkhead in the X-Ray department, or on wards observing rounds, and when the time is appropriate they can still track their competencies, obtain written feedback and get a valid electronic signature from their preceptor.  As soon as they sync their device via wireless, their placement coordinator from any Internet-connected browser will know what they’ve been working on, how well they’re doing, and if any intervention is required.  Oh – and no jammed fax machines to contend with!
Given the portability, capability, and accessibility of mobile competency applications, the bottom line on ROI is difficult to dispute.
4. Workload and Job Satisfaction
My informal research indicates workload and job satisfaction are positively impacted through the use of mobile technology.  With my customers and on a personal note, with many of my extended family working, or having worked as educators, I was very pleased with these findings.
From our CompTracker® Benefits Assessment Survey: 
  • We need current data on a student’s competency status and how they are progressing.  Paper is too long and takes many man hours to tally.
  • .... Our gradebook used at the college requires overnight results.  It is easy for me to get the results to input my grades on our college system.
  • Allows time for me to ‘teach.’
  • Makes more efficient use of time in a fast-paced program, and also it is recognized by our accrediting body as an acceptable way of tracking and recording student progress.
In our survey, we specifically questioned customers about the impact to their workload.  We asked, “How has CompTracker Reporting affected the time you spent previously monitoring or calculating student progress when it was done on paper?” Refer to Figure 1 for the results.

Figure 1
To summarize this chart, 58% of survey respondents experienced a workload reduction of at least 30% (the orange, pale blue, pink and pale green pie pieces).  An equal number of respondents (14%) saved either 10-20% or 20-30%.  Only a small minority of 15% of respondents said that they saw “No noticeable reduction” or only up to 5%.  
0-5% is a very low rate of workload savings, and even 10-20% seems low.  Based on comments made in other parts of the survey, there may be some reasons to account for this.  
First, some customers, due to their role, use only one component of our system, and they find no time savings when electronically marking attendance, for example.  One customer commented it took them more time as compared to doing it on paper.  This is understandable when nothing further needs to happen with that attendance record.  If that same paper record needs to be filed or entered by someone electronically in another system, however, and they are not one in the same person gathering and entering the information, they may not experience the time savings.
Some of our customers are also still quite dependent on paper tracking methods.  They may even duplicate some of their efforts by validating CompTracker® data against paper records.  As comfort and confidence utilizing in an electronic competency tracking system increases, the amount of paper that needs to be utilized or validated should decrease and result in lower workloads.
From the Blog, Reducing Teacher Workload by Mark Bethelemy, “Talk to most teachers in England... about what would make the biggest improvement to their professional lives and they will say ‘reduce the paperwork.’” 8
The question, “Has CompTracker affected your personal job satisfaction? produced the following results; see Figure 2 below.

Figure 2
To summarize, a convincing 59% of respondents rated their job satisfaction at least a 4 out of 5 with 5 being “Significant positive impact.”  At the other end of the scale, 15% of customers reported they were “Not noticeably” affected; a rating of 1.
Mobile competency tracking is not without its challenges, but utilizing what this technology has to offer both in and outside the classroom may lead to improved student behavior, and can assist learners as well as educators to stay better organized and focused on requirements and outcomes. 
Mobile technology readily facilitates feedback.  For ‘Digital Immigrants,’ it presents us with an alternative method to face-to-face collaboration which we may want to consider embracing to the same extent as younger generations for the convenience and benefits it can offer.
With its many capabilities, its portability and accessibility, mobile technology is cost effective.  Depending, perhaps, on your role and also the extent to which you need to manage paper and then extrapolate information from it in the future, it can be quite time and workload effective as well.  
With 78% of our survey respondents indicating their job satisfaction rated at least 3 out of 5 as a result of using our solution, and taking the other factors in this discussion into account, I can reflect  - ‘Am I making a difference and is electronic competency tracking efficient and effective?’ the answer is a Great Big Solutions – YES!
1  Wagner, Ellen D., “Enabling Mobile Learning.” Educause Review, vol. 40, no. 3, (May/June 2005): 40-53.
2  Livingston, A., “Smartphones and Other Mobile Devices: The Swiss Army Knives of the 21st Century.”  Educause Quarterly, Volume 27, Number 2, 2004.
3  Prensky, M., “Digital Natives, Digital Immigrants.” On the Horizon, Vol. 9, No. 5, 2001. Pp. 1-2.
4  Prensky, M., “Digital Natives, Digital Immigrants, Part II: Do They Really Think Differently?”  On the Horizon, Vol. 9, No. 6, 2001, p. 5.
5  Norris, C., & Soloway, E., “Handhelds: Getting Mobile.” (Cover story), District Administration, 44(8), 20-24, 2008, July.
6  G. Ferenchick, S. Sneed, D. Solomon & A. Mohmand. “The effect of a grading incentive and a problem-specific mobile electronic clinical evaluation tool (eCEX) in the direct observation of medical student’s clinical competencies: A pilot study.” The Internet Journal of Internal Medicine. 2010 Volume 8 Number 2.
7  Educause Learning Initiative, M-Learning and Mobility, 2006,
8  Berthelemy, Mark.  “Reducing teacher workload,” Learning Conversations (blog).  November, 2007.
Posted by: Wendy Kostiuk AT 02:35 pm   |  Permalink   |  0 Comments  |  Email
Friday, September 03 2010
Have you compared results with your key colleagues and are you in agreement, to a good extent, on the key features and user interface that the application must have?  Are you in agreement about the short and long term costs, resources and timelines to implement?  Have you decided on a packaged, custom built application or blended approach? 

If you’ve been able to get agreement on the above, you have done exceptionally well and will make a great project champion!  (And prepare for I.T. firms to start asking for your resume!)  This initial phase can be the most challenging part of the project.  Once the decision is made, the project often develops a life (and a head of steam) of its own. 

If you have been unable to obtain agreement, you are not alone.  You may need to revisit the biggest stumbling blocks and see if you can come to terms.  Or perhaps you feel discouraged and tempted to abandon the project altogether – but don’t give up!  The work you’ve invested, despite lack of a clear outcome, is extremely valuable.  It should point you towards those key factors standing in the way of a decision on whether ‘To Build or Buy’.  Once those factors are isolated, you can focus on them specifically without other distractions.  

This may be a good time to share some of the remaining load with a Software Development Business Analyst or a Solutions Specialist.  With your invaluable, in-depth knowledge of the project and requirements, a third party can take over the reins and oversee final negotiations while you provide firm direction.  It may not be possible to get complete agreement, but when communication is open, issues are thoroughly researched and decisions are given alongside the rationale behind them, project participants may be agreeable to proceed after all.

To sum up this Blog Series, you may have realized that the more you research, the more complex it may be to decide whether ‘To Build or to Buy’ a software application.  But really, that’s a good thing! 

It’s far better to appreciate the complexity now than to potentially have a VERY EXPENSIVE, unknown, unused and unwanted application that keeps rearing its ugly head during your budget discussions for years to come. 

Whatever your choice, we wish you much success.

How have your productivity projects gone in the past?  Do you have any further advice for our readers?  Please comment.  We’d love to keep the conversation going!
Posted by: Wendy Kostiuk AT 01:19 pm   |  Permalink   |  0 Comments  |  Email
Friday, September 03 2010
This Exercise assumes you are leaning more towards Buying an off-the-shelf application rather than Building one.  Still, if you are considering the latter, you may be interested in off-the-shelf application considerations....

Take a moment to reflect back on your application’s key features and user interface from Exercise 2.  How many key features did you document?  Are they all of equal, #1 importance?  Can any be reduced in importance or set aside for now?  It may be exceptionally difficult if not impossible to find a suitable off-the-shelf application if all of your requirements are an equally high, #1 priority or #2 for that matter.   Realistically, you’re going to need to consider prioritizing your requirements and to look for an application that meets the most important priorities for now.

Imagine your requirements in a bell curve like the graph shown below.  On a scale of 1 to 5 with 1 being Highest and 5 being Lowest, the majority of your requirements should fall in the 2 to 4 range. 

Using your results from Exercise 2 or the To Build or Buy Worksheet Tool from Exercise 3, try, as best you can, to prioritize your requirements using a bell curve type of distribution.  In a spreadsheet, list the features in rows going down and the priorities in columns going across.  Put a number 1 in the appropriate row/column intersection.  Then, total your numbers for each column.  You can eyeball your results to roughly evaluate the distribution.  Alternatively, you can take the total number for each column divided by the total number of all requirements and calculate the percentage. 

If your results are very skewed from a bell curve type of distribution, review your list of priorities again.  Go through this process with your key contacts.  Chances are, they may see things from a different perspective and can help you ‘normalize’ the curve. 

After you have a more even bell curve of requirements, your product research begins.  Search engines, blogs, and industry colleagues may provide some promising suggestions.  If your search turns up few/no results, your I.T. department, software product or consulting companies or a Software Product Specialist may be able to point you in a helpful direction.

At the end of the day, if you find an off-the-shelf application that meets more than 75% of your highest priority requirements (i.e., #1 – 3 or 4) this may be most cost effective for your organization in the long run.  Mark Lutchen, head of Pricewaterhouse Coopers IT Effectiveness practice and former global CIO said “Everybody knows that the more standardized you are and the more you buy off-the-shelf, the more cost effective it will be for both implementation and ongoing maintenance.”1

If possible, evaluate more than one off-the-shelf application before narrowing down your list.  Product webinars or one-on-one demo’s could give you a good sense as to whether an application will meet your highest priority requirements.  Invite key colleagues to attend a follow up webinar or demo if you find a promising solution or two.  Include your I.T. staff in the process and the decision making if implementation of the product will impact them in any way.  Product brochures may also contain helpful information to assist you in doing an initial comparison of product versus product.  Also document if the company offers a pilot, risk free trial or other offer suited to your needs.    

Once you have obtained a shortlist of solutions, compare them against each other based on your #1 - #3 (4) priorities.  Which is the best fit?  What is the initial cost and effort to implement?  Remember to include the costs of documentation, training, support and even software and hardware updates following implementation.  Those maintenance and support costs can really add up after your initial purchase.

Finally, it bears mentioning – again – to try and limit the #1 or highest priority requirements that you and your colleagues may have.  (Do you sense this is a recurring theme in this type of endeavour?)  Not reigning in the scope of what you absolutely need could end up being cost, time and/or resource prohibitive and your application may never see the light of day....

Minimizing the number of your highest priority requirements can be a major challenge.  Have you gone through this process?  How were you able to manage it?  If you’re struggling with it, perhaps we can help.  We and our readers would love to hear your words of wisdom on this difficult, but ultimately rewarding process.

1 Polly S. Traylor, InfoWorld, November, 2009.
Posted by: Wendy Kostiuk AT 12:46 pm   |  Permalink   |  0 Comments  |  Email
Monday, August 23 2010
In this Exercise, let’s start by reviewing Exercise 4 and build on what you discovered about your application development project, that is:
  1. the people and their capabilities as well as the software, hardware and other resources required; availability of all resources;
  2. the application’s complexity and Kill points for time, budget and scope;
  3. the maintenance and support costs over time;
  4. the anticipated launch date as well as the costs and timelines multiplied by a factor of 2 or 3 times.
What were your responses to the above?  Were you unsure or fairly confident about people’s skills, their availability and all the resources that might be needed?  Were you unsure or fairly confident about the application’s complexity, the effort it would take to build it, and also how much it could cost to support it over time?  Were you sure about the Kill points and would your key end user group agree with you?  Were you aware of how common scope change and bugs are in the application development process and how significantly they could impact the cost of your application and even double or triple your cost and/or time estimates? 

The more unsure you are of the information above, the more predictive it could be of a project at risk of failure or of running over budget, time or not meeting requirements.  From Exercise 4 and the Standish Group’s statistics, you will remember 68% of all I.T. projects fall under these categories.1  Still, it is far better to know and accept what you don’t know now than to be in the midst of application development and losing sleep over the lack of progress you are making, the amount of time it is taking and/or the money it is costing. 

Given a substantial amount of uncertainty, you may want to reconsider your decision to Build a software application.  It may be time to obtain the services of a Software Development Business Analyst or a Solutions Specialist.  You may also want to look for an off-the-shelf solution or consider a blended approach. 

What is a blended approach?  If you can find a ready-made solution that meets many of your requirements, you may be able to plan internally or to contract for some customization immediately after the ‘off-the-shelf’ implementation that will ‘top up’ your application.  You may be able to get this additional development done internally.  Or, sometimes, off-the-shelf companies provide these services on a fee for service basis following implementation.  Another option could include contracting with a third party for the extra development to have modules or tools built to fill in some of the off-the-shelf software gaps.

Finally, one more question to consider - -  what is the driving force behind development of your  application?  If revenue or competitive advantage are crucial to your business, you may still want to spend the time, money, and effort on custom development according to Bob Laird, IT Chief Architect at MCI/Verizon.2  Weigh out the importance of revenue and competitive advantage to your business compared against the certainty or lack thereof of factors from Exercise 4, and summarized at the beginning of this Exercise. 

Are you closer to an answer?  Is a custom application still for you or have you reconsidered?  What was it that tipped the balance for you either way?

We and our readers would love to hear what you’ve decided to do and what was most compelling in your decision.

1 The Standish Group Newsroom, Chaos Report 2009.  Boston Massachusetts, April 23, 2009.

Polly S. Traylor, InfoWorld, November, 2009.

Posted by: Wendy Kostiuk AT 02:01 pm   |  Permalink   |  0 Comments  |  Email
Tuesday, August 10 2010
If the ‘Build vs Buy’ tool clearly suggests you develop your own custom application, take a deep breath and review the statistics below.  (And if the tool suggested you should probably Buy, you may still be interested in seeing the latest research and questions to ponder....)

I.T. Analysis Experts, Standish Group, found in 2009, only 32 percent of all software projects succeeded!  24 percent failed or were never used, and 44 percent ran over time, over budget and/or didn’t have all the requirements!

So this is where your research really hits the road.  Review the following questions, and where you don’t have definitive answers, you are going to need to do some digging and documenting.  Discuss questions and findings with your key end users to ensure you consider the most important and necessary factors.

Do you have a good sense of the project effort involved to develop your custom application?   It takes the combined time and talents of a number of people, as well as other resources to formally design, architect, develop, test and deliver a software application.  The breadth and complexity, the time you have to build it and the budget you have to work with will all influence the end result.

Using what you have learned so far about your key end users as well as the features and UI, create a spreadsheet, and in a column, draft a list of all the project personnel you think you will need.  These are people not only developing the application, but approving your project plan and budget, assisting you with the management, etc.. 

Below the list of project personnel, in the same column, add the software, hardware and related resources that will be required for your application.  Do the best you can researching and documenting this information.  Ask your key end users for help. 

For the most part, Exercise 4 presupposes that you have an I.T. department with software developers, who have the time, resources and capabilities to respond to work on this project.  If they cannot respond for whatever reason, though, you may then have to look at recruiting, training, increased employee benefits, workspace, and management costs.  Add appropriate items to your list of people and resources if you do not have software developers.

Do you know your approximate budget and roughly how complex the application is?  Consider a simple, web-based application of 10-15 screens that has minimal data entry, calculation and storage requirements and is designed for less than 1000 users.  Is your application more or less complex than this?  (Determining complexity of applications is a blog series unto itself, so do your best guesstimating for now.)

For a very simple application like that above, if 2 developers each make $25 per hour, work 8 hours per day, 5 days per week over 8 weeks, that equals $16,000.  Is that within your budget?  Do these developers have the necessary skills to ‘hit the ground running’ or will they need time and training to get up to speed?  Do they require the guidance of a business analyst, project manager, or software development lead?  Have you considered the cost for design and project meetings as well as computers, other hardware, and software needed? 

Add a new column to your spreadsheet created in sub-section a. above and note your estimated project costs.  Total all those costs. 

Some other considerations - - Can you afford to expand the project if it goes over time, over budget or the initial requirements change?  Do you have a KILL point in mind if the project is taking too long, is costing too much or is not going to provide what you need in your application?   Make a note of your KILL point limits for each factor, and discuss with your key end users and stakeholders.

Have you considered the costs of post implementation maintenance and support? That is, once your application is up and running, who will continue to update your application and to support and train the people using it?  What are those costs?  According to Mark Lutchen, head of Pricewaterhouse Coopers IT Effectiveness practice, “...70 percent of software costs occur after implementation.”
[ii]  In other words, the time, money and effort that you estimate as the ‘cost of the application’ is just the tip of the iceberg.  As technology and your needs evolve, your application will also need to be updated.  In addition, the costs of training and support need to be factored into department budgets – yours or someone else’s or perhaps both.  Application support and maintenance contracts can be the bread and butter of I.T. consulting companies for good reason.   There is always a need for support, training and updating of an application following its initial release.

Start a new row on your resources and costs spreadsheet.  Multiply the cost of your project by 70 percent.  Divide by the expected lifecycle of the application, up to approximately 8 years, according to Mark Lutchen
[iii].  This is the annual maintenance and support budget.  Is it acceptable to you, your I.T. department or other stakeholders?

Do you have sufficient time before you need to launch your application? With the web based application in mind from sub-section b. above, additional time and costs need to be factored in to address software bugs as well as changes in requirements as you see the application take shape.  (The latter is referred to as scope change and both ‘bug fixes’ and scope change are inevitable in projects.)  If you have a set launch date, do you have the money and resources to get your application built and tested well before that proposed date to take these ‘bug fixes’ and scope changes into account?

Whatever your research has told you, it is usually safe to double or triple all cost as well as time estimates in consideration of the staggering statistics from The Standish Group above.  In a new row in your spreadsheet, double then triple the initial project cost.  Also, add two new columns to the right of your resource list costs.  Estimate their start date available, then estimate the end date they are required.  Total the days or months, then add a third and fourth column doubling then tripling those end dates as well.

Custom application development is not for the faint of heart.  Review the above sections and questions carefully and discuss with your key end user group and other important stakeholders.  Perhaps you may decide to reconsider your tentative decision to Build a software application.  You may wish to go back and try worksheet Exercise 3 again, or at the very least, Exercise 4 may have given you some important factors to consider with respect to your timelines, budget and the extent of the features/UI you would like in your custom application. Considered individually, the timelines, cost or scope could be somewhat daunting, but furthermore, changing any one of those aspects will have a significant impact on the other two and that will ultimately impact the quality of your application as well as the final outcome.

Are you ready to take the plunge into the world of software application development, or have you already done so?  What are your biggest concerns before beginning, or, if you have already implemented a custom application, was it a success, partial success or a failure?  What valuable lessons did you learn?  Our readers would love to hear from you, please comment below!

[i] The Standish Group Newsroom, Chaos Report 2009.  Boston Massachusetts, April 23, 2009.

[ii] Polly S. Traylor, InfoWorld, November, 2009.

[iii]  Polly S. Traylor, InfoWorld, November, 2009.

[iv] Project management triangle.  Wikipedia.  Last modified June 15, 2010.

Posted by: Wendy Kostiuk AT 03:00 pm   |  Permalink   |  0 Comments  |  Email
Thursday, August 05 2010
By now, you will have a good sense of who your key end user group is, and they will be on board with you in this endeavour.  You will also have a good idea of the features and user interface you would like for your application and this information has been documented and agreed to with your group, preferably, in electronic format.

Did you know that there is a spreadsheet tool that could help you calculate whether you should lean more towards whether ‘To Build or Buy’? 
Try a workbook tool developed by Craig Borysowich, Chief Technology Tactician, Toolbox for IT:  “Using the Build vs Buy Decision Workbook” 2

As a proviso, when I started this blog series, the application I had in mind was web based and hardware independent, so some of the workbook questions were not relevant and skewed my results.  That said, it should be fairly obvious how your responses will impact the end results, so weight your answers accordingly.  Try the exercise a second time, changing some of your weighted responses to gauge the impact.  Regardless, the process of going through the exercise itself may give you reason to pause and reflect and to potentially consider some new factors.

See the page with a link to the worksheet just below.  Save the worksheet with the rest of your project documents for future reference and discussion.  Once this is done, and you’re ready to start, you may need to unprotect at least one worksheet.  If you’re using Excel 2007, click Review –> Unprotect Sheet or use Excel’s Help feature in other versions.  Click here for a link to the workbook page:

Note: Mr. Borysowich has detailed instructions in the use of the workbook via the link above.  Also, if you are  unfamiliar with some abbreviations, most can be found through Wikipedia.

As with Exercise 2 previously, you may want to complete this exercise independently to start, then compare results in a facilitated group session.  What were your results?  Were they similar or very different?  Were those differences a result of varying opinions, a lack of clarity about the application or something else? 

We would really appreciate your feedback about this exercise!
  Let us know if it helped, if it didn’t and how.  Do you know of any other similar resources that could assist our readers?

In Exercise 4 next week, we will dig into some of the statistics and considerations to be aware of if your end results lean towards 'Building' your application.


Craig Borysowich, Toolbox for IT, ‘Using the Build vs Buy Decision Workbook’

Posted by: Wendy Kostiuk AT 10:00 am   |  Permalink   |  0 Comments  |  Email
Wednesday, July 28 2010

In these first two exercises we will explore the importance of knowing your key end-users and why you need to enlist their help.  We will also look at the start of gathering requirements and how crucial that is in the process of making a decision as to whether 'To Build or Buy'.

1.  Are you going to be a key end-user of the new application?  If yes, great!  Please carry on.  If no, if you're not sure, or if you are only one of a handful or more of key end-users, you will need to consider who may be involved.  Enlist their help sooner rather than later in the requirements gathering process starting in Exercise 2. Not to do so could mean your investigation is already on shaky ground and here's why....

The Importance of Different Perspectives
When building or buying an application, considering different perspectives is absolutely essential or the design and eventual use and adoption of the application could be in jeopardy.  Moreover, it's the management of these different viewpoints that may be crucial.  Perhaps this quote sums it up best, "Poor requirements gathering and management are directly responsible for 60-80 percent of [project] failures."1 (Meta Group)

Get other key people on board as soon as possible and consider enlisting the help of a software development Business Analyst (BA) or Certified Project Manager (CPM).

Document Findings
As well, document, document, document.  The further you continue in this decision making process, the more likely you are to need either more input or approval for funding/resources so a paper and audit trail can clearly support your endeavour.  Start a centrally accessible file repository where you keep electronic records of all your research as well as any decisions made, who made them, and when. Document the rationale or why certain decisions have been made too.  If your project ever goes south, this information could be vital.

 2.  As a key end-user, can you list the purpose, objectives and key functions of the application including how it will act (features) and how it will look (user interface or UI)?  Whether you use an electronic spreadsheet and list features with their associated actions, or draw pictures on loose leaf representing what happens when you click a particular button - if you can do this for your entire application, you are doing amazingly well!  Carry on. 

If you are one of a number of key end-users or you're not fully confident that you've captured all of the necessary features and the look of the application, that's OK.  It's important, and necessary, really, to have help from others with this requirements gathering phase.  

Set formal time aside to meet briefly with other key end-users and ask them to complete the spreadsheet/loose leaf exercise above, independently if possible, with little input/bias from you.  In a few days, you can come together as a group with all key end-users to compare results. 

Start with Independent Requirements Gathering
Why do the feature and UI exercise independently?  Group dynamics can sometimes squelch the input of valued, but soft spoken team members.  If everyone has an opportunity to put their thoughts down on paper first, an adept group facilitator, BA or CPM mentioned above could synthesize and present everyone's results for consideration. 

The Cost of Re-work
Your group requirements gathering meetings (yes - there's likely a plural on meetings) will highlight how different people's perspectives can be, but this is vital so that important features of your application are not overlooked.  This might be comparable to waking up one morning, going into your spare bathroom and suddenly realizing that you need six light bulbs in a space where you only have two sockets.  You can't understand why you didn't see this before, but you realize now that you absolutely have to have six sockets.  'If only I had talked to my wife/husband/kids...' you think to yourself as you're faced with the prospect of cutting into the drywall, adding more wire and electrical components (if it's even possible), patching the wall, etc. - all the while wondering when you're going to have the time and money to do this.  

On a similar note, your software application may have to be 'reworked' significantly and at great cost and effort if you initially overlook a key feature or function.  Two heads or more are truly better than one when it comes to application requirements gathering, as time consuming as that may seem....

As you identify your key end-users and begin gathering your application requirements, you should be ready in about a week for our next installment in this blog series.  In Exercise 3, we will build on what we have covered today and you should be ready to work through a spreadsheet tool to help you calculate whether you should lean more towards whether 'To Build or Buy' - - and how strongly.    


Have you been through a simple or complex requirements gathering process?  How did it work for you? Our readers would love to hear your experiences and insights!

1SSQ Staff, Software Quality News, Beating the odds: Managing a successful software project:,289202,sid92_gci1511496,00.html


Posted by: Wendy Kostiuk AT 02:14 pm   |  Permalink   |  0 Comments  |  Email
Tuesday, July 27 2010

Have you ever struggled with this age-old I.T. question: whether to build a custom software application or to buy one off the shelf?  This blog series was motivated by my recent conversations with both current and prospective customers and it really hit me - we are all facing this issue more and more often as we increasingly rely on technology to enhance productivity, to connect our computer and web-based applications, and to work smarter overall.  

As we are driven 'to do more with less', our new productivity tools need to be effective, and we need to have done our due diligence investigating all the options, otherwise, we may end up duplicating our work effort down the road, having spent valuable time and resources on a solution that is simply not viable - and no one can afford that.

My aim with this blog series is to provide food for thought for anyone, in any position or industry, seeking a software solution to improve productivity.  I have referenced the articles and tools where I utilized them, should you desire to dig deeper than I have.  Please note, we neither endorse nor profit from any recommendations made.

These exercises will require some investment on your part, but doing some homework now could save you a bundle of time, money and effort (not to mention aspirin and antacids) in the long run. 

In this blog series I will cover exercises on the topics below, so check back tomorrow for the first 2 Exercises to get you started on whether 'To Build or Buy'. 

  1. The importance of knowing your key end-users.
  2. Having a clear picture of your end application in mind - why this is an absolute must!
  3. Estimating whether 'To Build or Buy' a worksheet tool you can use. 
  4. Failure rate of custom application development and things to consider before you leap.
  5. Understanding your key business drivers in the decision making process.
  6. Off the shelf applications and key points to ponder.
  7. Decision time.
What are your experiences purchasing off the shelf solutions or having custom applications built for you?  We'd like to hear from you and any of your words of wisdom or about pitfalls you encountered. 
Posted by: Wendy Kostiuk AT 05:08 pm   |  Permalink   |  0 Comments  |  Email
Monday, May 31 2010

I had the pleasure of recently attending and presenting CompTracker on May 13 during the National CSRT Conference in St. John's Newfoundland.  What a charming city!  But more on that later....

Following a whirlwind 65+ minute presentation (sorry for going over Ted!) and during which time some very thoughtful and insightful questions were asked (thank you Mary, Emily and many others!!)  I was left with a profound impression that was reinforced during the rest of my stay in St. John's.  From the young, very poised students I met from NBCC to interesting Therapists, Institution Clinical Coordinators and Administrators, and even the Conference Organizers all from many Canadian locales - I was blown away by the dedication, caring and sense of purpose in most everyone I was fortunate enough to talk with.  I started out my life as a very preemie baby in the devoted, knowledgeable and loving care of this group of professionals and it reassures me, given their depth of commitment, to think that I may someday see them towards the end of my journey - but that's hopefully a long way's away yet!

And to Faculty at the College of the North Atlantic - many thanks for a warm, St. John's Welcome.  I feel I have a bit better appreciation for the wisdom you are endeavouring to impart in your young students and I would like to continue learning more about your challenges and opportunities.  I also look forward to a day when I can return and spend much more time basking in the glow of a culturally rich, warm group of people and celebrating and exploring some of our founding geological and historical roots. 

So long for now from Edmonton, and seriously - thanks for the fish!

Posted by: Wendy Kostiuk AT 10:00 am   |  Permalink   |  0 Comments  |  Email
Monday, April 12 2010
We here at Great Big Solutions have been working on what the next greatest thing for CompTracker might be.  There has been a lot of talk about the iPad coming out this quarter, which has raised a few questions about whether we should support it, and when.

There is no question that this device is cool.  (Come on, it's Apple!)  The other great thing about this device is the unquestionable practicality it would have for our users.  Although a bit more expensive than an iPod Touch, it would be a nice balance between the benefits of data tracking on the website (more screen size to take advantage of, keyboard) and the mobility of the handheld (signatures, entering preceptors on the fly, offline entry).  What we're not sure about are the mixed reviews of the longevity of this device for the mass consumer market: If LOTS of people don't buy it, Apple will (probably) not continue to manufacture and sell it.

Because of that big question, and the reasons FOR supporting this new device, we will implement in stages.  As of right now, our iPhone/iPod Touch app will work on the iPad interface as it is.  Into the near future, as the device gains traction in the marketplace, we will expand the CompTracker iPad interface to be its own, and build the application to take advantage of the enlarged screen size we will now have to work with.

The Truth About iPad (by Jay Baer)
This reviewer has some great insights into the first release of the iPad, and you'll notice in the comments section that many people have noted the usefulness in Health Sciences fields, such as: "I can absolutely see education (and mobile medicine) potential..."

What are your thoughts?
Posted by: Kathryn AT 05:00 pm   |  Permalink   |  0 Comments  |  Email

Add to favorites
 Educator Newsletter 

(We do not abuse the privilege of having your email address in our system, and will not be providing it to ANYONE else for ANY reason, without explicit permission from you.)

MORE than custom software development
Great Big Solutions Ltd.
Edmonton, AB
Phone: 1-866-432-3280