mobile assessments: opportunity, challenges, and best ... · on a mobile device will become much...

cebglobal.com

1

MOBILE ASSESSMENTS

© 2013-2016 CEB. All rights reserved. MB-082013-UKeng-GLO-MDetail about CEB Inc. and its subsidiaries can be found at cebglobal.com/offices

Mobile Assessments: Opportunity, Challenges, and Best Practices

Over the last two decades changes in information technology have driven two substantial shifts in employee assessment. The introduction of the personal computer meant that assessments previously only available in pencil and paper format could be administered via computers, thereby simplifying administration and scoring. Next, the emergence of the Internet precipitated a second major shift by enabling employers to deliver assessments remotely to candidates using Unproctored Internet Testing (UIT). Thus, the need for expensive and time-consuming on-site candidate assessment sessions was dramatically reduced.

The recent explosive growth in mobile Internet use heralds another shift. Mobile Internet networks and devices have matured to the point now where it is becoming practical to deliver employee assessments through such devices. Over the next few years, a rapidly increasing proportion of employee assessments will likely be conducted on mobile devices. According to Smith (2012), 33% of Internet-based cell phone users do most of their Internet browsing on their phones as opposed to on a computer, and 60% use their phones to do web-browsing of some kind. With such a large proportion of people accessing the Internet primarily through their phones, employee assessment on a mobile device will become much more in demand. Employers have noticed this trend and are beginning to demand mobile device capabilities for many aspects of business, including hiring assessments. In 2011, only 9% of human resources (HR) practitioners reported that candidates were requesting to take assessments via mobile devices. In 2012, 19% of practitioners reported receivingsuch requests (Fallaw, Kantrowitz & Dawson, 2012a). Of course, such a significant alteration of the nature of the assessment delivery media raises a number of questions for providers of assessment content, administrators and users of assessment content, and assessment candidates. In this white paper, we explore and share our current perspectives on some of the most important issues and questions.

Mobile Assessments: Opportunity, Challenges, and Best Practices

MOBILE ASSESSMENTS


cebglobal.com

2

Why Mobile Assessment?

A key driver of the move towards mobile assessment is the eventual ubiquitous nature of mobile Internet: candidates and organizations are likely to take for granted that it will be available. In other words, because other online services are and are moving towards being delivered in mobile format, users will expect that online assessments can be accessed through mobile devices. This expectation of mobile availability will be especially strong among younger applicants in the near future who will always have known mobile Internet to be ubiquitous and are likely to find products delivered exclusively through wired Internet devices constraining and unintuitive. This is particularly true in fast-growing mobile usage countries such as China, where 35% of HR practitioners reported their candidates are requesting assessments via mobile, compared to 19% globally (Fallaw, Kantrowitz & Dawson, 2012b).

Mobile assessment also offers direct benefits for organizations. This testing method boasts many of the same benefits as UIT, such as getting closer to candidates, potentially reaching larger and more diverse applicant pools and lowering costs. In other respects mobile assessment may go further. For example, traditional UIT promised that candidates could “apply anytime, anywhere.” However, until recently most candidates were still using a wired Internet connection. Mobile assessment extends the ways in which organizations can be in touch with the potential applicant pool as well as offering potential benefits regarding branding and organizational image.

Looking Forward – Growth in Mobile Usage

While the total volume of mobile phone sales in the Asia Pacific region far outstrips any other region (see Chart 1), the overwhelming majority of these sales are of feature phones that would not allow for mobile testing deployment. In markets such as North America and Europe, however, smartphones already make up more than half of the volume of mobile phone sales, with North America verging on almost three quarters. This trend is more clearly illustrated in Chart 2 (Mobile Megatrends, May 2012), which shows how feature phones still massively outnumber smart phones in the Asia Pacific, Africa Middle East, and Latin America region.

Chart 1: Mobile sales volumes per region in 2011 (millions of units, estimated)

Mobile Internet networks and devices have matured to the point now where it is becoming practical to deliver employee assessments through such devices.

Smartphones Feature phones

Europe

Asia Middle East

North America

Latin America

145

153

37

109

28

603

149

172

63

139

Source: Mobile Megatrends Report (Vision Mobile, May 2012)

Asia Pacific

MOBILE ASSESSMENTS


cebglobal.com

3

What do these sales trends suggest for mobile assessment? The markets that show a high penetration of smartphones (i.e. North America and Europe) are the same markets where scientific assessments are most widely used by employers. This overlap in mobile technology readiness and employee assessment use strongly suggests that the markets that will see widespread serious use of mobile assessment are North America and Europe. However, this is not to say that other countries will not see any serious use of mobile assessment, as the absolute volume of smartphones in Asia is actually the same as in North America and Europe. It is likely that some areas in Asia with high smartphone penetration may make use of mobile assessments for their job candidates as well.

Chart 2: Market penetration of smartphones vs. feature phones across global regions

Chart 3: Reported Phone Ownership, by Type

63%

37%

North America

51%49%

Europe Asia Pacific

81%

19%

82%

18%

Africa Middle East

83%

17%

Latin America

Source: Mobile Megatrends Report (Vision Mobile, May 2012)

Smartphones Feature phones

31%

38%

47%

39%

US

30%

45%

57%

43%

UK

27%

38%

45%

39%

France

18%23%

57%

50%

Germany

6%

17%21%

18%

Japan

Smartphone Jan-Feb 2011 Feature phone Jan-Feb 2011 Smartphone Sep-Oct 2011 Feature phone Sep-Oct 2011

Source: US Consumers Shifting from Feature Phones to Smartphones (Google, January 2012)

MOBILE ASSESSMENTS


cebglobal.com

4

Benefits to Organizations

There are many potential benefits to mobile assessment testing for organizations. One of the greatest benefits is the ability to reach candidates wherever they may be located. The ITU (2011) estimated that 1.2 billion people have mobile-broadband subscriptions in the world. Furthermore, comScore (2011) reported that in the US and Western Europe, 90% of mobile users have a phone that can access the Internet. Even accounting for the fact that some of those individuals are using feature phones, this is still massive internet access. With that much ubiquity, mobile assessment testing can have far-reaching consequences in the way of garnering a wider range of applicants, a more diverse demographic pool, and a more variegated group of media users.

First, mobile testing is an excellent way to increase a client’s ability to appeal to a more diverse demographic pool. Companies are constantly trying to attract young, new talent, and according to Martin (2005), flexibility is a key aspect in attracting Generation Y applicants. Having flexible, cutting edge products to represent the company’s assessment process is a great second step (after a cutting-edge recruiting site) toward attracting a more diverse range of age groups. Furthermore, being able to quickly deploy and score candidates allows employers to hire the best candidates available before other employers with less efficient hiring practices have the opportunity to make an offer.

Mobile testing also boasts all of the typical benefits associated with an unproctored testing scenario, and because there is evidence that validities tend to be similar across proctored and unproctored conditions, these benefits are worth pursuing (Beaty, Borneman, Kantrowitz, Drasgow, & Grauer, 2011). The mobile testing process is efficient and cost saving because candidates can take the assessment at their own convenience. There are significant cost savings associated with no longer needing an office for assessment and also no longer needing test proctors, computers, or IT maintenance that would typically be required for a proctored assessment test. In fact, a study by Lahti & Dekoekkoek (2006) showed that savings associated with UIT outweigh potential loss in validity for high-volume programs. Another UIT benefit shared with mobile testing is that candidates can take the test at any time, from any location. Along these same lines, the flexibility associated with unproctored testing will allow the client to attract more applicants due to the ability to test at any hour of the day from anywhere around the world. Finally, the use of mobile assessment can potentially enhance the consistency of testing in that all candidates receive the same instructions, timing, and scoring (Tippins et al., 2006; Beaty, Dawson, Fallaw, & Kantrowitz, 2009).

Another benefit to mobile testing is that it allows those who may not own laptops to still have access to an assessment test from any location. Because Internet-capable mobile devices are generally less expensive and tend to have many of the same capabilities of a laptop or desktop computer, many people opt to have only a phone. Zickuhr and Smith (2012) found that minority groups often had greater utilization of mobile devices, particularly smartphones, and were more likely to utilize their mobile devices for accessing the Internet. Thus, mobile assessment would make testing available to a wider variety of people, including those who do not own computers. A mobile option also allows for immediate access to the test. There is no need to wait to apply for a position, thus making it easier for an applicant to initially engage in the assessment process.

One of the greatest benefits is the ability to reach candidates wherever they may be located.

MOBILE ASSESSMENTS


cebglobal.com

5

Finally, and possibly most importantly, having a mobile assessment test option ultimately gives clients more flexibility in their offerings. A client with a mobile offering now has the ability to reach candidates who don’t have laptops, appeal to younger generations looking for cutting edge companies to join, and provide immediate, portable access to the assessment process. This type of flexibility can help to ensure a client reaches the widest applicant pool possible in the search for talent.

Defining “Mobile”

We define mobile assessment as more than simply the type of device being used. Specifically, features of mobile devices that might be perceived as limiting and problematic should be considered, as well as the social and contextual usage variables that would be expected to vary more dramatically with a truly mobile device.

What is a Mobile Device, Really?

A mobile device is defined by its portability and ease of use such that the transportability becomes the driver of where (physically, geographically) the device is used and what can be done with it. Depending on the environmental factors in the physical space (i.e. ambient noise, desired versus undesired distractions, focused versus casual attention, stakes of accuracy and attentiveness to task, etc.), making the device portable in the greatest number of environments typically means it can be carried on one’s person, preferably within clothing pockets or accessories that are nearly always “with” a person (e.g. purses, briefcases, folios).

While a desktop computer could be moved and would therefore be portable, the amount of effort required is substantial relative to that offered by a laptop computer, tablet pc, or handheld mobile phone. Essentially, mobile devices are best thought of as relatively light weight (under 1kg/ 2 pounds) and hand-held (Wikipedia, 2012). These criteria would suggest that tablet PCs and mobile phones are mobile devices. While small laptop computers (netbooks/ultrabooks) might be considered, the need for a surface to rest the device rather than holding it precludes it from the mobile device definition here.

Another possible defining characteristic of a mobile device is its actual computational power. Up until recently, the processors utilized in smartphones and tablets were unable to perform much of the visual presentation of information on screen comparably to a laptop computer. Laptops often have either a dedicated video card or special video processing capabilities included with the primary cpu. However, since 2010 the newest generations of smartphones and tablets have video, graphics, and screen capabilities that allow them to display video, play high fidelity games, and more with high levels of graphics and realism. While not true until the past few years, it is fair to say that mobile devices now possess the necessary computational power and software to perform tasks that have previously been the sole domain of larger (and less mobile) computational devices.

In the context of psychological employment assessment, much of the current debate surrounding mobile testing mirrors that of unproctored Internet testing (UIT) from the last decade. Specifically, issues with the move from a physical paper medium to a computer monitor or screen (first addressed in computer-based testing (CBT), and then magnified with the advent of UIT); cheating; equivalence of assessment; and equal access questions.

MOBILE ASSESSMENTS


cebglobal.com

6

Apparent Challenges to Mobile Testing

What could inhibit the use of mobile assessment? One such area concerns whether assessments delivered through mobile devices will generate scores that are psychometrically equivalent to assessments delivered in traditional online formats. Any substantial differences that are attributable to the nature of the medium could make the introduction of mobile assessment problematic. A current challenge, however, is that there is relatively little rigorous research available to address and allay such concerns, which may make organizations move cautiously into mobile assessment.

Furthermore, at this stage it is likely that there is variability across segments of applicant population in terms of access to mobile devices capable of running assessments. There could be concerns around the extent to which mobile assessment could introduce unfairness into recruitment processes. Another potential barrier is that some may see mobile Internet applications as oriented to fun and leisure rather than serious work activities. Some organizations may perceive mobile assessment as a potential threat to their image and may take a cautious approach in making use of the administration method. Similarly, to the extent that some candidates see mobile Internet as fun-oriented, they may be disinclined to take part in an assessment process that makes use of mobile assessment components for highstakes decisions. Furthermore, participants may not understand the consequences of their participation or may take the process too lightly because they, too, see it as fun and leisurely when administered by mobile device.

It is also likely that some assessment types will be more and less suitable for delivery through mobile Internet devices. Of the current set of assessment product categories, products such as interest inventories and personality questionnaires may more easily lend themselves to mobile delivery, whereas maximum performance assessments (such as timed ability tests) are probably less likely to be suitable. For instance, a study by Doverspike, Arthur, Taylor, & Carr (2012) reported score decrements associated with General Mental Ability tests delivered on a mobile device. On the other hand, this study also reported that though completion time was longer for test takers on a mobile device, personality scale scores remained constant across administration mode. Furthermore, in our 2012 study comparing a mobile administration of a personality test to a PC administration, results showed that there was adequate construct equivalence and comparable reliability across both modes of administration. These types of preliminary findings are evidence that the types of tests administered on mobile devices should be carefully considered and researched before being deployed into a commercial setting, but there is evidence to indicate that the practice is feasible with regard to certain types of tests.

Another consideration is the variable types of mobile devices that can be used for assessment testing. There is a fairly large distinction between the size and usability of a tablet versus a phone. Current tablets typically boast a screen size between 7 and 13 inches featuring a 1280 x 800 HD display or better (Kingsley-Hughes, 2012). Smartphones, on the other hand, have a maximum screen size of 4.5 inches with an average display resolution of 480 x 800 (Hisona, 2012). These differences between the devices may cause differences in the ability to take certain mobile assessments. Dexterity and visual acuity become important factors in determining the usability of each device for assessment purposes, and certain tests, such as simulations, may be more amenable to administration on a tablet versus a phone because of the larger screen size and increased screen resolution.

Some types of assessment are simply better suited to delivery via mobile assessment than others at this stage in mobile assessment development and delivery. While advances in compatibility between mobile devices and desktop PCs continue, there are some current limitations based on assessment type.

MOBILE ASSESSMENTS


cebglobal.com

7

Specific Test Considerations

Some types of assessment are simply better suited to delivery via mobile assessment than others at this stage in mobile assessment development and delivery. While advances in compatibility between mobile devices and desktop PCs continue, there are some current limitations based on assessment type.

For example, a short question with a True or False response format would be much easier for a mobile device user to read than a lengthy verbal comprehension passage. Beyond character size and word-count limitations, additional issues that dictate the mobile experience include: simulations using animation, video, or images; speeded tests; and power/knowledge tests.

These issues deal with the fundamental issues of visual capacity and related interactions between user and device, and both bandwidth and lag in mobile network connections. This does not necessarily rule out certain types of assessment content from mobile delivery, though it does suggest that more modifications would be required in some cases. In general, it is not the type of test (i.e. personality, cognitive, biodata, situational judgment, simulation) that is mobile-constrained but the features of its current delivery and operationalization that determine what level of modification is needed given current requirements for acceptable mobile user experience.

Another concern with mobile devices is the effect of potentially disruptive environments. A study by Nagata (2003) examined the impact of disruptions while completing a task on a mobile device compared to a desktop computer. Interruptions on the mobile device resulted in longer task completion times compared to desktop users. Nguyen and Strazzulla (2012) found however that reading comprehension did not appear to be impacted by either the smaller screen size of mobile devices or the level of noise disruption in the physical environment. While both studies utilized small samples, the initial conclusions suggests that the same guidance given to CBT and UIT test takers follows here: avoid or otherwise minimize disruptions. If disruptions are experienced, it would appear to have a greater impact on speeded tests and less so on the ability to accurately determine the best or correct answer.

Because both the potential for distractions and the features of a mobile device require special consideration, testing on small mobile devices, such as phones, may be best suited to short, one to two minute assessments. The mobility and the range of settings in which people use mobile phones likely increases opportunities for distractions, disconnections, and collusion when taking an assessment. Alternatively, using small mobile devices for workplace performance appraisals may prove beneficial. Because of the constant presence of mobile phones, these short but frequent performance appraisals may have more positive results than using longer, infrequent appraisals, giving employers more reliable and frequent snapshots of employee performance over the long-term and also providing employees with more opportunities to receive developmental feedback (Beer, 1981). However, more research is needed in order to evaluate whether small mobile devices are best suited to this type of assessment format as opposed to traditional pre-employment testing.

Finally, given the focus on psychometric equivalence with the advent of first CBT and then UIT, many of the same considerations apply to mobile assessment. Specifically, does administering the same content on a mobile device alter the measurement properties of the assessment? While continued research is needed, a few studies have examined these types of questions.

MOBILE ASSESSMENTS


cebglobal.com

8

In a 2008 research article, Ramachandran, Lundy, and Coons demonstrated that assessments delivered on tablet PCs with touchscreens yielded equivalent results when compared to a paper and pencil administration of the same assessment. In addition, it has been shown by one test publisher that there are no differences in test scores between mobile and non-mobile devices (Scott, Morelli, Illingworth, and Lance, 2012). While results are encouraging, more research needs to verify these results and understanding some of the conditions under which this holds true.

What Do We Know?

So far the research on mobile assessment testing has been scant. The vast majority of research in the field focuses on mobile learning and testing in the classroom as well as unproctored Internet testing (UIT). We can, however, glean a few lessons from the research that has been done in the areas of mobile learning and UIT and apply it to mobile testing for hiring assessment purposes.

Focusing on research in education, Coulby, Hennessey, Davies, and Fuller (2011) found that students were widely receptive to the idea of completing academic tests on such devices. The main complaint that was posed by students was that inputting free text was too labor intensive. Another study by Triantafillou, Georgiadou, and Economides (2008) looked at the use of a computerized adaptive test on a mobile device and found similar student reactions. Students indicated that the use of a mobile device was both “interesting and attractive” (p.1328). The main complaint that was raised was that the small screen size caused images to be unclear. Both of these studies indicate that, overall, younger generations seem to be receptive to the use of mobile devices for assessment.

However, the types of tests used and the design of the tests will need careful consideration when being administered on a mobile device. In the world of work, UIT has generally become a widely accepted and widely utilized method for assessing job candidates (Tippins et al., 2006). In general, UIT boasts a lot of benefits over traditional paper-and-pencil testing. According to Naglieri et al. (2004) Internet testing is known for being “better, faster, and cheaper” (p. 151). However, this method is not without complications. Candidate integrity, test security, appropriate accommodations for disabled test takers, and respondents test-taking behaviors all become a concern when dealing with UIT. Therefore, research will be needed to explore ways to mitigate these concerns with mobile device testing as well.

Overall, reactions to UIT have been positive. According to Potosky & Bobko (2004), this is important because research has shown that there can be negative consequences to a selection process if candidates dislike the test administration mode. In their study on UIT, Potosky & Bobko (2004) found that participants enjoyed taking unproctored Internetbased tests more than traditional paper-and-pencil, and furthermore, that participants find Internet-based tests to be faster. On the other hand, some participants feel uncomfortable with an Internet-based mode of administration. For those who are not entirely familiar with the Internet and computers, there may be more skepticism regarding the accuracy of the test parameters. There are other respondent concerns when dealing with UIT as well. For instance, normal test tasking strategies (e.g. calculating math problems on scratch paper) may not be an apparent alternative strategy during the administration of a UIT. Thus, it is likely that mobile device assessment will not be a viable option for those who are unfamiliar with mobile devices or the Internet. However, as mobile devices become more and more ubiquitous, this concern will lessen considerably.

The types of tests used and the design of the tests will need careful consideration when being administered on a mobile device.

MOBILE ASSESSMENTS


cebglobal.com

9

What Do We Need to Know?

Because of the general dearth of research in the field, the jury is still out regarding the use of mobile devices for selection purposes. Thus, there are many research questions that much be answered. First, norms and equivalence testing will need to be conducted to ensure that the mobile test taking experience is similar enough to other methods of testing to be feasible. In order for mobile testing to be appropriate, scores across modes of administration should be invariant. Furthermore, subgroup performance including age, ethnicity, and gender should be similar to what would be expected with normal modes of administration. Finally, for cognitive and timed tests, knowing how scores compare across screen options will be an important consideration in deciding whether these types of tests will be amenable to mobile administration.

Next, reliability and validity studies must be conducted to assess the differences across modes of administration, including both personal computers and mobile devices. Preliminary research by Morelli, Illingworth, Scott, & Lance (2012) showed that scores on a non-cognitive personality and biodata test were invariant across PC, Mac, Phone, and Tablet devices, and thus validity and reliability can be assumed to be equivalent across testing modes. However, Doverspike, Arthur, Taylor, & Carr (2012) reported score decrements associated with General Mental Ability tests delivered on a mobile device. Thus, more research is needed to tease out which tests will hold equivalent validities and reliabilities. Along this same research agenda, criterion related validity studies will be needed to answer the question as to whether validities are similar in a group of mixed PC and mobile test takers.

Also, because of the unproctored, flexible nature of mobile testing, there will be additional threats to validity and reliability that will need to be considered. Knowing how these threats affect the psychometric properties of the test will be important. Finally, given the potential increase in methods for cheating on mobile devices, the question of whether or not cheating deterrents affect scores and validity will need to be addressed.

Another area that will require more research will be test taker reactions. Although existing research based on student reactions appears to be positive (Coulby et al., 2011; Triantafillou et al., 2008), gauging whether or not test takers feel mobile devices and PCs are fair and whether they affect a user’s opportunity to perform on selection assessments will be important. Also, reactions to realistic job previews that have been administered on a mobile device will provide useful feedback for assessing whether candidates intend to apply for jobs and stay in the application process. Analyzing how the mobile delivery of a realistic job preview affects perceptions of the organization will provide useful information regarding the attractiveness and acceptability of the mobile administration.

Accessibility and process issues will also need to be analyzed. For one, knowing whether there is any interest from test takers in completing tests on mobile devices and also knowing what types of settings test takers will be likely to complete mobile tests in will be important. Along these same lines, the way in which the external environment (e.g. economic conditions, testing stakes) influences test scores delivered by mobile devices will need to be examined. Also, with regard to adverse impact, research will be needed on whether mobile testing affects the applicant pool in a way that puts minorities at a disadvantage, and furthermore, the kinds of accommodations that should be made for disabled test takers will need to be addressed.

MOBILE ASSESSMENTS


cebglobal.com

10

Finally, examining testing time differences between mobile devices and PCs will be important, as this could have a significant impact on how the test parameters should be modified to promote the equivalence of the tests across administration mode.

Lastly, test security will be a concern. In 2012, 20% of HR practitioners indicated that mobile testing may encourage cheating, while 56% were unsure if it would (Fallaw et al., 2012a). With the portability and accessibility of mobile devices, cheating could become a larger issue than is now seen with PC and Internet administration of tests. Examining how candidates cheat on tests delivered by mobile devices and what mechanisms could be put in place to reduce exposure of test content will be important. Also, finding effective methods for detecting suspicious test responses and examining testing scores over time (i.e. whether mean scores increase over time due to increased exposure) will be crucial for maintaining reliability and validity of the test content.

Best Practice Recommendations for a Mobile Assessment Programme

With the potential benefits and challenges of implementing mobile testing depicted above, what are the best practices for implementing mobile assessments today? Many of these will feel familiar, as they mimic basic best practices in testing in general, as well as testing in unproctored settings via the Internet. Companies that wish to employ a mobile assessment strategy should consider the following:

1. Remember that just because an assessment can be mobile, doesn’t mean it should be. Consider it carefully.

2. Offer both mobile and traditional Internet-based delivery methods to candidates to ensure equal access to assessments.

3. Minimize reliance on scrolling - Include assessments that include relatively short questions and response options (i.e., reading comprehension paragraphs would be best left to larger screens).

4. Minimize time for assessments – use shorter, potentially computer-adaptive tests to minimize potential distractions

5. Ensure testing platform is accessible via most if not all operating systems on mobile devices

6. Ensure candidate ease of use by following up testing experience with reactions survey

7. Specifically examine the norms, equivalence, and predictive validity of assessments deployed in mobile environments.

8. More broadly examine business outcomes of assessments for both mobile and Internet-based tests

As the move to mobileenabled assessment practices continues, the points listed in this white paper serve as considerations for organizations who wish to provide this convenience of administration to their candidates and HR professionals.

MOBILE ASSESSMENTS


cebglobal.com

11

Conclusion

The Future of Assessment is Mobile

Assessment in the world of work follows technology: first from paper and pencil to computer, then to the Internet and now to mobile devices. As the move to mobile-enabled assessment practices continues, the points listed in this white paper serve as considerations for organizations who wish to provide this convenience of administration to their candidates and HR professionals. While there are areas that still need to be researched, including measurement equivalence of mobile delivery versus traditional Internet-based testing via PCs and applicant reactions to taking assessments in a mobile environment, the pace of change requires organizations to seriously consider the use of such devices in light of their recruiting and selection practices. We provide some best practices recommendations on implementing mobile testing in this paper, and some areas for future research. Mobile devices are replacing personal computers, and organizations who can take advantage of this fact will have major advantages in the war for talent.

MOBILE ASSESSMENTS


cebglobal.com

12

ReferencesBeaty, J., Nye, C., Borneman, M., Kantrowitz, T., Drasgow, F., & Grauer, E. (2011). Proctored versus unproctored Internet tests: Are unproctored noncognitive tests as predictive of job performance? International Journal of Selection and Assessment, 19, 1-10.

Beer. M. (1981). Performance appraisal: dilemmas and possibilities. Organizational Dynamics, 9, 24-36.

Bridgeman, B., Lennon, M.L., and Jackenthal, A. (2003). Effects of Screen Size, Screen Resolution, and Display Rate on Computer-Based Test Performance. Applied Measurement in Education, 16, 191-205.

Coulby, C. Hennessey, S., Davies, N., Fuller, R. (2011). The use of mobile technology for work-based assessment: the student experience. British Journal of Educational Technology, 42, 251-265. Doi:10.1111/ j.1467-8535.2009.01022.x

Doverspike, D., Arthur, W., Taylor, J., Carr, A. (2012). Mobile Mania: Impact of Device Type on Remotely Delivered Assessments. Panel Presentation as Annual SIOP Conference, San Diego, CA.

Dyson, M. C. (2004) How physical text layout affects reading from screen. Retrieved from http:// en.wikipedia.org/wiki/Mobile_device on June 22, 2012.

Dyson, M. C. & Haselgrove, M. (2000). The effects of reading speed and reading patterns on the understanding of text read from screen. Journal of Research in Reading, 23, 210-223.

Fallaw, S. S., Kantrowtitz, T. M., & Dawson, C. R. (2012a). 2012 Global Assessment Trends Report. CEB.

Fallaw, S. S., Kantrowtitz, T. M., & Dawson, C. R. (2012b). 2012 Global Assessment Trends Report: China. CEB.

Google. (January 2012). US Consumers Shifting from Feature Phones to Smartphones. Retrieved from http://www.marketingcharts.com/direct/us-consumers-shifting-from-feature-phones-tosmartphones-20902/

Hisona, H. (2012). Nokia Lumia 920 Windows phone 8 smartphone unveiled, specs inside. [Review of the Nokia Lumia 920, produced by Nokia, 2012]. Retrieved from nokia-lumia-920-windows-phone-8-smartphone-unveiled-specs-inside

ITU. (2011). Annual Report. Retrieved from http://www.itu.int/ITU-D/ict/facts/2011/material/ICTFactsFigures2011.pdf

Kingsley-Hughes, A. (2012). Google Nexus 7 tablet specs revealed. [Review of the Google Nexus tablet,produced by Google, 2012]. Retrieved from http://www.zdnet.com/blog /hardware/google-nexus-7-tablet-specs-revealed/21038

Lahti, K., & Dekoekkoek, P. (2006). Unproctored Internet Testing: What do the Data Say? Presentation at Annual SIOP Conference, Dallas, TX.

MOBILE ASSESSMENTS


cebglobal.com

13

Martin, C. (2005). From high maintenance to high productivity: What managers need to know about Generation Y. Industrial and Commercial Training, 37, 39 – 44.

Morelli, N., Illingworth, A., Scott, J., & Lance, C. (2012). Are Internet Based, Unproctored Assessments on Mobile and Non-Mobile Devices Equivalent? Panel Presentation at the Annual SIOP Conference, San Diego, CA.

Nagata, S. F., and TNO Human Factors. (2003). Multitasking and Interruptions During Mobile Web Tasks. Proceedings of the Human Factors and Ergonomics Society.

Naglieri, J., Drasgow, F., Schmit, M., Handler, L., Prifitera, A., Margolis, A., Velasquez, R. (2004). Psychological testing on the Internet: new problems, old issues. American Psychologist, 59, 150-162.

Nguyen, T., and Strazzulla, D. (2012). Investigating the Effect of Different Form Factors on Reading Ability. Unpublished Technical Report. Palo Alto, CA: Stanford University.

Potosky, D., & Bobko, P. (2004). Selection testing via the Internet: practical considerations and exploratory empirical findings. Personnel Psychology, 57, 1003-1034.

Radwanick, S. (2011). Mobile Year in Review [White Paper]. Retrieved [6.25.12] from http:// www. comscore.com/Press_Events/Presentations_Whitepapers/2011/2010_ Mobile_Year_in_Review.

Ramachandran, S., Lundy, J.J, and Coons, S.J. (2008). Testing the measurement equivalence of paper and touch-screen versions of the EQ-5D visual analog scale (EQ VAS). Quality of Life Research, 17, 1117-1120.

Scott, J. C., Morelli, N., Illingworth, A. J., and Lance, C. E. (2012). Are Internet-Based Unproctored Assessments on Mobile and Nonmobile Devices Equivalent? In J. C. Scott (Chair), Chasing the Tortoise: Zeno’s Paradox in Technology-Based Assessment. Symposium presented at the annual conference for the Society of Industrial and Organizational Psychology, San Diego, CA.

Inceoglu, I., and Affourtit, M. (2012). Good on the go? A look at iPQ (OPQ32r mobile) data. United Kingdom: CEB.

Smith, A. (2012). Cell Internet Use 2012. Retrieved from http://www.pewInternet.org/Reports/ 2012/ Cell-Internet-Use-2012/Key-Findings.aspx

Tippins, N., Beaty, J., Drasgow, F., Wade, M., Pearlman, K., Segall, D., Shpherd, W. (2006). Unproctored Internet testing in employment settings. Personnel Psychology, 56, 189-225.

Triantafillou, E., Georgiadou, E., Economides, A. (2008). The design and evaluation of a computerized adaptive test on mobile devices. Computers and Education, 50, 1319-1330.

Triantafillou, E., Georgiadou, E., Economides, A. (2008). CAT-MD: Computerized adaptive testing on mobile devices. International Journal of Web-Based Learning and Teaching Technologies, 3, 13-20.

Vision Mobile (May 2012). Mobile Megatrends 2012. Accessed from http://www. visionmobile.com/product/mobile-megatrends/ on June 19th 2012.

Wikipedia (2012). Mobile Device. Retrieved June 22, 2012 from http://en.wikipedia.org/wiki/Mobile_device.

Zickuhr, K., and Smith, A. (2012). Digital differences. Pew Internet and American Life. Retrieved June 22, 2012 from: http://www.pewinternet.org/~/media//Files/Reports/2012/PIP_Digital differences_041312.pdf

Contact Us to Learn More

cebglobal.com/assess/uk

www.cebglobal.com/assess/uk

mobile assessments: opportunity, challenges, and best ... · on a mobile device will become much...

Documents