Are you ready? Assessing Digital Competencies for Online Learning via the General Technology Confidence and Use (GTCU) Instrument

Maurice DiGiuseppe (maurice.digiuseppe@uoit.ca), Roland vanOostveen (roland.vanoostveen@uoit.ca),Elizabeth Childs (elizabeth.childs@royalroads.ca), Todd Blayone (todd.blayone@uoit.ca), and

Wendy Barber (wendy.barber@uoit.ca)

Abstract: This brief paper reports on one part of a case study of a fully online course conducted in the Faculty of Education of a mid-sized university in the province of Ontario, Canada. This course—part of the faculty’s Bachelor of Arts in Educational Studies and Digital Technology (BA-ESDT) program—was focused on Problem-Based Learning (PBL); included both synchronous and asynchronous components; employed the Fully Online Learning Community (FOLC) learning model; and required its learners to complete an online version of the General Technological Competence and Use (GTCU) survey, both near the beginning (pre) and end (post) of the course, to assess learners’ digital competency for fully online learning. This paper reports specifically on some preliminary pre-post GTCU survey results, especially in terms of the potential usefulness—for learners, faculty, and program—of the instrument’s just-in-time availability and informative graphical displays.

Introduction

Post-secondary learners around the world are increasingly enrolling in fully online learning programs—programs in which all teaching and learning occurs online (Allen & Seaman, 2014; Murphy, Gallagher, Krumm, Mislevy, & Hafter, 2014). In general, online courses are offered in one of the following configurations: fully synchronous (involving activities within online virtual classrooms only; videoconferencing), asynchronous (involving only blog/discussion board postings, etc., but no virtual classrooms), or hybrid (involving a combination of synchronous and asynchronous modalities) (Hrastinski, 2008). In each case, learners (and instructors) are required to have ready access to: suitable devices, such as desktops, laptops, tablets or other mobile devices, headphones, and webcams; strong, stable Internet connections; and appropriate, distraction-free environments that facilitate online course participation. Furthermore, to be successful, they require adequate competency and skill in the use of online learning systems—minimum levels of competency they may not know they possess. Thus, learners (and faculty) should be afforded timely opportunities to assess their digital competencies, so that they may engage in more highly focused remediation, whether on their own or with assistance, if/when required. The online General Technological Competence and Use (GTCU) survey application attempts to provide such a service.

The GTCU Instrument/Framework

In this study, we employed an online version of the GTCU survey instrument (EILab, 2017), adapted from the instrument originally developed by Desjardins (2005). This instrument is based on the premise that interactions between computer-based technologies and human users may be categorized into four types of interactions, namely, Technical Interactions (T) (in which users interact with the basic operational functions of digital devices), Social Interactions (S) (in which users interact with others through digital devices), Informational interactions (I) (in which users interact with information through digital devices), and Computational/Epistemological interactions (C) (in which users interact creatively with data through digital devices) (Desjardins, 2005).

In this study, the GTCU instrument was used to assess learners’ use of a large variety of computer-based technologies (laptops, smartphones, tablets, wearables, etc.). The GTCU questionnaire includes 26 items in total (5 T items, 7 S items, 7 I items, 7 C items), each assessed via three Likert scale indicators, one for assessing “Importance of Use”, one for “Frequency of Use” and the other to assess “Confidence of Use” (Table 1). Please note that, for reasons of brevity, we will not be reporting on Importance of Use in this short paper.

Table 1. Likert scale used to measure importance, frequency, and confidence of use

Importance of use Frequency of use Confidence of use1 = not important2 = somewhat important3 = very important4 = indispensable

1 = never2 = few times a year3 = few times a month4 = few times a week5 = daily

1 = do not know how to use2 = not confident, require assistance to use3 = confident, can solve some problems4 = quite confident, can use with no assistance5 = very confident, can teach others how to use

Furthermore, in order to assess competency in each of the four types of interaction (T, S, I, C), the GTCU application categorizes the technologies included in the instrument according to interaction type, and aggregates user inputs accordingly. For example, the GTCU application considers technologies such as email, word processing, and audio recording editing as indirect indicators of underlying general operational competencies, and, as such, treats these technologies as Technical Interactions (T). Further, the GTCU system treats texting, sharing documents, and social media as Social Interactions (S); searching for movies, searching for articles, and searching for music as Informational Interactions (I); and sorting data, creating concept maps, and performing complex calculations as Computational Interactions (c).

The Fully Online Learning Community (FOLC) Environment

In this study, an online version of the GTCU instrument was employed near the beginning (pre) and end (post) ofan online course focused on Problem-Based Learning (PBL), which was offered in 2015, within the faculty ofeducation of a mid-sized university in Ontario, Canada—part of the faculty’s Bachelor of Arts in Educational Studies and Digital Technology (BA-ESDT) program. This course included both synchronous and asynchronous (hybrid) components, and employed the Fully Online Learning Community (FOLC) model of education (Figure 1) (Blayone,vanOostveen, Barber, DiGiuseppe, & Childs, 2017; van Oostveen, DiGiuseppe, Barber, Blayone, & Childs, 2016;DiGiuseppe, VanOostveen, & Petrarca, 2015). In general, the FOLC Model integrates elements of morefoundational theories guiding practice in distance and online education, including the Theory of TransactionalDistance (TTD) (Moore, 1993) and the Community of Inquiry (CoI) framework (Garrison, Anderson, & Archer,2000).

Figure 1: Fully Online Learning Community (FOLC) model

The CoI framework, in particular, recognizes three presences essential to supporting distance education: Social Presence, Teaching Presence, and Cognitive Presence. Additionally, the FOLC model is grounded in the GTCU framework (involving T, S, I, and C interactions), which itself builds on the CoI model, and “considers that a technology object serves as an interface between the user and: 1) other users, 2) stored information, and 3) information processing tools or software” (Desjardins, 2014, para. 1). Thus, within the FOLC Model, Collaborative

Learning emerges through the integration of Social Presence and Cognitive Presence within a Digital Space created by instructors, teaching assistants, and learners acting as co-creators of the learning environment (Figure 1).

In this paper, we report on some preliminary pre-post GTCU survey results, focusing on the potential usefulness of the instrument for assessing learner digital competency within an undergraduate course modelled on the FOLC framework.

MethodsContextThis paper reports on only one component of a much broader mixed methods (qualitative/quantitative) case study of an undergraduate course on PBL in Spring Term (May 4-June 29) 2015, in a mid-sized university in Ontario, Canada. The broader study addressed the research question: What are the effects of learning activities within a Problem Based Learning (PBL) course on learner conceptual development of “learning,” and technical competency and use of digital technologies? Note that this course was about PBL and provided opportunities to learn about PBL by engaging in PBL activities involving the design, creation, testing, and critique of Problem Based Learning Objects (PBLOs). This 36-hour (3-credit) course employed a variety of learning strategies, including asynchronous video clip-based flipped classroom scenarios, synchronous group tutorials in Adobe Connect videoconferencing, and online activities, including small group PBL-based knowledge creation activities, using a wide variety of online resources chosen primarily by learners.

A total of 42 (n = 42) learners, 1 (n = 1) instructor, and 1 (n = 1) teaching assistant participated in the study. All learners (n = 42) completed the pre (May, 2015) and post (June, 2015) online GTCU questionnaire.

The GTCU Graphical Display and Relative Competency Index Figure 2 illustrates a typical graphical display generated by the online GTCU application after an individual has completed the GTCU questionnaire and clicks “submit.” These are composite aster plots depicting a user’s (or group’s) competency levels in the use of technologies categorized according to T, S, I, and C types of interactions.

Figure 2: A typical composite GTCU profile

In these graphs, the radius of the white circle is calibrated in relative competency index (RCI) units, with 0 at the circle’s centre, and 10 at the circle’s perimeter. The “thickness” of the four sectors sweeping around the geometric centre of the white circle represent the person’s or cohort’s RCI for each of the four GTCU interactions, with the RCI for T being represented by the sector in the bottom-left quadrant (green) of the circle, S by the top-left sector

(blue), I by the top-right sector (orange), and C by the bottom-right sector (purple). In general, the “thicker” the sector, the higher the RCI, and the greater is the person’s or cohort’s competency in that particular category of the GTCU framework.

The RCI for GTCU graphs is determined by an algorithm that converts inputted frequency of use and confidence of use values for all items in each GTCU interaction category (T, S, I, and C), and converting these into an index ranging from 0 to 10. This is then displayed in the form of the circle graph illustrated in Figure 2.

Furthermore, the GTCU application also generates graphs illustrating users’ (or groups’) competency in the use of individual technologies (Figure 3).

Figure 3: Typical individual GTCU graphical displays

In Figure 3, the size of the translucent circle within each icon is proportional to the individual’s or group’s competency in using that particular technology. These representations may be used to compare competencies between technologies, and changes in competencies over time. It is obvious in Figure 3 that, in general, this individual possessed greater skill in using a computer/laptop than in using a cellphone/smartphone.

Please note that the graphs in Figure 2 and Figure 3, along with the algorithms employed in their creation, are exploratory and preliminary in nature, and continue to be the subject of intense investigation.

Results

Below are two pre/post graphs, Figures 4 and 5, generated by the online GTCU application profiling a single learner in the PBL course.

Figure 4: Sample Pre Learner GTCU profile Figure 5: Sample Post Learner GTCU profile

It is evident in Figure 4 that early in the course, this learner possessed greatest competence in T interactions (green sector) (RCI ≈ 5.5) and S interactions (blue sector) (RCI ≈ 5.3), and the least competence in C interactions (purple sector) (RCI ≈ 3.7) and I interactions (orange sector) (RCI ≈ 1.9). In Figure 5, it can be seen that near the end of the

course, the learner continued to possess greatest competence in T interactions (RCI ≈ 4.5) and S interactions (RCI ≈ 5.4), and relatively lower competence in C interactions (RCI ≈ 4.2) and I interactions (RCI ≈ 3.3). It is interesting to note that the learner’s competence in S, I, and C interactions increased over the course of the term, with competence in I and C interactions increasing the most.

Below are two pre/post graphs, Figures 6 and 7, generated by the online GTCU application profiling the whole PBL cohort, determined by aggregating the entire cohort’s GTCU data.

Figure 6: Sample Pre Cohort GTCU profile Figure 7: Sample Post Cohort GTCU profile

It is evident in Figure 6 that early in the course, the cohort (as a whole) possessed greatest competence in T interactions (RCI ≈ 3.5) and S interactions (RCI ≈ 2.9), and the least competence in I interactions (RCI ≈ 0.8). Note that the gap (blank) between the T and S sectors indicates that the cohort possessed virtually no competence in C type interactions. Furthermore, Figure 7 indicates that by the end of the course, the cohort as a whole continued to possess greatest competence in T interactions (RCI ≈ 3.5) and S interactions (RCI ≈ 4.0), and relatively lower competence in I interactions (RCI ≈ 2.1) and C interactions (RCI ≈ 0.7). It is interesting to note, however, that the cohort’s competence in all four types of interaction increased over the course of the term, and that the cohort’s relative competence in C interactions increased quite dramatically over this period of time.

Conclusions and Implications

The results of this limited component of a broader case study involving a post-secondary course employing the FOLC learning model and the online GTCU instrument demonstrates the potential usefulness of the graphic profiles generated by the GTCU application. The online GTCU instrument may be used by any individual or group, including all of the participants in a course or program (learners, instructors, curriculum developers, etc.), to assess current competence (and changes in competence over time) in using of a large variety of information communication technologies. Since the online GTCU application is freely available on the Internet, individuals and groups may complete it anywhere, and at any time, to obtain just-in-time profiles that may be compared with earlier and later iterations. These results may then be used by learners, instructors, curriculum/program developers, and others to more highly focus remediation efforts and/or professional development activities on specific elements of human-computer interaction addressed in the GTCU instrument. And, since ICT continues to grow and develop, the GTCU instrument continues to evolve in response. New and novel technologies are added to the instrument regularly, as required, and the graphical displays continue to be improved in terms of accuracy and visualization. A number of changes were being considered at the time of writing this article, including changes to the algorithm determining the RCI, and modifications to the form of the graphical display, changes that will be discussed in future publications.

References

Allen, I.E. & Seaman, J. (2014). Grade change: Tracking online education in the United States. Retrieved from http://www.onlinelearningsurvey.com/reports/gradechange.pdf

Blayone, T., vanOostveen, R., Barber, W., DiGiuseppe, M., & Childs, E. (2017). Democratizing digital learning:Theorizing the Fully Online Learning Community model. International Journal of Educational Technologyin Higher Education, 14(1), 13. DOI:10.1186/s41239-017-0051-4

van Oostveen, R., DiGiuseppe, M., Barber, W., Blayone, T. & Childs, E. (2016). New conceptions for digitaltechnology sandboxes: Developing a Fully Online Learning Communities (FOLC) model. Proceedings of EdMedia: World Conference on Educational Media and Technology 2016 (pp. 672-680). Association for the Advancement of Computing in Education (AACE).

DiGiuseppe, M., VanOostveen, R., & Petrarca, D. (2015). Evolving Strategies for Online Learning in Graduate Courses in Education. Proceedings of the Higher Education in Transformation Symposium, March 31-April1, 2015, Dublin, Ireland.

Desjardins, F. (2014). Technology Competency and Use (TCU) Framework. Retrieved from http://eilab.ca/technology-competency/

Desjardins, F. J. (2005). Information and communication technology in education: a competency profile offrancophone secondary school teachers in Ontario. Canadian Journal of Learning and Technology/La Revue Canadienne de L’apprentissage et de La Technologie, 31(1), 27-49.

EiLab (2017). General Technological Competence and Use Questionnaire. Retrieved from http://gtcu.eilab.ca

Garrison, R., Anderson, T., & Archer, (2000). Critical inquiry in text based environment: Computer conferencing in higher education. The Internet and Higher Education, 2(2–3), 87–105.

Hrastinski, S. (2008). Asynchronous & Synchronous e-Learning. Educause Quarterly, 4, 51-55. Retrieved from http://www.educause.edu/ir/library/pdf/eqm0848.pdf

Moore, M.G. (1993). Theory of transactional distance. In Keegan, D. (Ed.) Theoretical Principles of Distance

Education. New York: Routledge.

Murphy, R., Gallagher, L., Krumm, A., Mislevy, J. & Hafter, A. (2014). Research on the use of Khan Academy in schools. Menlo Park, CA: SRI International. Retrieved from https://www.sri.com/sites/default/files/publications/2014-03-07_implementation_briefing.pdf