Full length article
Development and validation of New Media Literacy Scale (NMLS) for
university students
Mustafa Koc a, *
, Esra Barut b
a
Department of Computer Education and Instructional Technologies, Suleyman Demirel University, Isparta, Turkey
b
Department of Computer Education and Instructional Technologies, Anadolu University, Eskisehir, Turkey
a r t i c l e i n f o
Article history:
Received 29 March 2016
Received in revised form
8 June 2016
Accepted 16 June 2016
Available online 23 June 2016
Keywords:
New media literacy
Scale development
University students
Validity
Reliability
a b s t r a c t
Along with the emergence of new media technologies, individuals are now expected not only to consume
but also produce, share and criticize digital contents. Being new media literate, they also need to know
socio-cultural and emotional aspects of new media beyond its technical characteristics. New media literacy
(NML) involves a series of crucial skills needed for living and working in the mediated and
participatory society of 21st century. Although there has been a growing interest in the conceptualization
of NML, the literature lacks a measuring instrument to operationalize NML. This study attempts to fill this
gap by developing and validating a NML scale (NMLS) for university students. The sample included 1226
students at a state university in Turkey. Both exploratory and confirmatory factor analyses as well as item
analyses including internal consistency coefficients, item-total correlations, and item discrimination
powers were conducted to determine construct validity and reliability. Consistent with the adopted
theoretical framework, the 35-item NMLS comprised four factors: Functional Consumption, Critical
Consumption, Functional Prosumption, and Critical Prosumption. Potential applications of NMLS for
measuring students’ new media competencies were discussed within the pedagogical and research
contexts.
© 2016 Elsevier Ltd. All rights reserved.
1. Introduction
The proliferation of advanced technologies in today’s digital era
has facilitated rapid access to new information and delivery of
media messages to crowd of people. However, just the ownership
of technological devices or the ability to use them is not sufficient
for reaching valid and reliable information as well as active
participation in digital media. Individuals need to have a variety of
media or digital literacy skills in order to both consume and (re)
produce functional media content. Such skills are required to survive
in the digital era. In his revised holistic model of digital literacy,
Eshet (2012) defined digital literacy as a multidimensional concept
that comprised technical, cognitive, motoric, sociological, and
emotional aspects. According to Eshet (2012), a digitally-literate
person should have photo-visual skills (understanding graphical/
visual messages), reproduction skills (creating meaningful media
content), branching skills (constructing knowledge from complex
and flexible hypermedia domains), information skills (judging the
accuracy and quality of media content), socio-emotional skills
(communicating and working with others in the cyberspace) and
real-time thinking (multi-tasking or processing different kinds of
multimedia stimuli). Known as a survival kit for the 21st century,
media literacy involves similar skills such as accessing, decoding,
analyzing, evaluating and producing both written and electronic
media content such as text, image, audio, video, and so on (Hobbs &
Jensen, 2009; Zhang, Zhu, & Sang, 2014). It helps people
consciously use media, distinguish and evaluate media content,
critically examine media types, investigate media effects, and
develop alternative media content (Kellner & Share, 2007). In
addition to such skills, it includes the knowledge of how messages
are created, commercialized, and diffused all around the world
(Thoman & Jolls, 2004).
1.1. New media literacy (NML)
The emergence of Internet technologies and mobile communication
tools has transformed old media and introduced the concept
of “new media”. Printed text and analog broadcasting have been
* Corresponding author. Department of Computer Education and Instructional
Technologies, Suleyman Demirel University, Isparta 32260, Turkey.
E-mail address: mustafakoc@sdu.edu.tr (M. Koc).
Contents lists available at ScienceDirect
Computers in Human Behavior
journal homepage: www.elsevier.com/locate/comphumbeh
http://dx.doi.org/10.1016/j.chb.2016.06.035
0747-5632/© 2016 Elsevier Ltd. All rights reserved.
Computers in Human Behavior 63 (2016) 834e843
superseded, if not completely suppressed, by online publication
and digital video. Now, not only authorities (e.g., website owners,
directors) but also ordinary users can (co)create digital media
content to represent their social values, ideologies, politics and so
on. This has blurred the boundaries between the sender and
receiver of information (Oberhelman, 2007) and created a convergence
culture in which old and new media intersect and the powers
of media producers and consumers interact in order to achieve
collective intelligence (Jenkins, 2006). Therefore, by the term new
media, we refer to all technology-based socio-cultural platforms in
which any messages are digitally coded and distributed by any
users. This new media is characterized by digital interactivity,
creative and collective participation, networkability, data manipulation,
modularity, hybridity, and virtuality (Chen, Wu, & Wang,
2011; Lin, Li, Deng, & Lee, 2013). Moreover, it has the feature of
ubiquitous computing environment through which digital messages
can be accessible anywhere and anytime on any digital devices
(e.g., tablets, smart phones).
In today’s world, the most prevalent forms of new media are
Web 2.0 tools. Web 2.0 refers to new version of World Wide Web
platforms that allow for user-generated content, dynamic and light
programming, active contribution, folksonomy (i.e., social tagging),
collective intelligence, and many-to-many web communication
(Butler, 2012; O’Reilly, 2005; Selwyn, 2007). Social networking sites
(e.g., Facebook, Twitter), image and video sharing sites (e.g., YouTube,
Instagram), blogs, wikis, mash-ups, 3D virtual worlds (e.g.,
Second Life) are common examples of these tools. In contrast to
Web 1.0 environments with centralized web sites distributing information
to passive visitors, Web 2.0 platforms promote proactive
participation and contribution, social networks, and diverse interactions
(Maloney, 2007; McLoughlin & Lee, 2007; Selwyn, 2007).
A user can create media messages individually or collaboratively in
the digital form of text, image, video or hybridity of these and share
them with other users, who can also edit or refine these messages
by adding their own touches or tag them by assigning keywords for
useful classification and retrieval purposes. Such a tagging mechanism
makes massive digital resources easily searchable and
sharable for all users, which in turn, enhances collective knowledge
acquisition (Held & Cress, 2009).
The rise of new media has demanded new kind of literacy,
which is called “new media literacy” (NML). Traditionally, media
literacy was viewed as teaching students about media and how to
access and understand its contents (i.e., consuming media). Since
the beginning of the 21st century, new media technologies have
expanded this notion of literacy to include (re)creating media
contents and sharing them with others (i.e., producing media).
Today’s individuals have perpetual contact with information and
each other through both consuming and producing media messages.
Therefore, the new focus in NML is on collective creation of
innovative media content over static content delivery, social
interaction over isolated surfing, and active participation and
engagement over passive reception (Jenkins, 2006; Maloney, 2007).
Furthermore, being able to read and write media needs to be
accompanied by critical and expressive thinking about the politics
of representation (Kellner, 2010). Individuals are now expected to
question potential biases, consequences and power of mediated
messages they create or receive within the various channels of new
media. In contrast to traditional forms of media literacy, NML addresses
some specific limitations or problems (e.g., Poe’s Law,
Streisand Effect). While communicating virtually, individuals
should look for contextual or intentional indicators in order to be
conscious of ironic expressions, parodies or satires and be able to
distinguish them from truthful ones. They need to know that any
digital message may rapidly spread to large group of people on the
cyberspace. In fact, it might be unintentionally publicized to larger
masses when it is tried to be hidden or censored.
1.2. Theoretical framework of NML
Chen et al. (2011) proposed a promising theoretical model to
unpack the notion of NML based on their analysis of historical
evolution of literacy and technical and socio-cultural characteristics
of new media. This was the first attempt to conceptualize NML in
the relevant literature. Basically, they identified NML as two continuums:
(a) from consuming to prosuming media literacy, and (b)
from functional to critical media literacy. Consuming media literacy
refers to competencies to access media messages and employ
media at various levels while prosuming media literacy, on top of
consuming abilities, regards to competencies to produce media
contents and participate in media environments. Using Toffler’s
(1981) concept of prosumer, Chen et al. (2011) argue that a media
prosumer is both a producer and a consumer because he/she usually
produces customized contents through using preexisting media
artifacts, ideas, and benefits of technological tools. In other
words, prosuming aspect is essentially grounded on the consuming
one. On the other hand, expanding Buckingham’s (2003) notion of
functional and critical literacy, Chen et al. (2011) define functional
media literacy as competencies to operate media tools in order to
access and create media messages and understand them at the
textual level whereas they view critical media literacy as abilities to
analyze and judge media messages and understand them at various
contextual levels. As is the case with the first continuum, functional
aspect is integrated as a basis in the critical one. That is, one should
already be familiar with technical or operational characteristics of
new media in order to have a good comprehension of its sociocultural
contexts.
Combining these continuums mentioned above, Chen et al.
(2011) proposed four components of NML: (1) functional
consuming, (2) critical consuming, (3) functional prosuming, and
(4) critical prosuming. A functional consumer is able to access
produced media content and understand its textual meaning. A
critical consumer is able to analyze and interpret socio-cultural,
economic, and political consequences of media content. Hence,
he/she has the ability of questioning media messages in terms of
their purposes, underlying ideologies, social values, and representations
of power. A functional prosumer is able to participate in
production of new media content in various media platforms
whereas a critical prosumer can also convey his/her own beliefs,
negotiate with others’ ideas, and consider expected impacts during
media construction and participation. Chen et al. (2011) associate
functional consuming and prosuming with computer literacy and
critical consuming and prosuming with information literacy, and
argue that NML is a convergence of all these four components with
critical prosuming being the most crucial in the 21st century.
A couple of years later, Lin et al. (2013) refined the theoretical
framework developed by Chen et al. (2011) so that it could provide
more comprehensive explanation of NML and put more emphasis
on distinguishing characteristics of Web 2.0 technologies. Essentially,
Lin et al. (2013) kept four components of NML same as before,
but proposed ten fine-grained indicators to further elaborate these
components.
In the refined framework by Lin et al. (2013), functional
consuming literacy is further represented by consuming skill and
understanding indicators. The consuming skill involves a number of
technical abilities in order to operate different hardware and software
for accessing various media content whereas the understanding
indicator refers to the ability to grasp literal meaning of
media messages. For example, a functional consumer knows how to
use computers and Internet search engines to locate information in
any form of media and he/she can capture and interpret others’
M. Koc, E. Barut / Computers in Human Behavior 63 (2016) 834e843 835
ideas submitted to discussion boards, chat rooms or social
networks.
Critical consuming literacy is identified by analysis, synthesis,
and evaluation indicators. The analysis indicator refers to individuals’
ability to deconstruct media messages in terms of their
authorships, formats, audiences, and purposes. It highlights the
recognition of media messages as subjective rather than simply
perceiving them as neutral. The synthesis indicator involves the
competency of sampling, remixing, and comparing media content
from different sources. On top of analysis and synthesis, the evaluation
indicator includes the ability to examine the reliability and
credibility of media content. It helps to reach true, relevant and
unbiased information. For example, a critical consumer can identify
manipulative or biased news spreading on the Internet by examining
its construction process and verifying it from multiple sources.
As can be seen, such an inquiry requires individuals to exercise
higher order thinking skills in order to critically engage with
mediated messages.
Lin et al. (2013) characterize functional prosuming literacy
based on three indicators: prosuming skill, distribution, and production.
Similar to consuming skill, the prosuming skill refers to a
number of technical skills in order to use various technologies for
creating digital artifacts. The distribution indicator involves individuals’
activities to share their own feelings, ideas and digital
artifacts with others in new media platforms. Therefore, it relates to
the process of information dissemination and belongs to Web 2.0
technologies exclusively. The production indicator indicates the
competency of duplicating, rearranging, or combining text, audio
and video pieces into digital media formats. For example, a functional
prosumer can open a new Facebook profile or Twitter account,
create a slideshow or video clip by mixing his/her pictures,
and share this with friends.
As the last but the most complex and crucial media literacy,
critical prosuming is represented by participation and creation indicators
(Lin et al., 2013). The former refers to individuals’ interactive
and critical participation in new media platforms. This
indicator is exclusively related to active contribution and collective
intelligence characterized by Web 2.0 technologies. Therefore, it
requires individuals to own social skills to achieve digital
communication and collaboration with others. A critical prosumer
in a discussion forum, for instance, can identify deception, improve
others’ comments or go into a negotiation with others by presenting
his/her ideas as well as being respectful for diverse values
and ideologies. The latter indicator involves individuals’ initiations
to create original media contents in which their own socio-cultural
values and ideologies embedded or to combine preexisting media
content to create new meanings. For example, a critical prosumer
can develop a new website or open a new blog in order to promote
awareness about a topic of interest or to start discussion on a
matter of opinion.
1.3. Purpose of the study
As can be seen from the explanations above, NML is a new
concept and involves a series of crucial skills needed for living and
working in the mediated and participatory culture of 21st century.
Educational institutions take it as one of the top priority issues in
their curriculums because new media environments revolutionize
learning and teaching. Media education activities help students
develop adequate self-confidence, intellectual curiosity and critical
thinking in order to make judgments about media messages that
they might come across in their future lives (Hobbs, 2011).
Although theoretical conceptualizations of NML may inform
teaching or building new media competencies, operational definition
is needed to diagnose both the capacity of students and the
effectiveness of educational practices related to NML. In this study,
we aimed to develop and validate a comprehensive scale for
measuring NML based on the theoretical framework of Lin et al.
(2013) explained earlier. Our literature review revealed that existing
instruments focused only consuming or information-seeking
skills germane to conventional media and thus they failed to
cover prosuming skills related to unique affordances of new media.
Therefore, to our best knowledge at the beginning of this study, no
instrument has ever been particularly developed to measure young
adults’ NML inclusively. In this regard, our study makes a unique
contribution to the kickoff and development of systematic investigation
of this emerging concept.
2. Methodology
2.1. Item development
The quantification of NML was accomplished based on qualitative
exploration of NML concepts and outcomes. We conducted an
item generation process for NML scale (NMLS) with the guidance of
adopted theoretical framework of NML (Lin et al., 2013), literature
review of previous measures and research on media and digital
literacy (Arke & Primack, 2009; Eshet, 2012; Inan & Temur, 2012;
Karaman & Karatas, 2009) and our focus group discussions. Each
item was constituted in three steps. Firstly, we examined theoretical
explanations of different dimensions of NML and their finegrained
indicators to get a conceptual understanding of NML
construct. Secondly, we discussed and articulated potential items
that could represent those literacy indicators. In this step, previous
scales were also inspected to find out if they had adaptable items.
Most items related to critical and prosuming dimensions of NML
framework were developed based on our focus group discussions
because they were not covered by traditional media literacy scales.
After reaching consensus on item wording, we finally wrote
candidate statements. The initial list of statements was carefully
reviewed to eliminate complex and illogical expressions, recurrences,
and typos. Finally, the item pool was made up of 45
positive items measured on a 5-point Likert-type scale with
“1 ¼ strongly disagree” and “5 ¼ strongly agree”. Sample items
include “I know how to use searching tools to get information
needed in the media”, “I can compare news and information across
different media environments”, “I am able to use software necessary
for developing media contents (text, image, video, etc.)”, and “I
can collaborate and interact with diverse media users towards a
common purpose”.
2.2. Expert review
Writing adequate number of items from all aspects of concept
domain is crucial for content validity during scale development.
Content validity refers to the systematic examination of the scale
content to determine whether it covers a representative sample of
the behavior domain to be measured (Anastasi & Urbina, 1997). It
depends on a theoretical basis and is largely a matter of judgment.
A usual approach to establishing content validity is to consult with
experts. We utilized two-stage process, as described by Lynn
(1986), to determine content validity of NMLS. In the first stage
(i.e., development stage), we initially identified the content domain
and conceptualized NML by adopting theoretical framework of Lin
et al. (2013). In this framework, NML is defined by four dimensions
with a total of ten indicators: functional consuming (consuming skill
and understanding), critical consuming (analysis, synthesis and
evaluation), functional prosuming (prosuming skill, distribution and
production) and critical prosuming (participation and creation). The
definition and sample competencies of these dimensions and
M. Koc, E. Barut / Computers in Human Behavior 63 (2016) 834e843836
indicators were already given in section 1.2 above. Next, we
generated potential items and formulated the scale based on this
theoretical framework, existing measures and focus group interviews.
Detailed information about these actions was also given
in the preceding section 2.1.
The second stage (i.e., judgment-quantification stage) involved
expert assessment for the relevance of scale’s content. Lynn (1986)
recommended at least three experts but stated that more than 10
were probably unnecessary. Accordingly, we worked with four academicians
from those fields related to our research scope and
methodology such as computer education and instructional technologies,
communication studies, and psychometrics. These experts
were given the definition of NML dimensions and indicators
as well as a list of 45 items. As advised by Lynn (1986), they were
asked to write their comments and rate the relevancy of each item
to given definitions on a 4-point ordinal scale in which 1 ¼ not
relevant, 2 ¼ somewhat relevant, 3 ¼ quite relevant and 4 ¼ highly
relevant. We also urged them to reword items as needed to make
them more clear and concise and add new ones relevant to NML
concepts. After gathering feedback and rating data from the experts,
we computed content validity index for each item (I-CVI) as
the number of experts giving a rating of either 3 or 4 divided by the
total number of experts. Lynn (1986) developed criteria for item
relevancy by incorporating the standard error of the proportion.
She recommended that the I-CVI should be 1 for three to five experts
and a minimum of 0.78 for six to ten experts. Results of the
analysis of expert ratings and I-CVIs for the 45 items revealed that
26 items met the criteria whereas 19 items had the I-CVI values less
than 1. Of these 19 items, eight items with no advice for revision
were eliminated and eleven items were rewritten based on the
experts’ recommendations. Three new items that experts thought
to be consistent with NML conceptual domain were also added to
the scale. The second round of expert review acknowledged these
revisions. As a result, this content validation process modified the
number of items in the pool to 40.
2.3. Pilot testing
Another crucial step in scale development is to ensure that
prospective respondents understand item statements. For this
purpose, we conducted focus group interviews with 15 volunteer
college students to pilot the revised 40 items. Students responded
the items in a questionnaire form at an empty classroom. We
observed students during this process and at the end hold discussions
about how they perceived and understood each item. They
were requested to make judgments about item wording, clarity of
the statements, and logical organization of the items. Such judgments
are based less on the technical elements of content validity
and more on what looks valid (i.e., face validity) (Anastasi & Urbina,
1997). Participants of this pilot-testing found the items and uniform
5-point Likert scale acceptable and easy to understand. The analysis
of their feedback revealed that all items demonstrated good clarity
in language and face validity from the perspectives of target
audience.
2.4. Participants
The revised version of 40-item NMLS emerged from the aforementioned
scale development processes was administered to 1311
volunteer university students at a large state university in Turkey.
We inspected returned questionnaires thoroughly and eliminated
those that were simply blank, considerably incomplete or negligently
responded. Consequently, the final sample included 1226
students. Participants were drawn from nine different colleges
including Arts and Sciences (26%), Forestry (15%), Agriculture (12%),
Theology (12%), Economics and Administrative Sciences (11%), Engineering
(10%), Education (5%), Technology (5%), and Law (4%).
Almost half of the students (49%) were freshman, 13% were sophomore,
20% were junior, and 18% were senior. Of the sample, 55%
were female and 45% were male students and their age ranged from
18 to 30 with a mean age of 21.05 (SD ¼ 1.96).
We contacted college administrations to obtain necessary
permission for data collection by writing a petition explaining the
scope and aim of our research. The questionnaires were conducted
in students’ lecture halls either before or after their lessons. Students
were initially asked to give their written consent and
informed that any information they provided would be voluntary,
confidential, and used only for research purposes. Those students
who had volunteered to participate were informed about how they
fill out the questionnaire form.
3. Results
3.1. Construct validity phase
The construct validity of the NMLS was judged via exploratory
factor analysis (EFA) and confirmatory factor analysis (CFA). Using
the function of SPSS software for selecting cases, we randomly split
the sample into two halves: (a) development subsample (n ¼ 613)
and (b) cross-validation subsample (n ¼ 613). We firstly employed
EFA on the former to identify the factorial structure of the items,
and then CFA on the latter to cross-validate and refine the emerged
measurement model from the EFA.
3.1.1. Exploring factorial structure of the NMLS
Prior to performing EFA, the suitability of the data in the
development subsample for the factor analysis was assessed. There
were several missing values randomly distributed within the items
and they were estimated using series mean method. The inspection
of Boxplots indicated four to seven univariate outliers in several
items. Because the sample was quite large and the differences between
5% trimmed mean and overall mean of the related items
were very small, we decided that these cases could not have a
distorting effect on the distribution and thus retained them in the
data set. The normality assumption was also supported by absolute
values of skewness and kurtosis less than 1 and normal probability
plots representing reasonably straight lines. The presence of
possible multivariate outliers was checked by calculating Mahalanobis
distance values. Those cases whose Mahalanobis values
exceeding the critical Chi-square value of 80.08 (df ¼ 39,
alpha ¼ 0.001) were considered as outliers and removed from the
analysis (Pallant, 2007; Tabachnick & Fidell, 2007). The KaiserMeyer-Olkin
(KMO) measure of sampling adequacy was 0.95,
exceeding the recommended value of 0.60 and Barlett’s Test of
Sphericity was statistically significant (c2
¼ 12,227.16, df ¼ 780,
p < 0.01), indicating highly acceptable factorability of the data and
correlation matrix (Pallant, 2007). The development subsample
size of the study (n ¼ 613) was more than adequate because the
number of cases from 100 to 400 can be regarded as suitable for
factor analysis (Hair, Black, Babin, & Anderson, 2010).
In the EFA, all 40 items were subjected to principal component
analysis with varimax orthogonal rotation using SPSS 20. The initial
solution revealed seven factors with eigenvalues over 1. However,
those after the fourth factor had eigenvalues and percentages of
explained variances that were relatively small and close to each
other. An inspection of Catell’s (1966) scree test showed a clear
break after the fourth factor in the screeplot (Fig. 1). Furthermore,
we performed a parallel analysis through Monte Carlo PCA software
(Watkins, 2000) and found that only four factors had eigenvalues
greater than the corresponding criterion values for 100 randomly
M. Koc, E. Barut / Computers in Human Behavior 63 (2016) 834e843 837
generated data matrix of the same size as our real sample. Based on
these findings, we decided to retain four factors for further
investigation.
The replication of EFA with four-factor solution indicated that
five items had factor loadings less than recommended value of 0.50
(Hair et al., 2010) or similar cross loadings whose differences were
less than 0.10. After removal of these items, the final four-factor
model with the remaining 35 items accounted for 55% of the total
variance, which is an acceptable rate in social sciences. Examining
items’ meanings and coherency in accordance with the adopted
theoretical framework of NML (Lin et al., 2013), we labeled factors
as functional consumption (FC, 7 items), critical consumption (CC,
11 items), functional prosumption (FP, 7 items), and critical prosumption
(CP, 10 items) respectively. Item wordings, factor loadings,
eigenvalue, variance explained, and Cronbach’s alfa for each
factor are given in Table 1. The factor loadings ranged from 0.50 to
0.85, suggesting that all items were good measures of their
respective factors (Hair et al., 2010). As indicated by Cronbach’s
alphas (0.85, 0.87, 0.89, and 0.93) well above the threshold value of
0.70, all factors were internally consistent and well defined by their
items (DeVellis, 2003).
3.1.2. Confirming factorial structure of the NMLS
The four-factor measurement model extracted from EFA was
subjected to CFA using the cross-validation subsample and LISREL
8.80 software. Like in the EFA, we initially checked the assumptions
of CFA. There were four to seven univariate outliers observed in the
Boxplots of several items. We decided to retain these cases since the
sample was quite large and their %5 trimmed means were not as
much different than overall means. In fact, none of the items had
absolute values of skewness and kurtosis larger than 1, suggesting
that the data could be treated as univariately normally distributed
(Kline, 2005). This was also supported by visual examination of
normal probability plots, which showed reasonably straight lines.
Mahalanobis distance values were calculated for the inspection of
multivariate outliers. The cases with Mahalanobis values greater
than critical Chi-square value of 65.25 (df ¼ 34, alpha ¼ 0.001) were
treated as outliers and thus removed from the analysis (Pallant,
2007; Tabachnick & Fidell, 2007).
Because our data met normality assumptions, we employed
maximum likelihood estimation, which is the most widely used
estimation technique in structural equation modeling (SEM). The
measurement model comprised of factors as latent constructs and
items as observed variables (Fig. 2). In accordance with the NML
theoretical framework, all constructs were allowed to correlate
with others. Chi-square goodness of fit test, which is the fundamental
measure of model fit in SEM analysis, was found to be
statistically significant (c2
¼ 1279.25, df ¼ 544, p < 0.01). However,
c2
is known to be biased towards large samples and complex
models. Thus c2
/df ratio is recommended to sufficiently evaluate
model fit and values less than 3 indicate a good model fit (Kline,
2005). This ratio was found to be 2.35 in our analysis. Moreover,
we used alternative model fit indices based on residuals and independent
model including Standardized Root Mean Square Residuals
(SRMR), Root Mean Square of Error of Approximation
(RMSEA), Goodness of Fit Index (GFI), Comparative Fit Index (CFI),
and Normative Fit Index (NFI). The SRMR and RMSEA values equal
or less than 0.05 signify perfect fitness and the GFI, CFI and NFI
values greater than 0.90 indicate good fitness (Brown, 2006; Hair
et al., 2010). We found these index values as follows:
SRMR ¼ 0.050, RMSEA ¼ 0.049, GFI ¼ 0.89, CFI ¼ 0.98 and
NFI ¼ 0.97. As shown in Fig. 2, standardized parameter estimates for
all items were statistically significant (p < 0.01) and not less than
recommended value of 0.50 (Hair et al., 2010), ranging from 0.62 to
0.72 for FC, 0.50 to 0.68 for CC, 0.64 to 0.82 for FP, and 0.64 to 0.82
for CP factor. Since each item was loaded only on its respective
factor in the model, the standardized parameter estimates can be
regarded as indicators of item-factor correlations (Kline, 2005).
Therefore, relatively large and significant estimates provide evidence
for convergent validity which refers to the degree to which
theoretically similar measures are in fact related to each other
(Maruyama, 1998). All estimates for each factor in our model were
significant and relatively large (0.50 and above) and hence supported
convergent validity. Collectively, these results suggest that
four-factor 35-item measurement model of NMLS fits well with the
observed data and exhibits adequate construct validity.
Fig. 1. Scree plot for exploratory factor analysis of the NMLS.
M. Koc, E. Barut / Computers in Human Behavior 63 (2016) 834e843838
Table 1
Results of exploratory factor analysis of the NMLS.
Factor/item Factor
loading
Eigenvalue Variance
explained
Cronbach’s
alpha
Functional consumption (FC) 3.73 10.65% 0.85
FC1: Know how to use searching tools to get information needed in the media. 0.50
FC2: Catch up with the changes in the media. 0.62
FC3: Make use of various media environments to reach information. 0.60
FC4: Realize explicit and implicit media messages. 0.67
FC5: Notice media contents containing mobbing and violence. 0.71
FC6: Understand political, economical and social dimensions of media contents. 0.70
FC7: Perceive different opinions and thoughts in the media. 0.62
Critical consumption (CC) 5.02 14.35% 0.87
CC1: Distinguish different functions of media (communication, entertainment, etc.). 0.54
CC2: Determine whether or not media contents have commercial messages. 0.59
CC3: Classify media messages based on their producers, types, purposes and so on. 0.59
CC4: Compare news and information across different media environments. 0.55
CC5: Combine media messages with own opinions. 0.59
CC6: Consider media rating symbols to choose which media contents to use. 0.53
CC7: Make decision about the accuracy of media messages. 0.71
CC8: Analyze positive and negative effects of media contents on individuals. 0.67
CC9: Evaluate media in terms of legal and ethical rules (copyright, human rights, etc.) 0.68
CC10: Assess media in terms of credibility, reliability, objectivity and currency. 0.64
CC11: Fend against the risks and consequences caused by media contents. 0.54
Functional prosumption (FP) 4.20 12.01% 0.89
FP1: Create user accounts and profiles in media environments. 0.68
FP2: Use hardware necessary for developing media contents (text, image, video, etc.). 0.69
FP3: Use software necessary for developing media contents (text, image, video, etc.). 0.69
FP4: Use basic operating tools (button, hyperlinks, file transfer etc) in the media. 0.70
FP5: Share digital media contents and messages on the Internet. 0.74
FP6: Make contribution or comments to media contents shared by others. 0.67
FP7: Rate or review media contents based on personal interests and liking. 0.64
Critical prosumption (CP) 6.19 17.67% 0.93
CP1: Influence others’ opinions by participating to social media environments. 0.73
CP2: Make contribution to media by reviewing current matters from different perspectives (social, economical,
ideological etc.).
0.71
CP3: Collaborate and interact with diverse media users towards a common purpose. 0.72
CP4: Construct online identity consistent with real personal characteristics. 0.55
CP5: Make discussions and comments to inform or direct people in the media. 0.70
CP6: Design media contents that reflect critical thinking of certain matters. 0.82
CP7: Produce opposite or alternative media contents. 0.85
CP8: Produce media contents respectful to people’s different ideas and private lives. 0.73
CP9: Create media contents that comply with legal and ethical rules. 0.74
CP10: Develop original visual and textual media contents (video clips, web page, etc.) 0.68
0.70
0.79
0.640.60
0.57
0.50
0.70
0.63 0.63 0.63 0.67 0.62 0.72
0.50
0.74
0.76
0.82
0.65
0.76
0.77
0.78
0.68
0.64
0.67
0.57
0.61
0.63
0.64
0.65
0.65
0.68
0.67
0.64
0.58
0.820.68 0.75 0.71 0.77 0.70 0.64
Functional
Consumption
(FC)
Critical
Consumption
(CC)
FC1
(3.77±0.88)
Functional
Prosumption
(FP)
Critical
Prosumption
(CP)
FC2
(3.61±0.94)
FC3
(3.74±0.90)
FC4
(3.48±0.99)
FC5
(3.70±1.05)
FC6
(3.88±0.93)
FC7
(3.97±0.85)
CC5
(3.90±0.85)
CC6
(3.81±0.99)
CC4
(3.81±0.91)
CC3
(3.87±0.90)
CC2
(4.30±0.81)
CC1
(4.29±0.78)
CC11
(3.79±0.91)
CC10
(3.87±0.85)
CC9
(3.87±0.90)
CC8
(4.11±0.82)
CC7
(4.00±0.79)
CP5
(3.44±1.08)
CP6
(3.14±1.15)
CP4
(3.52±1.21)
CP3
(3.20±1.10)
CP2
(3.26±1.13)
CP1
(3.19±1.17)
CP10
(2.95±1.24)
CP9
(3.30±1.23)
CP8
(3.44±1.18)
CP7
(3.07±1.21)
FP1
(4.04±0.99)
FP2
(3.81±1.04)
FP3
(3.55±1.10)
FP4
(3.83±1.00)
FP5
(3.91±1.03)
FP6
(4.05±0.97)
FP7
(3.90±1.05)
Fig. 2. Results of confirmatory factor analysis of the four-factor model of NMLS. The numbers in parentheses represent item mean scores and standard deviations (Mean ± SD).
M. Koc, E. Barut / Computers in Human Behavior 63 (2016) 834e843 839
3.1.3. Second-order confirmatory factor analysis
After verifying the four-factor measurement model of NMLS, we
also developed a second-order model in order to investigate if four
factors can be explained by a single higher-order factor. This onefactor
model assumed that the relationships among four constructs
emerged from the first-order model (FC, CC, FP, and CP) can
be exclusively accounted for a broader latent construct (NML).
Therefore, it did not allow four constructs to be correlated with
each other but rather loaded them onto the NML construct with
unidirectional connections. We tested this model by running CFA in
the cross-validation subsample. Similar to the first-order model,
the results showed a good model fit (c2
/df ¼ 2.36, SRMR ¼ 0.054,
RMSEA ¼ 0.049, GFI ¼ 0.88, CFI ¼ 0.98, NFI ¼ 0.97). The standardized
parameter estimates between the second-order factor of
NML and the first-order factors of FC, CC, FP, and CP (i.e., structural
loadings) were 0.91, 0.83, 0.79, and 0.65 respectively and statistically
significant (p < 0.01). Besides, those between the first-order
factors and observed items (i.e., measurement loadings) were statistically
significant (p < 0.01) and not less than recommended
value of 0.50 (Hair et al., 2010), ranging from 0.63 to 0.71 for FC,
0.50 to 0.68 for CC, 0.64 to 0.82 for FP, and 0.64 to 0.82 for CP factor.
Similar to first-order model, all standardized estimates in the
second-order model were significant and relatively large, which
provided evidence for convergent validity (Maruyama, 1998). These
findings as a whole suggest that NML can also be considered as an
underlying unidimensional construct.
3.2. Construct reliability and item analysis phase
3.2.1. Internal consistency level
Both factor and entire reliability analyses of four-factor model of
NMLS were carried out by calculating Cronbach alpha internal
consistency coefficients on the whole sample. The subscale coefficient
values for each factor were as follows: 0.85 for FC, 0.87 for CC,
0.89 for FP, and 0.93 for CP. Moreover, the coefficient value for the
whole NMLS was found to be 0.95. These findings exceed the
acceptable Cronbach’s alpha level of 0.70 for scale development
(DeVellis, 2003; Nunnally & Bernstein, 1994), and thus indicate a
good convergence or internal consistency.
3.2.2. Item-total correlations
As an important phase of item analysis, the corrected itemfactor
total correlations were also examined to determine the coherency
of items within the same factor (i.e., level of serving the
purpose of factor). All item-factor total correlations were much
greater than threshold value of 0.30 (Pallant, 2007), ranging from
0.55 to 0.66 for FC, 0.47 to 0.64 for CC, 0.62 to 0.73 for FP, and 0.59 to
0.80 for CP factor. Furthermore, the corrected factor-scale total
correlations were also above 0.30, ranging from 0.45 to 0.65. These
findings suggest that NMLS has significant item-factor and factorscale
relationships. In other words, each item within the same
factor serves the general purpose of that factor as well as the
general purpose of the NMLS.
3.2.3. Item discrimination power
Another important step in item analysis is to determine how
well the four-factor model of NMLS discriminates between individuals
with high competency and those with low competency in
terms of NMLS. In order to accomplish this, we observed the differentiation
between the lowest 27% and highest 27% of the participants
in the whole sample. For this reason, we ranked the raw
scores from the highest to the lowest and then identified the first
27% of them as the higher group (n ¼ 331) and the last 27% of them
as the lowest group (n ¼ 331) for each item, factor, and the whole
scale. This was followed by conducting independent sample t-test
to statistically test the difference between the mean scores of the
two groups. The t-test values for all item scores were statistically
significant (p < 0.01), ranging from 49.12 to 72.58 in FC, 40.87 to
74.75 in CC, 62.83 to 66.45 in FP, and 57.92 to 84.59 in CP factor. The
t-test values for the total scores were also statistically significant
(p < 0.01), 65.78 for FC factor, 66.84 for CC factor, 69.12 for FP factor,
80.91 for CP factor, and 76.01 for the whole NMLS. These results
suggest that each item, factor and the whole scale have adequate
power to distinguish individuals in terms of their levels of
regarding NML competency. Moreover, as shown in Table 2, the
inter-correlations between the factors were moderately small
enough (varied between 0.44 and 0.63) and did not exceed 0.70,
suggesting that the four factors were adequately distinct and thus
the scale demonstrated discriminant validity (Ping, 2004).
4. Discussion and conclusion
Consistent with the adopted theoretical framework of NML (Lin
et al., 2013), our study produced a four-factor NMLS comprising FC,
CC, FP, and CP as seemingly distinct but related factors, each of
which is measured by multiple Likert type items. This finding
empirically suggests that NML can be operationalized in the future
studies by employing the four-factor NMLS (Appendix). In addition
to measuring this first-order model, we also developed and tested
the second-order model to measure overall general NML ability.
This is a natural next step in scale development. After a researcher
discovers multidimensional structure of the data and forms subscales,
he/she also investigates whether it is possible to develop a
generalized or total scale by combining these subscales. In fact, it is
recommended to test alternative models for comparison of various
conceptualizations of factorial structure of the instrument to be
developed (Noar, 2003). Our second-order one-factor model represents
the hypothesis that whether FC, CC, FP, and CP factors
extracted from the first-order four-factor model can load on one
common or unidimensional underlying construct (i.e., overall NML
ability). However, we found that both fit indices and c2
/df ratio
were almost identical for both models though the second-order
model produced two extra degrees of freedom. These results provide
evidence that both models are statistically equivalent and
applicable for measuring NML. The only difference between them is
that the second-order model is a special case of the first-order
model and provides an alternative account of the association between
the first-order factors. The decision on which model to be
used depends ultimately on what theory suggests (Byrne, 2001).
Since we take the four dimensional theoretical model of NML as a
reference in our scale development, the first-order four-factor
model can be safely utilized in measuring NML.
On the other hand, the second-order model analysis also provided
evidence for unidimensional usage of NMLS. Researchers can
administer the scale and generate a composite variable by summing
up the scores of all items (i.e., total score) rather than producing
four variables with subscale scores. The relevant literature
recommends that second-order models can offer several advantages
over the first-order models in some research contexts. For
example, they can allow researchers to test whether the total score
actually account for the relationships between the subscores,
explain the covariance in a more parsimonious way with fewer
parameters, separate variance owing to specific factors from the
measurement error, and interpret complex research structures
such as multitrait-multimethod models or latent state-trait models
in a more simple and useful manner (Chen, Sousa, & West, 2005).
Using second-order model might be particularly beneficial when
researchers are interested in exploring if first-order factors can
predict a criterion variable. Therefore, the choice of which model of
the scale to be used belongs to researchers and their research
M. Koc, E. Barut / Computers in Human Behavior 63 (2016) 834e843840
purposes. Future studies are needed to compare both models in
different samples and research contexts for further cross-validation
of the items.
From the theoretical aspect, our findings empirically validate
the preliminary work of Chen et al. (2011) and continuing efforts of
Lin et al. (2013) in which they conceptualize NML as four types of
literacy comprising two continuums from consumption to prosumption
and from functionality to criticality. The study also corroborates
the technical and socio-cultural characteristics of new
media as the scale involves relevant items. One of the most
important practical implications of our study is to promote research
studies on NML from conceptual level to empirical one and thus to
stimulate more research on new media and competencies needed
to live with it. Recently, there has been a growing interest in conceptual
discussion about the new media emerged with the advancements
in information and communication technologies.
However, the relevant literature lacks an operational tool to measure
individuals’ new media competencies especially producing
and criticizing digital media contents. The NMLS developed in the
present study can fill this gap and be utilized in the data collection
phases of descriptive and experimental research contexts. This
enables researchers to explore a variety of variables with which
NML is associated.
The descriptive statistics for NML factors show that participants
have good levels of FC, CC and FP whereas they have average level of
CP. This is expected because CP is the most complex and crucial part
of NML. The results suggest that current generation of students
should be supported in critical and active participation in new
media platforms and creation of original media contents that
convey their own socio-cultural values and ideologies. School education
should put more focus on how to produce and criticize
media contents than how to access and understand them. NML
impacts current generation’s performance in school and business. It
is regarded as both learning and job skill. Students and employees
are expected to learn new media skills in the 21st century. Therefore,
the validated NMLS can be readily utilized within a school or
business to promote awareness about new media and its competencies.
Furthermore, it can be employed as a diagnostic or needs
assessment tool to access students’ level of NML so that teachers
can develop and implement suitable curricular activities to
improve new media usage and test their educational effectiveness.
The data collected through the NMLS can be used for pedagogical
decision making in technology-enhanced learning in the schools.
Teachers might use the NMLS to understand how their students
interact with new media in learning. Kong et al. (2014) foresee that
the goals of education in digital classrooms include the development
of 21st century skills beyond learning domain knowledge,
and indicate the development of tools for assessment of these skills
as a critical research issue in the coming ten years. This study can
contribute in this regard by providing a measuring tool for NML.
The NMLS can also aid employers to assess their employees’ performance
gaps with regard to new media competencies and then
design and implement need-based professional development
programs.
In conclusion, our study produced a reliable and psychometrically
valid scale for the assessment of NML. Along with the
emerging new media and digital technologies, individuals are expected
to actively participate in new media by producing, sharing
and questioning digital contents beyond merely consuming them.
Being new media literate, they also need to know social, emotional
and cultural aspects of new media as well as its technical features.
The NMLS involves such newest and complex media literacy
competencies that emerged in the last decades. Its availability can
stimulate to conduct future research studies and thus add to the
growing body of literature on NML. The present study is limited to
content and construct validity analysis including convergent and
discriminant validity. Future studies may check additional validities
such as criterion and nomological ones by taking the account of
several variables related to NML. Another limitation of this study is
germane to the sample. We worked with college students only
because our scale targets this generation’s new media competencies.
However, the recruitment of participants was based on
convenient sampling and could not be representative for all college
students. This may decrease the potentiality of our study for
generalizing results to larger populations (i.e., hindering external
validity). Fortunately, the sample is quite cosmopolite (including
students from different colleges) and large in size (n ¼ 1226) and
thus the power of the statistical techniques used in the study is
rather high. Future studies may focus on different combination of
college students to corroborate our findings or adaptation of NMLS
to younger generation such as primary and secondary school
students.
Appendix. New Media Literacy Scale (NMLS)
Directions: The term “media” used in the following items, unless
otherwise specified, refers to current digital technology platforms
including but not limited to web sites, online forums, social networks,
video sharing sites and virtual worlds in which anyone can
share any digital content. Please indicate how you feel about your
knowledge and skills for each of the following statements.
1 ¼ Strongly disagree, 2 ¼ Disagree, 3 ¼ Neither agree nor
disagree, 4 ¼ Agree, 5 ¼ Strongly agree.
Functional consumption (FC)
1 I know how to use searching tools to get information needed in
the media.
2 I am good at catching up with the changes in the media.
3 It is easy for me to make use of various media environments to
reach information.
4 I realize explicit and implicit media messages.
5 I notice media contents containing mobbing and violence.
6 I understand political, economical and social dimensions of
media contents.
7 I perceive different opinions and thoughts in the media.
Table 2
Descriptive statistics and correlation coefficients for NMLS factors.
Factor Min-Max Mean SD Skewness Kurtosis Correlation coefficient
CC FP CP
Functional consumption (FC) 7e35 25.85 4.93 À0.39 0.18 0.63*
0.56*
0.45*
Critical consumption (CC) 16e55 43.11 6.51 À0.47 0.06 0.56*
0.44*
Functional prosumption (FP) 7e35 26.72 5.73 À0.63 0.01 0.51*
Critical prosumption (CP) 10e50 32.22 8.97 À0.36 À0.43
*
p < 0.01.
M. Koc, E. Barut / Computers in Human Behavior 63 (2016) 834e843 841
Critical consumption (CC)
8 I can distinguish different functions of media (communication,
entertainment, etc.).
9 I am able to determine whether or not media contents have
commercial messages.
10 I manage to classify media messages based on their producers,
types, purposes and so on.
11 I can compare news and information across different media
environments.
12 I can combine media messages with my own opinions.
13 I consider media rating symbols to choose which media
contents to use.
14 It is easy for me to make decision about the accuracy of
media messages.
15 I am able to analyze positive and negative effects of media
contents on individuals.
16 I can evaluate media in terms of legal and ethical rules
(copyright, human rights, etc.).
17 I can assess media in terms of credibility, reliability, objectivity
and currency.
18 I manage to fend myself from the risks and consequences
caused by media contents.
Functional prosumption (FP)
19 It is easy for me to create user accounts and profiles in media
environments.
20 I can use hardware necessary for developing media contents
(text, image, video, etc.).
21 I am able to use software necessary for developing media
contents (text, image, video, etc.).
22 I can use basic operating tools (button, hyperlinks, file
transfer etc) in the media.
23 I am good at sharing digital media contents and messages on
the Internet.
24 I can make contribution or comments to media contents
shared by others.
25 I am able to rate or review media contents based on my
personal interests and liking.
Critical prosumption (CP)
26 I manage to influence others’ opinions by participating to
social media environments.
27 I can make contribution to media by reviewing current
matters from different perspectives (social, economical,
ideological etc.).
28 I am able to collaborate and interact with diverse media users
towards a common purpose.
29 It is easy for me to construct online identity consistent with
real personal characteristics.
30 I can make discussions and comments to inform or direct
people in the media.
31 I am skilled at designing media contents that reflect critical
thinking of certain matters.
32 I am good at producing opposite or alternative media
contents.
33 I produce media contents respectful to people’s different
ideas and private lives.
34 It is important for me to create media contents that comply
with legal and ethical rules.
35 I am able to develop original visual and textual media contents
(video clips, web page, etc.)
References
Anastasi, A., & Urbina, S. (1997). Psychological testing (7th ed.). New York:
Macmillan.
Arke, E. T., & Primack, B. A. (2009). Quantifying media literacy: Development,
reliability, and validity of a new measure. Educational Media International, 46(1),
53e65.
Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York:
Guilford Publications.
Buckingham, D. (2003). Media education: Literacy, learning and contemporary culture.
Cambridge, UK: Polity Press.
Butler, J. (2012). Grappling with change: Web 2.0 and teacher education. In D. Polly,
C. Mims, & K. A. Persichitte (Eds.), Developing technology-rich teacher education
programs: Key issues (pp. 135e150). Hershey, PA: IGI Global.
Byrne, B. (2001). Structural equation modeling with AMOS: Basic concepts applications,
and programming. London: Lawrence Erlbaum Associates.
Catell, R. B. (1966). The scree test for number of factors. Multivariate Behavioral
Research, 1, 245e276.
Chen, F. F., Sousa, K. H., & West, S. G. (2005). Testing measurement invariance of
second-order factor models. Structural Equation Modeling, 12(3), 471e492.
Chen, D.-T., Wu, J., & Wang, Y.-M. (2011). Unpacking new media literacy. Journal on
Systemics, Cybernetics and Informatics, 9(2), 84e88.
DeVellis, R. F. (2003). Scale development: Theory and applications (2nd ed.). Thousand
Oaks, CA: Sage Publication.
Eshet, Y. (2012). Thinking in the digital era: A revised model for digital literacy.
Issues in Informing Science and Information Technology, 9, 267e276.
Hair, J., Black, W., Babin, B., & Anderson, R. (2010). Multivariate data analysis (7th
ed.). Upper Saddle River, NJ: Prentice-Hall.
Held, C., & Cress, U. (2009). Learning by foraging: The impact of social tags on
knowledge acquisition. In U. Cress, V. Dimitrova, & M. Specht (Eds.), Learning in
the synergy of multiple disciplines (pp. 254e266). Germany: Springer-Verlag
Berlin Heidelberg.
Hobbs, R. (2011). The state of media literacy: A response to potter. Journal of
Broadcasting & Electronic Media, 55(3), 419e430.
Hobbs, R., & Jensen, A. (2009). The past, present and future of media literacy education.
The Journal of Media Literacy Education, 1(1), 1e11.
Inan, T., & Temur, T. (2012). Examining media literacy levels of prospective teachers.
International Electronic Journal of Elementary Education, 4(2), 269e285.
Jenkins, H. (2006). Convergence culture: Where old and new media collide. New York:
New York University Press.
Karaman, M. K., & Karatas, A. (2009). Media literacy levels of the candidate teachers.
Elementary Education Online, 8(3), 798e803.
Kellner, D. (2010). Interview with douglas kellner. Retrieved November 05, 2015 from
http://www.medialit.org/sites/default/files/Voices_of_ML_Douglas_Kellner.pdf.
Kellner, D., & Share, J. (2007). Critical media literacy, democracy, and the reconstruction
of education. In D. Macedo, & S. R. Steinberg (Eds.), Media literacy: A
reader (pp. 3e23). New York: Peter Lang Publishing.
Kline, R. B. (2005). Principles and practice of structural equation modeling (2nd ed.).
New York: The Guilford Press.
Kong, S. C., Chan, T.-W., Griffin, P., Hoppe, U., Huang, R., Kinshuk, L., et al. (2014). Elearning
in school education in the coming 10 years for developing 21st century
skills: Critical research issues and policy implications. Educational Technology &
Society, 17(1), 70e78.
Lin, T.-B., Li, J.-Y., Deng, F., & Lee, L. (2013). Understanding new media literacy: An
explorative theoretical framework. Educational Technology & Society, 16(4),
160e170.
Lynn, M. R. (1986). Determination and quantification of content validity. Nursing
Research, 35, 382e385.
Maloney, E. J. (2007). What Web 2.0 can teach us about learning. Chronicle of Higher
Education, 53(18), B26.
Maruyama, G. M. (1998). Basics of structural equation modeling. London: Sage
Publications.
McLoughlin, C., & Lee, M. J. W. (2007). Social software and participatory learning:
Pedagogical choices with technology affordances in the Web 2.0 era. In Proceedings
of ASCILITE conference (pp. 664e673) (Singapore).
Noar, S. M. (2003). The role of structural equation modeling in scale development.
Structural Equation Modeling, 10(4), 622e647.
Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York:
McGraw-Hill.
Oberhelman, D. (2007). Coming to terms with web 2.0. Reference Reviews, 21(7),
5e6.
O’Reilly, T. (2005). What is Web 2.0?. Retrieved November 05, 2015 from http://
www.oreilly.com/pub/a/web2/archive/what-is-web-20.html.
Pallant, J. (2007). SPSS survival manual: A step by step guide to data analysis using
SPSS version 15 (3rd ed.). New York: Open University Press.
Ping, R. A. (2004). On assuring valid measures for theoretical models using survey
data. Journal of Business Research, 57(2), 125e141.
Selwyn, N. (2007). Web 2.0 applications as alternative environments for informal
learning e A critical review. In Paper presented at the OECD-KERIS expert meeting
on ICT and educational performance, Cheju island, South Korea.
M. Koc, E. Barut / Computers in Human Behavior 63 (2016) 834e843842
Tabachnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics (5th ed.). Pearson
Education, Inc.
Thoman, E., & Jolls, T. (2004). Media literacy-A national priority for a changing
world. American Behavioral Scientist, 48(1), 18e29.
Toffler, A. (1981). The third wave. New York: Morrow.
Watkins, M. W. (2000). Monte Carlo PCA for PCA for parallel analysis [computer
software]. State College, PA: Ed & Psych Associates.
Zhang, H., Zhu, C., & Sang, G. (2014). Teachers’ stages of concern for media literacy
education and the integration of MLE in Chinese primary schools. Asia Pacific
Education Review, 15(3), 459e471.
M. Koc, E. Barut / Computers in Human Behavior 63 (2016) 834e843 843