Reanalyzing the DISMAS Test Data: Comparing IRT and CTT Based
Estimates of the Error of Measurement

a 2014

Reanalyzing the DISMAS Test Data: Comparing IRT and CTT Based Estimates of the Error of Measurement

CÍGLER, Hynek, Michal JABŮREK a Jan ŠIRŮČEK

Základní údaje

Originální název

Reanalyzing the DISMAS Test Data: Comparing IRT and CTT Based Estimates of the Error of Measurement

Autoři

CÍGLER, Hynek (203 Česká republika, garant, domácí), Michal JABŮREK (203 Česká republika, domácí) a Jan ŠIRŮČEK (203 Česká republika, domácí)

Vydání

The 9th Conference of The International Test Commission : Global and Local Challenges for Best Practices in Assessment, 2014

Další údaje

Jazyk

angličtina

Typ výsledku

Konferenční abstrakt

Obor

50100 5.1 Psychology and cognitive sciences

Stát vydavatele

Španělsko

Utajení

není předmětem státního či obchodního tajemství

Odkazy

Book of Abstract

Kód RIV

RIV/00216224:14230/14:00076015

Organizační jednotka

Fakulta sociálních studií

ISBN

978-84-9860-997-4

Klíčová slova česky

DISMAS; IRT; DIF; diagnostika matematických schopností

Klíčová slova anglicky

DISMAS; IRT; DIF; assessment of math ability

Příznaky

Mezinárodní význam

Změněno: 21. 7. 2014 16:39, Mgr. Michal Jabůrek, Ph.D.

Anotace

V originále

DISMAS test (Traspe & Skalkova, 2013) is an original Czech assessment tool for the diagnosis of the structure of math disability in children in grades 1–5. It consists of five main areas divided into 14 unidimensional subtests and it was developed and standardized using classical test theory. Authors verified validity. Subtests’ correlations with WISC-III were low, but significant, differences between normal and clinical population were high and significant for all grades. Regardless the high validity, reliability seemed to be poor – Cronbach alpha ranged from .44 to .95 on subtest level, on the test level from .69 to .76 depending on the grade level. The aim of this paper is to show why. We questioned the adequacy of CTT and internal consistency as a method for estimating reliability in this case. That is why we applied simple Rasch model and Partial Credit Rasch model to reanalyze the data. Consequently, the latent trait estimates as well as their error of measurement were transformed back to the raw score scale. We compared IRT and CTT based errors of measurement on the same scale for all observed raw scores according to the distribution of the standardization sample. In most subtests the error of estimate based on CTT was smaller around the average score but bigger in the extremes, especially at the low end of the scale (DISMAS is designed to test disability, so it has a high ceiling effect). We conclude that IRT analysis produces more accurate estimate for children with math disability, for whom the test is designed, so in this case Rasch model seems to be more appropriate method to estimate confidence intervals than CTT. Finally, we discuss other sources of relatively low reliability.

Citovat

CÍGLER, Hynek, Michal JABŮREK a Jan ŠIRŮČEK. Reanalyzing the DISMAS Test Data: Comparing IRT and CTT Based Estimates of the Error of Measurement. In The 9th Conference of The International Test Commission : Global and Local Challenges for Best Practices in Assessment. 2014. ISBN 978-84-9860-997-4.

@proceedings{1193136,
   author = {Cígler, Hynek and Jabůrek, Michal and Širůček, Jan},
   booktitle = {The 9th Conference of The International Test Commission : Global and Local Challenges for Best Practices in Assessment},
   keywords = {DISMAS; IRT; DIF; assessment of math ability},
   language = {eng},
   isbn = {978-84-9860-997-4},
   title = {Reanalyzing the DISMAS Test Data: Comparing IRT and CTT Based Estimates of the Error of Measurement},
   url = {http://www.itc2014ss.com/uploads/Libro%20grab%20CD%20Ong.pdf},
   year = {2014}
}

TY  - CONF
ID  - 1193136
AU  - Cígler, Hynek - Jabůrek, Michal - Širůček, Jan
PY  - 2014
TI  - Reanalyzing the DISMAS Test Data: Comparing IRT and CTT Based Estimates of the Error of Measurement
SN  - 9788498609974
KW  - DISMAS
KW  - IRT
KW  - DIF
KW  - assessment of math ability
UR  - http://www.itc2014ss.com/uploads/Libro%20grab%20CD%20Ong.pdf
L2  - http://www.itc2014ss.com/uploads/Libro%20grab%20CD%20Ong.pdf
N2  - DISMAS test (Traspe & Skalkova, 2013) is an original Czech assessment tool for the diagnosis of the structure of math disability in children in grades 1–5. It consists of five main areas divided into 14 unidimensional subtests and it was developed and standardized using classical test theory. Authors verified validity. Subtests’ correlations with WISC-III were low, but significant, differences between normal and clinical population were high and significant for all grades. Regardless the high validity, reliability seemed to be poor – Cronbach alpha ranged from .44 to .95 on subtest level, on the test level from .69 to .76 depending on the grade level. The aim of this paper is to show why. We questioned the adequacy of CTT and internal consistency as a method for estimating reliability in this case. That is why we applied simple Rasch model and Partial Credit Rasch model to reanalyze the data. Consequently, the latent trait estimates as well as their error of measurement were transformed back to the raw score scale. We compared IRT and CTT based errors of measurement on the same scale for all observed raw scores according to the distribution of the standardization sample. In most subtests the error of estimate based on CTT was smaller around the average score but bigger in the extremes, especially at the low end of the scale (DISMAS is designed to test disability, so it has a high ceiling effect). We conclude that IRT analysis produces more accurate estimate for children with math disability, for whom the test is designed, so in this case Rasch model seems to be more appropriate method to estimate confidence intervals than CTT. Finally, we discuss other sources of relatively low reliability.
ER  -

CÍGLER, Hynek, Michal JABŮREK a Jan ŠIRŮČEK. Reanalyzing the DISMAS Test Data: Comparing IRT and CTT Based Estimates of the Error of Measurement. In \textit{The 9th Conference of The International Test Commission : Global and Local Challenges for Best Practices in Assessment}. 2014. ISBN~978-84-9860-997-4.

Podrobný výpis o publikaci