DISMAS test (Traspe & Skalkova, 2013) is an original Czech assessment tool for the diagnosis of the structure of math disability in children in grades 1–5. It consists of five main areas divided into 14 unidimensional subtests and it was developed and standardized using classical test theory. Authors verified validity. Subtests’ correlations with WISC-III were low, but significant, differences between normal and clinical population were high and significant for all grades. Regardless the high validity, reliability seemed to be poor – Cronbach alpha ranged from .44 to .95 on subtest level, on the test level from .69 to .76 depending on the grade level. The aim of this paper is to show why. We questioned the adequacy of CTT and internal consistency as a method for estimating reliability in this case. That is why we applied simple Rasch model and Partial Credit Rasch model to reanalyze the data. Consequently, the latent trait estimates as well as their error of measurement were transformed back to the raw score scale. We compared IRT and CTT based errors of measurement on the same scale for all observed raw scores according to the distribution of the standardization sample. In most subtests the error of estimate based on CTT was smaller around the average score but bigger in the extremes, especially at the low end of the scale (DISMAS is designed to test disability, so it has a high ceiling effect). We conclude that IRT analysis produces more accurate estimate for children with math disability, for whom the test is designed, so in this case Rasch model seems to be more appropriate method to estimate confidence intervals than CTT. Finally, we discuss other sources of relatively low reliability.