Not checkedNot checked for notices yet.
Comparison of two Bayesian methods to detect mode effects between paper-based and computerized adaptive assessments: a preliminary Monte Carlo study
Riley Barth B, Carle Adam C
BMC Medical Research Methodology · 2012
doi:10.1186/1471-2288-12-124 (opens in a new tab)
We email you when a new notice or a question about this paper appears.
Abstract
<p>Abstract</p> <p>Background</p> <p>Computerized adaptive testing (CAT) is being applied to health outcome measures developed as paper-and-pencil (P&P) instruments. Differences in how respondents answer items administered by CAT vs. P&P can increase error in CAT-estimated measures if not identified and corrected.</p> <p>Method</p> <p>Two methods for detecting item-level mode effects are proposed using Bayesian estimation of posterior distributions of item parameters: (1) a modified robust <it>Z</it> (<it>RZ</it>) test, and (2) 95% credible intervals (<it>CrI</it>) for the CAT-P&P difference in item difficulty. A simulation study was conducted under the following conditions: (1) data-generating model (one- vs. two-parameter IRT model); (2) moderate vs. large DIF sizes; (3) percentage of DIF items (10% vs. 30%), and (4) mean difference in <it>θ</it> estimates across modes of 0 vs. 1 logits. This resulted in a total of 16 conditions with 10 generated datasets per condition.</p> <p>Results</p> <p>Both methods evidenced good to excellent false positive control, with <it>RZ</it> providing better control of false positives and with slightly higher power for <it>CrI</it>, irrespective of measurement model. False positives increased when items were very easy to endorse and when there with mode differences in mean trait level. True positives were predicted by CAT item usage, absolute item difficulty and item discrimination. <it>RZ</it> outperformed <it>CrI</it>, due to better control of false positive DIF.</p> <p>Conclusions</p> <p>Whereas false positives were well controlled, particularly for <it>RZ</it>, power to detect DIF was suboptimal. Research is needed to examine the robustness of these methods under varying prior assumptions concerning the distribution of item and person parameters and when data fail to conform to prior assumptions. False identification of DIF when items were very easy to endorse is a problem warranting additional investigation.</p>
Abstract licence: CC BY-NC-ND
Reads
Read by people on this page
None yet.
Checked or read by AI through naulon
None yet.
A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.
Are you an author?
Is this your paper?
Claim it to add your note and be paid when agents cite it. Free.