Very different WLE Person separation reliability and EAP/PV reliability

[Anne Ham]

Hi,
I am running a 3 dimensional model and get very different values for the WLE Person separation reliability and EAP/PV reliability for two of the dimensions (dimension 2: WLE 0.124, EAP/PV 0.738; dimension 3: WLE 0.056, EAP/PV 0.734).
Is there something wrong with my data/syntax? Or does this mean that  the abilities of individual students are not trustworthy but population parameters can be well estimated?

codes 0,1,2,3;
recode (0,1,2,3) (0,0,1,2) !items (1);
recode (0,1,2)   (0,0,1)   !items (5,30);

score (0,1,2) (0,0.5,1) (   ) (   )   !items (1);
score (0,1)   (0,1)     (   ) (   )   !items (2,6,7,8,12,16,19,20,23,25,26,31,32,35,39,42,43,44,45,48);
score (0,1)   (0,1)     (   ) (   )   !items (4,17,18,34,38,46,49);
score (0,1,2) (0,0.5,1) (   ) (   )   !items (11,47);
score (0,1)   (0,1)     (   ) (   )   !items (10,21,24,33,37);
score (0,1)   (   )     (0,1) (   )   !items (3,5,9,14,27,36,40);
score (0,1)   (   )     (   ) (0,1)  !items (13,15,22,28,29,30,41);

model item + item*step;
set constraint=cases;
estimate   !  method=montecarlo, nodes=5000, converge=.001, stderr=none;

Many thanks!
Anne

[dan_c]

we will need to see you data and outputs to check this. Can you also let us know what version of ConQuest you are running?

Alternatively, send through a reproducible example: https://conquest-forums.acer.edu.au/index.php?topic=1239.0

Note, you don't technically need your recode AND your score statements, see: https://conquestmanual.netlify.app/s4-00.html#score

Code: [Select]

codes 0,1,2,3;

score (0,1,2,3) (0,0,0.5,1) (   )   (   )   !items (1);
score (0,1)     (0,1)       (   )   (   )   !items (2,6,7,8,12,16,19,20,23,25,26,31,32,35,39,42,43,44,45,48);
score (0,1)     (0,1)       (   )   (   )   !items (4,17,18,34,38,46,49);
score (0,1,2)   (0,0.5,1)   (   )   (   )   !items (11,47);
score (0,1)     (0,1)       (   )   (   )   !items (10,21,24,33,37);
score (0,1,2)   (   )       (0,0,1) (   )   !items (5);
score (0,1)     (   )       (0,1)   (   )   !items (3,5,9,14,27,36,40);
score (0,1)     (   )       (   )   (0,1)   !items (13,15,22,28,29,30,41);
score (0,1,2)   (   )       (   )   (0,0,1) !items (30);


This is will give your items 1, 5, and 30, properties like in the ordered partition model.

[dan_c]

Thanks for the files.

I'd strongly encourage you to upgrade to a more recent version of ConQuest (version 5 is new this year).

Your model seems to run fine and converges happily. The estimated reliabilities are the same in a new build of ConQuest (attached).

The WLE reliability is low primarily because there are very few items in dimensions 2 and 3. Reviewing your WLE file, there are many cases where the max possible score for dimension 2 and 3 is 4.00. This can be interpreted as meaning there is little information available to space cases out along the latent trait. Your PV reliability is getting a boost because of the multidimenisfal structure. Your latent traits are highly correlated (> 0.8 ) and provide a lot of information that reduces the uncertainty in dimension 2 and 3. I think it's fair to conclude that this data is more useful for estimation of population parameters than individual abilities.

 

[Anne Ham]

Thank you so much for looking at my model and your advice!