Very different WLE Person separation reliability and EAP/PV reliability

[Anne Ham]

Hi,
I am running a 3 dimensional model and get very different values for the WLE Person separation reliability and EAP/PV reliability for two of the dimensions (dimension 2: WLE 0.124, EAP/PV 0.738; dimension 3: WLE 0.056, EAP/PV 0.734).
Is there something wrong with my data/syntax? Or does this mean that  the abilities of individual students are not trustworthy but population parameters can be well estimated?

codes 0,1,2,3;
recode (0,1,2,3) (0,0,1,2) !items (1);
recode (0,1,2)   (0,0,1)   !items (5,30);

score (0,1,2) (0,0.5,1) (   ) (   )   !items (1);
score (0,1)   (0,1)     (   ) (   )   !items (2,6,7,8,12,16,19,20,23,25,26,31,32,35,39,42,43,44,45,48);
score (0,1)   (0,1)     (   ) (   )   !items (4,17,18,34,38,46,49);
score (0,1,2) (0,0.5,1) (   ) (   )   !items (11,47);
score (0,1)   (0,1)     (   ) (   )   !items (10,21,24,33,37);
score (0,1)   (   )     (0,1) (   )   !items (3,5,9,14,27,36,40);
score (0,1)   (   )     (   ) (0,1)  !items (13,15,22,28,29,30,41);

model item + item*step;
set constraint=cases;
estimate   !  method=montecarlo, nodes=5000, converge=.001, stderr=none;

Many thanks!
Anne

[dan_c]

we will need to see you data and outputs to check this. Can you also let us know what version of ConQuest you are running?

Alternatively, send through a reproducible example: https://conquest-forums.acer.edu.au/index.php?topic=1239.0

Note, you don't technically need your recode AND your score statements, see: https://conquestmanual.netlify.app/s4-00.html#score

Code: [Select]

codes 0,1,2,3;

score (0,1,2,3) (0,0,0.5,1) (   )   (   )   !items (1);
score (0,1)     (0,1)       (   )   (   )   !items (2,6,7,8,12,16,19,20,23,25,26,31,32,35,39,42,43,44,45,48);
score (0,1)     (0,1)       (   )   (   )   !items (4,17,18,34,38,46,49);
score (0,1,2)   (0,0.5,1)   (   )   (   )   !items (11,47);
score (0,1)     (0,1)       (   )   (   )   !items (10,21,24,33,37);
score (0,1,2)   (   )       (0,0,1) (   )   !items (5);
score (0,1)     (   )       (0,1)   (   )   !items (3,5,9,14,27,36,40);
score (0,1)     (   )       (   )   (0,1)   !items (13,15,22,28,29,30,41);
score (0,1,2)   (   )       (   )   (0,0,1) !items (30);


This is will give your items 1, 5, and 30, properties like in the ordered partition model.

[dan_c]

Thanks for the files.

I'd strongly encourage you to upgrade to a more recent version of ConQuest (version 5 is new this year).

Your model seems to run fine and converges happily. The estimated reliabilities are the same in a new build of ConQuest (attached).

The WLE reliability is low primarily because there are very few items in dimensions 2 and 3. Reviewing your WLE file, there are many cases where the max possible score for dimension 2 and 3 is 4.00. This can be interpreted as meaning there is little information available to space cases out along the latent trait. Your PV reliability is getting a boost because of the multidimenisfal structure. Your latent traits are highly correlated (> 0.8 ) and provide a lot of information that reduces the uncertainty in dimension 2 and 3. I think it's fair to conclude that this data is more useful for estimation of population parameters than individual abilities.

 

[Anne Ham]

Thank you so much for looking at my model and your advice!

[Isa89]

Hey,

I seem to have a very similar problem: okish EAPs/PVs and very low WLEs. Can you probably also have a look at my example? Are there other possible explanations for low WLEs than the boost of the multidimensional model?

And one more question: I have conducted a pre-post-study, so I want to show learning effects. Is it possible to show these effects for groups using the WLE scores (rather than for individuals), even though the WLE reliabilities are that bad?

Thanks a lot in advance!
Isa

[dan_c]

Hi Isa,

I can only see your labels file attached. Can you attach the rest of your files?

If you are trying to explain a population parameter, like average learning gain over time, I would recommend you use PVs. Point estimates, including EAPs will result in biased secondary analysis. See for example:

https://doi.org/10.1016/j.stueduc.2005.05.005

[Isa89]

Hey Dan,

see attached my syntax as well as my datafile. Thanks for your help!

Isa

[dan_c]

Hi Isa,

If you have pre and post data, I would suggest estimating the growth in a single IRT model. One way is to create a wide file, load time 1 items onto one latent, and time 2 items onto a second latent, and anchor the difficulties of the items to be equal. The growth is then a function of the theta (theta2 - theta1 = growth). If you want to analyse the growth (e.g., regress covariates on the growth) you could choose a slightly different parameterisation, or you could then take PVs out of ConQuest and and fit whatever model (ANCOVA, mixed effects etc) you like in another stats package.

If you want to do it all in ConQuest, I would encourage you to read this article:

Wilson, M., Zheng, X., & McGuire, L. (2012). Formulating latent growth using an explanatory item response model approach. Journal of Applied Measurement, 13(1), 1.

I ran your model, and I think the difference between PV and WLE reliabilities is as described above. You have:

- Dim 1 and 2 have items that are very easy
- some cases have no observations on a dimension and therefore a WLE cannot be calculated
- generally few items per dimension (see Spearman-Brown prophecy formula to think about what the maximum reliability you might expect is)

[Isa89]

Dear Dan,

thanks SO MUCH for your help, ideas and the article you sent me!

Kind regards,
Isabel