Author Topic: Very different WLE Person separation reliability and EAP/PV reliability  (Read 411 times)

Anne Ham

  • Newbie
  • *
  • Posts: 2
    • View Profile
    • Email
Hi,
I am running a 3 dimensional model and get very different values for the WLE Person separation reliability and EAP/PV reliability for two of the dimensions (dimension 2: WLE 0.124, EAP/PV 0.738; dimension 3: WLE 0.056, EAP/PV 0.734).
Is there something wrong with my data/syntax? Or does this mean that  the abilities of individual students are not trustworthy but population parameters can be well estimated?

codes 0,1,2,3;
recode (0,1,2,3) (0,0,1,2) !items (1);
recode (0,1,2)   (0,0,1)   !items (5,30);

score (0,1,2) (0,0.5,1) (   ) (   )   !items (1);
score (0,1)   (0,1)     (   ) (   )   !items (2,6,7,8,12,16,19,20,23,25,26,31,32,35,39,42,43,44,45,48);
score (0,1)   (0,1)     (   ) (   )   !items (4,17,18,34,38,46,49);
score (0,1,2) (0,0.5,1) (   ) (   )   !items (11,47);
score (0,1)   (0,1)     (   ) (   )   !items (10,21,24,33,37);
score (0,1)   (   )     (0,1) (   )   !items (3,5,9,14,27,36,40);
score (0,1)   (   )     (   ) (0,1)  !items (13,15,22,28,29,30,41);

model item + item*step;
set constraint=cases;
estimate   !  method=montecarlo, nodes=5000, converge=.001, stderr=none;

Many thanks!
Anne


dan_c

  • Administrator
  • Jr. Member
  • *****
  • Posts: 89
    • View Profile
    • Email
we will need to see you data and outputs to check this. Can you also let us know what version of ConQuest you are running?

Alternatively, send through a reproducible example: https://conquest-forums.acer.edu.au/index.php?topic=1239.0

Note, you don't technically need your recode AND your score statements, see: https://conquestmanual.netlify.app/s4-00.html#score

Code: [Select]
codes 0,1,2,3;

score (0,1,2,3) (0,0,0.5,1) (   )   (   )   !items (1);
score (0,1)     (0,1)       (   )   (   )   !items (2,6,7,8,12,16,19,20,23,25,26,31,32,35,39,42,43,44,45,48);
score (0,1)     (0,1)       (   )   (   )   !items (4,17,18,34,38,46,49);
score (0,1,2)   (0,0.5,1)   (   )   (   )   !items (11,47);
score (0,1)     (0,1)       (   )   (   )   !items (10,21,24,33,37);
score (0,1,2)   (   )       (0,0,1) (   )   !items (5);
score (0,1)     (   )       (0,1)   (   )   !items (3,5,9,14,27,36,40);
score (0,1)     (   )       (   )   (0,1)   !items (13,15,22,28,29,30,41);
score (0,1,2)   (   )       (   )   (0,0,1) !items (30);


This is will give your items 1, 5, and 30, properties like in the ordered partition model.


dan_c

  • Administrator
  • Jr. Member
  • *****
  • Posts: 89
    • View Profile
    • Email
Thanks for the files.

I'd strongly encourage you to upgrade to a more recent version of ConQuest (version 5 is new this year).

Your model seems to run fine and converges happily. The estimated reliabilities are the same in a new build of ConQuest (attached).

The WLE reliability is low primarily because there are very few items in dimensions 2 and 3. Reviewing your WLE file, there are many cases where the max possible score for dimension 2 and 3 is 4.00. This can be interpreted as meaning there is little information available to space cases out along the latent trait. Your PV reliability is getting a boost because of the multidimenisfal structure. Your latent traits are highly correlated (> 0.8 ) and provide a lot of information that reduces the uncertainty in dimension 2 and 3. I think it's fair to conclude that this data is more useful for estimation of population parameters than individual abilities.

 
« Last Edit: May 29, 2020, 01:47:21 AM by dan_c »

Anne Ham

  • Newbie
  • *
  • Posts: 2
    • View Profile
    • Email
Thank you so much for looking at my model and your advice!

Isa89

  • Newbie
  • *
  • Posts: 6
    • View Profile
    • Email
Hey,

I seem to have a very similar problem: okish EAPs/PVs and very low WLEs. Can you probably also have a look at my example? Are there other possible explanations for low WLEs than the boost of the multidimensional model?

And one more question: I have conducted a pre-post-study, so I want to show learning effects. Is it possible to show these effects for groups using the WLE scores (rather than for individuals), even though the WLE reliabilities are that bad?

Thanks a lot in advance!
Isa

dan_c

  • Administrator
  • Jr. Member
  • *****
  • Posts: 89
    • View Profile
    • Email
Hi Isa,

I can only see your labels file attached. Can you attach the rest of your files?

If you are trying to explain a population parameter, like average learning gain over time, I would recommend you use PVs. Point estimates, including EAPs will result in biased secondary analysis. See for example:

https://doi.org/10.1016/j.stueduc.2005.05.005

Isa89

  • Newbie
  • *
  • Posts: 6
    • View Profile
    • Email
Hey Dan,

see attached my syntax as well as my datafile. Thanks for your help!

Isa

dan_c

  • Administrator
  • Jr. Member
  • *****
  • Posts: 89
    • View Profile
    • Email
Hi Isa,

If you have pre and post data, I would suggest estimating the growth in a single IRT model. One way is to create a wide file, load time 1 items onto one latent, and time 2 items onto a second latent, and anchor the difficulties of the items to be equal. The growth is then a function of the theta (theta2 - theta1 = growth). If you want to analyse the growth (e.g., regress covariates on the growth) you could choose a slightly different parameterisation, or you could then take PVs out of ConQuest and and fit whatever model (ANCOVA, mixed effects etc) you like in another stats package.

If you want to do it all in ConQuest, I would encourage you to read this article:

Quote
Wilson, M., Zheng, X., & McGuire, L. (2012). Formulating latent growth using an explanatory item response model approach. Journal of Applied Measurement, 13(1), 1.

I ran your model, and I think the difference between PV and WLE reliabilities is as described above. You have:

- Dim 1 and 2 have items that are very easy
- some cases have no observations on a dimension and therefore a WLE cannot be calculated
- generally few items per dimension (see Spearman-Brown prophecy formula to think about what the maximum reliability you might expect is)

Isa89

  • Newbie
  • *
  • Posts: 6
    • View Profile
    • Email
Dear Dan,

thanks SO MUCH for your help, ideas and the article you sent me!

Kind regards,
Isabel