Author Topic: Interpretation of the reliabilities reported by Conquest  (Read 68 times)

JerredJolin

  • Newbie
  • *
  • Posts: 7
    • View Profile
    • Email
Interpretation of the reliabilities reported by Conquest
« on: August 06, 2021, 10:11:22 PM »
Hello,

I have a question regarding the reliabilities reported by Conquest 4. What are the interpretational differences between MLE person separation, WLE person separation, and EAP/PV person separation reliabilities?

Is it convention to choose the category of reliability that matches the estimation model used (i.e., since I used WLE, use the WLE reliability index)?

Thanks,

-Jerred

dan_c

  • Administrator
  • Full Member
  • *****
  • Posts: 114
    • View Profile
    • Email
Re: Interpretation of the reliabilities reported by Conquest
« Reply #1 on: August 09, 2021, 03:59:35 AM »
ConQuest reports 4 kinds of reliability:

  • item separation reliability (displayed in Show files, under the item parameters in the "ResponseModel" table - Table 2). This is the proportion of observed item variance not due to error variance = 1 - (MSE(items)/Var(items)). There is a very good description on page 92 of Rating Scale Analysis https://research.acer.edu.au/measurement/2/
  • WLE/MLE reliability (displayed in Show files, under in the "Reliabilities" table - Table 3). This has the same interpretation as above, but for persons rather than items and can be found on p 105 of Rating Scale Analysis https://research.acer.edu.au/measurement/2/. Another good discussion is here: https://doi.org/10.1016/j.stueduc.2005.05.008 (see Eq 6).
  • PV reliability is a reliability index for Marginal IRT models where the person abilities are not parameters of the model and may not be computed at all. PV reliability is the average ratio of the case posterior variance over the population variance. This means that the regression model is involved in the reliability measure in addition too the item responses, whereas the person separation reliability only involves the item responses. If the regression model has even modest explanatory power, this will increase the reliability. Also discussed in detail here: https://doi.org/10.1016/j.stueduc.2005.05.008 (see Eq 9 onwards).

You are right about choosing to report W/MLE reliability based on the case estimates you generate. Whether you also report PV reliability will typically depend on what you are trying to argue - if you are reporting population parameters (means, variances, sub-group deviations from the mean etc) then PV reliability is relevant. If you are reporting the uncertainty of the test scores you are giving to individuals, then it is less relevant.