Fit indices in ConQuest

[navid]

Hi,
We are doing a ConQuest analysis of a language proficiency test. We intend
to compare four models: a unidimensiona Rasch model, an MRCMLM with 6
factors and between-item multidimensionality, an MRCMLM with 5 factors and
between-item multidimensionality, and a testlet model (with one general
dimension and six specific factors). We want to compare the relative fits
of the models. I have a couple of questions and would be grateful to
receive an explanation from you.

1. Is the deviance statistic reported in ConQuest equal to the -2 Log
Likelihood statistic? Is it the same as G2 reported in Rasch papers?

2. How is the AIC and BIC statistics computed from deviance? (I know these can be estimated from chi-square but is an overall chi-square reported in ConQuest?)
3. Can we consider the testlet model to also have a hierarchical relation
to other models in our study? (so that we can compare their relative fit using the difference in their deviance indices).

Many thanks in advance.

Navid

[georgeha]

Hi Navid,

I'll give my best shot at answering your questions and see if anyone negates what I say.

1) Yes, the deviance statistic is the -2LL estimate (there's no need to take its log or multiply it by -2).
2) AIC = deviance + 2(n_parameters). BIC = deviance + Ln(N_persons)(n_parameters).
3) Good question.  Does anyone know?  It would make sense to me to consider the testlet model as nested since you're not modeling within-item multidimensionality. However, I'm not sure about comparing deviance statistics when you model inter-item dependence. I hope someone answers this.

George

[navid]

Thank you George for your response.
As for the third question, we are fitting a testlet model which I assume falls within the category of within-item multidimensional models. Doesn't it? Can we still use the difference between deviances to compare relative fit?

Navid

[Eveline Gebhardt]

1) Deviance  = -2 * loglikelihood. I don’t know what G2 is.
 
2) AIC  = 2* number of parameters – deviance
BIC = deviance + number of parameters * ln(number of cases)  ……  but I am not sure this is applicable
 
3) Yes, the testlet model is possible.  To decide if one model is a special case of another, you ask the question can I get from one to the other by setting some parameters to a fixed value or equal to each other.  Eg if you have two dimensional then set correlation=1 and you have one dimensional

[Shirley]

If the number of cases differs across items (e.g., some participants skipped some items), is the total number of participants in the data set being used as number of cases in the calculation of BIC? 

[Eveline Gebhardt]

Hi Shirley

BIC is not computed by ConQuest but AIC is. The number of cases are not required for computing AIC.

Best wishes
Eveline

[Isa89]

Hello everyone,

Is there a way to make ConQuest compute BIC and what do I need to write into my snytax then?

Best,
Isa

[dan_c]

Isa,

BIC is reported in the output from the SHOW command in ConQuest Version 5. See example from Example 1 (https://www.acer.org/au/conquest/notes-tutorials):

Code: [Select]

The Data File: ex1.dat
The format:  id 1-5 responses 12-23
No case weights
The regression model:
Grouping Variables:
The item model: item
Slopes are fixed
Cases in file: 1000  Cases in estimation: 1000
Final Deviance:                                13274.87615
Akaike Information Criterion (AIC):            13300.87615
Akaike Information Criterion Corrected (AICc): 13300.56785
Bayesian Information Criterion (BIC):          13364.67697
Total number of estimated parameters: 13
The number of iterations: 45
Termination criteria:  Max iterations=1000, Parameter Change= 0.00010
                       Deviance Change= 0.00010
You can download ConQuest 5 form the shop: https://shop.acer.edu.au/acer-conquest-5.html