Author Topic: Fit indices in ConQuest  (Read 893 times)

navid

  • Newbie
  • *
  • Posts: 7
    • View Profile
    • Email
Fit indices in ConQuest
« on: November 29, 2013, 10:49:10 AM »
Hi,
We are doing a ConQuest analysis of a language proficiency test. We intend
to compare four models: a unidimensiona Rasch model, an MRCMLM with 6
factors and between-item multidimensionality, an MRCMLM with 5 factors and
between-item multidimensionality, and a testlet model (with one general
dimension and six specific factors). We want to compare the relative fits
of the models. I have a couple of questions and would be grateful to
receive an explanation from you.

1. Is the deviance statistic reported in ConQuest equal to the -2 Log
Likelihood statistic? Is it the same as G2 reported in Rasch papers?

2. How is the AIC and BIC statistics computed from deviance? (I know these can be estimated from chi-square but is an overall chi-square reported in ConQuest?)
3. Can we consider the testlet model to also have a hierarchical relation
to other models in our study? (so that we can compare their relative fit using the difference in their deviance indices).

Many thanks in advance.

Navid
« Last Edit: November 29, 2013, 10:51:11 AM by navid »

georgeha

  • Newbie
  • *
  • Posts: 5
    • View Profile
    • Email
Re: Fit indices in ConQuest
« Reply #1 on: December 02, 2013, 12:57:07 AM »
Hi Navid,

I'll give my best shot at answering your questions and see if anyone negates what I say.

1) Yes, the deviance statistic is the -2LL estimate (there's no need to take its log or multiply it by -2).
2) AIC = deviance + 2(n_parameters). BIC = deviance + Ln(N_persons)(n_parameters).
3) Good question.  Does anyone know?  It would make sense to me to consider the testlet model as nested since you're not modeling within-item multidimensionality. However, I'm not sure about comparing deviance statistics when you model inter-item dependence. I hope someone answers this.

George

navid

  • Newbie
  • *
  • Posts: 7
    • View Profile
    • Email
Re: Fit indices in ConQuest
« Reply #2 on: December 06, 2013, 03:41:19 PM »
Thank you George for your response.
As for the third question, we are fitting a testlet model which I assume falls within the category of within-item multidimensional models. Doesn't it? Can we still use the difference between deviances to compare relative fit?

Navid

Eveline Gebhardt

  • Administrator
  • Full Member
  • *****
  • Posts: 103
    • View Profile
    • Email
Re: Fit indices in ConQuest
« Reply #3 on: December 10, 2013, 09:16:44 PM »
1) Deviance  = -2 * loglikelihood. I don’t know what G2 is.
 
2) AIC  = 2* number of parameters – deviance
BIC = deviance + number of parameters * ln(number of cases)  ……  but I am not sure this is applicable
 
3) Yes, the testlet model is possible.  To decide if one model is a special case of another, you ask the question can I get from one to the other by setting some parameters to a fixed value or equal to each other.  Eg if you have two dimensional then set correlation=1 and you have one dimensional

Shirley

  • Newbie
  • *
  • Posts: 2
    • View Profile
Re: Fit indices in ConQuest
« Reply #4 on: March 04, 2016, 08:14:52 AM »
If the number of cases differs across items (e.g., some participants skipped some items), is the total number of participants in the data set being used as number of cases in the calculation of BIC? 


Eveline Gebhardt

  • Administrator
  • Full Member
  • *****
  • Posts: 103
    • View Profile
    • Email
Re: Fit indices in ConQuest
« Reply #5 on: March 18, 2016, 12:43:08 AM »
Hi Shirley

BIC is not computed by ConQuest but AIC is. The number of cases are not required for computing AIC.

Best wishes
Eveline

Isa89

  • Newbie
  • *
  • Posts: 6
    • View Profile
    • Email
Re: Fit indices in ConQuest
« Reply #6 on: October 23, 2020, 01:46:42 PM »
Hello everyone,

Is there a way to make ConQuest compute BIC and what do I need to write into my snytax then?

Best,
Isa

dan_c

  • Administrator
  • Jr. Member
  • *****
  • Posts: 89
    • View Profile
    • Email
Re: Fit indices in ConQuest
« Reply #7 on: November 09, 2020, 10:39:26 PM »
Isa,

BIC is reported in the output from the SHOW command in ConQuest Version 5. See example from Example 1 (https://www.acer.org/au/conquest/notes-tutorials):

Code: [Select]
The Data File: ex1.dat
The format:  id 1-5 responses 12-23
No case weights
The regression model:
Grouping Variables:
The item model: item
Slopes are fixed
Cases in file: 1000  Cases in estimation: 1000
Final Deviance:                                13274.87615
Akaike Information Criterion (AIC):            13300.87615
Akaike Information Criterion Corrected (AICc): 13300.56785
Bayesian Information Criterion (BIC):          13364.67697
Total number of estimated parameters: 13
The number of iterations: 45
Termination criteria:  Max iterations=1000, Parameter Change= 0.00010
                       Deviance Change= 0.00010

You can download ConQuest 5 form the shop: https://shop.acer.edu.au/acer-conquest-5.html