Three dimensional partial credit model won't run

[gsbeglia]

Hi all. I'm trying to run a three dimensional between item partial credit model for a 24 item instrument on ConQuest4. I was able to run the one dimensional model but when I try to run the three dimensional model, it runs for several hours, generates the itn and wle files and then ConQuest crashes. This three dimensional model worked fine on our old version of ConQuest (version 1). The syntax is below and the data set is attached. Any help would be much appreciated.

Thanks,
Gena

datafile pre_test_121917.dat;
format pid 1-11 responses 14-37;
codes 1,2,3,4,5;
set constraints=cases, warnings=no;
score (1,2,3,4,5) (1,2,3,4,5) ()           ()          !items(1-8);
score (1,2,3,4,5)   ()        (1,2,3,4,5)  ()          !items(9-16);
score (1,2,3,4,5) ()         ()            (1,2,3,4,5) !items(17-24);
Model item + item*step;
export parameters            >> pre_121917_D3.prm;
export reg_coefficients      >> pre_121917_D3.reg;
export covariance            >> ipre_121917_D3.cov;
Estimate !method=montecarlo, nodes=2000, converge = .005;
itanal                       >> pre_121917_D3.itn;
show cases !estimate=wle     >> pre_121917_D3.wle;
show !estimate=latent        >> pre_121917_D3.shw;
equivalence wle           >> pre_equivalence_121917_D3.wle;
plot icc;

[dan_c]

I ran your model in ConQuest Mac:

Code: [Select]

< about;
Version: 4.18.2
Professional Build:  Jan  6 2018
See the attached output - is this what you expect?

Can you post the version you are using (seems to be Win GUI version? Have you run at the console?) so we can narrow down your issue?

[gsbeglia]

Hi Dan,

Thank you for the reply. I am using Conquest version 4.14.2 and I am running the GUI version. I have not tried running at the console but am running the code on it now to see if this fixes the problem.

When I ran the model requesting only the shw file (not the wle or itn file), I was able to get an output that looks similar to the one you sent and has a similar final deviance (105017.74117). However, the WLE person separation reliability is unavailable in my output even though it is available in your output of the same model.

Something that seems strange to me about the outcome of the three dimensional model is that the final deviance is much higher than for the one-dimensional model using the same data and the code below. I thought shouldn't happen because models with more parameters will always have better fit.



Title pre_test_121917 1D;
datafile pre_test_121917.dat;
format pid 1-11 responses 14-37;
codes 1,2,3,4,5;
Labels << labels_ISEA.lab;
Model item + item*step;
Estimate ;
itanal                          >> pre_121917_D1.itn;
show cases !estimate=wle        >> ipre_121917_D1.wle;
show !estimate=latent           >> pre_121917_D1.shw;

[dan_c]

Great - let me know how you go with the console version. Thanks for helping work out where the crash is coming from.

I ran your 1 and 3 dimensional models with some minor changes (motecarlo estimation, the same model constraint, and the same coding of scores starting at 0 for both models), and have attached the output. The WLE reliabilities won't be available unless you estimate WLE factor scores (e.g., show cases !estimate=wle). The reason your final deviance is higher for the less-contstrained, 3 dimensional model is because you use montecarlo estimation for the 3 dimensional model and gaussian for the 1 dimensional model. See the attached, where I use motecarlo for both.

[gsbeglia]

Thanks Dan. The console version of Conquest is not running to completion. After running for several hours, it stopped while computing empirical errors.

I have a few questions and observations about the code you sent:

1) The GUI is mostly working for the code you sent as long as I run each model separately. When I run both models together, the three dimensional model runs fine and then after the itn file is produced for the one dimensional model, I get the error that "Matrices not the same dimension - can't add".

2) I'm confused as to the use of 0 in the score code. A 0 indicates missing data in my data set, which is why I left it out of my code statement. Must 0 be included in the code statement of a partial credit model? Should I change the missing data to something else instead? When i try to run this model with missing data scored as 9, the wle file is incorrect (raw total scores are very off) and the program crashes before generating the three dimensional shw file.

3) Also, I noticed that your three-dimensional model output says "At termination the solution was not the best attained solution. The reported results are for the earlier better solution. Rerunning this analysis using the current estimates as initial values is strongly advised." Did the model converge? Must I re-run the analysis in order to use the item and person scores in parametric statistics? How do I modify the code to do this?

[dan_c]

1) ... I get the error that "Matrices not the same dimension - can't add".

Okay - this is a potential bug. Let me follow it up and get back to you.

2) I'm confused as to the use of 0 in the score code. A 0 indicates missing data in my data set, which is why I left it out of my code statement. Must 0 be included in the code statement of a partial credit model? Should I change the missing data to something else instead? When i try to run this model with missing data scored as 9, the wle file is incorrect (raw total scores are very off) and the program crashes before generating the three dimensional shw file.

you don't need to include 0 in your code statement, this is just my natural reflex. I would, however, suggest that you not use the value 0 as a missing value, for two reasons. The first is that its not very intuitive - people tend to expect missing values to take values like NA, ., -9, 999, etc. the second is that using 0 as your first response category makes your itanal output more interpretable. for example, you will get estimates thresholds (see the attached examples). You can recode your data using something like:

Code: [Select]

recode (0,1,2,3,4,5) (99,0,1,2,3,4);
codes 0,1,2,3,4;
This will make sure your output is correct, as 99 is not in the codes list, it is considered to be missing by ConQuest.

I do, however, notice that the 1 dimensional model fails when you remove the missing responses (variance of dim 1 becomes negative). This is perhaps due to model misspecification, but also possibly to do with response patterns - you tend to have few observations in your first 2 or 3 response categories. Collapsing response categories works, but whether is appropriate is a judgement call for you:

Code: [Select]

recode (0,1,2,3,4,5) (99,0,0,0,1,2);
codes 0,1,2;

3) Also, I noticed that your three-dimensional model output says "At termination the solution was not the best attained solution. The reported results are for the earlier better solution. Rerunning this analysis using the current estimates as initial values is strongly advised." Did the model converge? Must I re-run the analysis in order to use the item and person scores in parametric statistics? How do I modify the code to do this?

Yes, the model converged (in the sense that it met your convergence criteria - largest change in any parameter estimate between successive iterations was less than 0.005). You should review the iterations using the history option to the estimatecommand  to check that your parameters are behaving sensibly (e.g., not jumping around wildly). You could run the model again with more iterations, a more stringent convergence criterion, and/or importing start values for your parameters. This would be a judgment call by you - this approach won't get around issues like very sparse data, model misspecification, etc.