Differences between version

[IsabelI]

Good morning everyone!

Thank you for accepting me to be part of the community! I am happy to be here to share issues with other Conquests users!
My question today is about differences between versions. One colleague and myself are replicating the same analyses for getting PV. However, we find differences in the results we are getting although we are using the same syntax and the same data set. The only difference between both processes is the Conquest version (one from December 2012 and one from February 2007). We have been checking default options in order to find the reason but everything seems to be equal. Do you know why is it happening?
Thank you very much and have a good day!
Isabel

[Eveline Gebhardt]

Hi Isabel

Could you please send me the two versions of the output files?

Cheers
Eveline

[IsabelI]

Dear Eveline,

Here you have the outputs! We have three version of results for the same database and the same syntax, but we cannot figure out about differences. in the attachment you will see three folders for the situations! We are finding differences also when using the same version of the software.

Thank you for your help!

[Eveline Gebhardt]

Dear Isabel

Thank you for your post and for joining the forum.

I have carefully checked your output files and can confirm that nothing is wrong with the estimation. The differences may look large, but they are small when taking the uncertainties in the estimation into account. I'll explain.

Fit statistics are computed from plausible values. Plausible values are random draws and will be different every time ConQuest estimates the model. The degree of variation between different runs depends on the size of the measurement error. Measurement errors are larger for smaller samples and for models without regressors. While the standard number of plausible values for case estimates is 5, only one value is drawn to compute fit statistics. Therefore, if the errors are large, the difference in fit statistics may appear to vary between runs. However, if you look at the t-values of the fit statistics, the conclusion stays usually the same. For example, if you look at your first item, the weighted fit appears quite large in the 2007 and the 2012A versions (larger than 1.2) and smaller (1.13) in the 2012B version (it actually is the Nov 2015 version), but when you look at the t-values you'll see they are all smaller than 2 and therefore not significantly different from a value of 1.

You can decrease the fluctuation for small samples by increasing the fit draws (set fitdraws=n;). The default for n is 1. You could use 5 for n or even 10.

I hope this answers your question. Please let us know if you have any further queries.

Eveline

[IsabelI]

Dear Eveline,

Thank you for your clear response. I know that there are fluctuations between plausible values between different runs and it makes sense if we consider the concept of "plausible values". However, the variations in the values are having a large impact in my results. After using PV for calculating total scores I find differences of between 5 and 35 points at the student level. But, when I group students according to different variables I still find differences of between 8 and 20 points. I know that it is "normal" but for my project these differences are very large as we try to provide accurate results and it is difficult to justify these results depend on the software.... Nevertheless, I perfectly understand the explanation and I am relaxed now because you confirmed I am not making mistakes, so thank you very much for the information.

Best regards,
Isabel

[Eveline Gebhardt]

Hi Isabel

I totally understand the differences appear large, but they are not when you take the measurement errors into acount when computing your group statistics.  To decrease the fluctuations between different runs, you could increase the number of PVs for for student abilities to 10. The secondary analysis will be more work, because you'll have to compute every statistic 10 times instead of 5 times.

Alternatively, you could use WLEs as estimates for respondents instead of plausible values. I suspect you want to describe your sample and not make inferences to the population with a representative sample. At individual level, WLEs are unbiased and they are much easier to work with.

Good luck with the analysis.

Eveline