1
Questions and Answers / Re: Traditional item analysis / covariance matrix not positive definite
« on: January 17, 2021, 10:59:05 PM »
of course - I have removed the attachments.
Show Posts
This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.
2
Questions and Answers / Re: Traditional item analysis / covariance matrix not positive definite
« on: January 16, 2021, 10:31:53 PM »
Hi Jonas,
your issues is with the data file you have crated. It is encoded UTF8 with a byte order marker (BOM). Early versions of ConQuest 4 did not support this encoding. I would recommend creating files with a BOM - it is not part of the Unicode standard, and only seems to appear with MS Excel when exporting files.
ConQuest 5 will attempt to strip the BOM when reading the file. I would encourage you to upgrade: https://shop.acer.edu.au/acer-conquest-5.html
Note that in ConQuest 5, you can read and write CSV files directly and do not need to create a text (.dat) or SPSS file.
Attached is a converted data file (I did this with VS Code - which is free to use and an excellent text editor) and output files. Note the issue with the covariance matrix is still present in your 2D model, I would guess it is to do with the very high correlation between your dimensions (>0.95), the very narrow variance (~0.1-0.3) and small sample size. Note the model deviance implies the 1D model is a better fit anyway.
your issues is with the data file you have crated. It is encoded UTF8 with a byte order marker (BOM). Early versions of ConQuest 4 did not support this encoding. I would recommend creating files with a BOM - it is not part of the Unicode standard, and only seems to appear with MS Excel when exporting files.
ConQuest 5 will attempt to strip the BOM when reading the file. I would encourage you to upgrade: https://shop.acer.edu.au/acer-conquest-5.html
Note that in ConQuest 5, you can read and write CSV files directly and do not need to create a text (.dat) or SPSS file.
Attached is a converted data file (I did this with VS Code - which is free to use and an excellent text editor) and output files. Note the issue with the covariance matrix is still present in your 2D model, I would guess it is to do with the very high correlation between your dimensions (>0.95), the very narrow variance (~0.1-0.3) and small sample size. Note the model deviance implies the 1D model is a better fit anyway.
3
Questions and Answers / Re: Differential Item Functioning
« on: December 13, 2020, 11:42:16 PM »
I notice you are using the ETS DIF categories. ConQuest will give you those directly using the mh command:
https://conquestmanual.acer.org/s4-00.html#mh
Note that to calculate mh, you can't estimate a facet for your grouping variable, but rather specify a group in your syntax.
If I imagine your "TERM 2: level" effect is zero, then it looks like to me, your item parameter for the omitted level for SUPK01 is -2.012, and the the item parameter for the group "low" should be (-2.012 + 0 + -0.755) = -2.767. If your term 2 is not zero, then you need to consider that you have subtracted-out the mean shift amongst the items first, and then estimated the item-specific shift for that group. If you only have two levels in your term 2, then you can just take the Wald: abs(-0.755/0.247) =~3.05, p < 0.05. If you take two seperate calibrations, I think the usual approach would be to mean-centre the calibrations first (which has a similar effect to subtracting out the main effect for group) unless you had some other way of establishing the metric (e.g., an anchored item). Otherwise your are assuming the two calibrations are producing meaningful 0 values which may or may not be reasonable (e.g., if you can assume parallel/equivalent samples by group?).
Perhaps you can share your syntax and data?
https://conquestmanual.acer.org/s4-00.html#mh
Note that to calculate mh, you can't estimate a facet for your grouping variable, but rather specify a group in your syntax.
If I imagine your "TERM 2: level" effect is zero, then it looks like to me, your item parameter for the omitted level for SUPK01 is -2.012, and the the item parameter for the group "low" should be (-2.012 + 0 + -0.755) = -2.767. If your term 2 is not zero, then you need to consider that you have subtracted-out the mean shift amongst the items first, and then estimated the item-specific shift for that group. If you only have two levels in your term 2, then you can just take the Wald: abs(-0.755/0.247) =~3.05, p < 0.05. If you take two seperate calibrations, I think the usual approach would be to mean-centre the calibrations first (which has a similar effect to subtracting out the main effect for group) unless you had some other way of establishing the metric (e.g., an anchored item). Otherwise your are assuming the two calibrations are producing meaningful 0 values which may or may not be reasonable (e.g., if you can assume parallel/equivalent samples by group?).
Perhaps you can share your syntax and data?
4
Questions and Answers / Re: Differential Item Functioning
« on: December 11, 2020, 12:17:49 AM »
Can you insert a table with your working?
In this case I would have thought that the Wald static is given by xsi/SE and is testing the null that the item by group interaction for that item and that group is 0.
The link you give to memo 25, is using two seperate calibrations (one for each group) and calculating the difference in xsi and the SE of the difference by hand.
Just another thing to keep in mind, if you are using "quick" errors, you may be under-estimating the SE (and increasing you chance of observing DIF). See the manual for a discussion.
In this case I would have thought that the Wald static is given by xsi/SE and is testing the null that the item by group interaction for that item and that group is 0.
The link you give to memo 25, is using two seperate calibrations (one for each group) and calculating the difference in xsi and the SE of the difference by hand.
Just another thing to keep in mind, if you are using "quick" errors, you may be under-estimating the SE (and increasing you chance of observing DIF). See the manual for a discussion.
5
Questions and Answers / Re: Differential Item Functioning
« on: December 09, 2020, 03:02:57 AM »
I think that is sound logic.
In the case of a binary DIF variable this will be equivalent - depending on specification you will either get a single DIF parameter (deviation of this group from the item parameter estimate for the omitted group), or you will get two (deviation of this group from the mean of the groups, where the second group is constrained to be the negative sum of the first).
In the case of more than two groups, then your proposed chi square test is reasonable.
In the case of a binary DIF variable this will be equivalent - depending on specification you will either get a single DIF parameter (deviation of this group from the item parameter estimate for the omitted group), or you will get two (deviation of this group from the mean of the groups, where the second group is constrained to be the negative sum of the first).
In the case of more than two groups, then your proposed chi square test is reasonable.
6
Questions and Answers / Re: Read EAP file
« on: November 23, 2020, 08:46:05 PM »
The format of the EAP file is described in the manual:
https://conquestmanual.acer.org/s4-00.html#show
If you use the option filetype to export a CSV, SPSS, or Excel file, you will see column headers providing a name for each of these columns.
https://conquestmanual.acer.org/s4-00.html#show
Quote
For plausible values (estimates=latent) and expected a-posteriori estimates (estimates=eap):
The file will contain one row for each case. Each row will contain (in order):
Sequence ID
PID (if PID is not specified in datafile or format than this is equal to the Sequence ID)
Plausible values. Note there will be np plausible values (default is 5) for each of nd dimensions. Dimensions cycle faster than plausible values, such that for nd = 2, and np = 3, the columns are in the order PV1_D1, PV1_D2, PV2_D1, PV2_D2, PV3_D1, PV3_D2.
the posterior mean (EAP), posterior standard deviation, and the reliability for the case, for each dimension. Note that these columns cycle faster than dimensions such that for nd = 2, and np = 3, the columns are in the order EAP_1, PosteriorSD_1, Reliability_1, EAP_2, PosteriorSD_2, Reliability_2.
If you use the option filetype to export a CSV, SPSS, or Excel file, you will see column headers providing a name for each of these columns.
7
Questions and Answers / Re: Fit indices in ConQuest
« on: November 09, 2020, 10:39:26 PM »
Isa,
BIC is reported in the output from the SHOW command in ConQuest Version 5. See example from Example 1 (https://www.acer.org/au/conquest/notes-tutorials):
You can download ConQuest 5 form the shop: https://shop.acer.edu.au/acer-conquest-5.html
BIC is reported in the output from the SHOW command in ConQuest Version 5. See example from Example 1 (https://www.acer.org/au/conquest/notes-tutorials):
Code: [Select]
The Data File: ex1.dat
The format: id 1-5 responses 12-23
No case weights
The regression model:
Grouping Variables:
The item model: item
Slopes are fixed
Cases in file: 1000 Cases in estimation: 1000
Final Deviance: 13274.87615
Akaike Information Criterion (AIC): 13300.87615
Akaike Information Criterion Corrected (AICc): 13300.56785
Bayesian Information Criterion (BIC): 13364.67697
Total number of estimated parameters: 13
The number of iterations: 45
Termination criteria: Max iterations=1000, Parameter Change= 0.00010
Deviance Change= 0.00010You can download ConQuest 5 form the shop: https://shop.acer.edu.au/acer-conquest-5.html
8
Questions and Answers / Re: Very different WLE Person separation reliability and EAP/PV reliability
« on: October 15, 2020, 05:52:40 AM »
Hi Isa,
If you have pre and post data, I would suggest estimating the growth in a single IRT model. One way is to create a wide file, load time 1 items onto one latent, and time 2 items onto a second latent, and anchor the difficulties of the items to be equal. The growth is then a function of the theta (theta2 - theta1 = growth). If you want to analyse the growth (e.g., regress covariates on the growth) you could choose a slightly different parameterisation, or you could then take PVs out of ConQuest and and fit whatever model (ANCOVA, mixed effects etc) you like in another stats package.
If you want to do it all in ConQuest, I would encourage you to read this article:
I ran your model, and I think the difference between PV and WLE reliabilities is as described above. You have:
- Dim 1 and 2 have items that are very easy
- some cases have no observations on a dimension and therefore a WLE cannot be calculated
- generally few items per dimension (see Spearman-Brown prophecy formula to think about what the maximum reliability you might expect is)
If you have pre and post data, I would suggest estimating the growth in a single IRT model. One way is to create a wide file, load time 1 items onto one latent, and time 2 items onto a second latent, and anchor the difficulties of the items to be equal. The growth is then a function of the theta (theta2 - theta1 = growth). If you want to analyse the growth (e.g., regress covariates on the growth) you could choose a slightly different parameterisation, or you could then take PVs out of ConQuest and and fit whatever model (ANCOVA, mixed effects etc) you like in another stats package.
If you want to do it all in ConQuest, I would encourage you to read this article:
Quote
Wilson, M., Zheng, X., & McGuire, L. (2012). Formulating latent growth using an explanatory item response model approach. Journal of Applied Measurement, 13(1), 1.
I ran your model, and I think the difference between PV and WLE reliabilities is as described above. You have:
- Dim 1 and 2 have items that are very easy
- some cases have no observations on a dimension and therefore a WLE cannot be calculated
- generally few items per dimension (see Spearman-Brown prophecy formula to think about what the maximum reliability you might expect is)
9
Questions and Answers / Re: How to read a wle.csv file?
« on: October 13, 2020, 10:52:25 PM »
Hi Isa,
Check out the new, online ConQuest Manual. It includes lots of key information, including the formatting of output file:
https://conquestmanual.acer.org/s4-00.html#show
Check out the new, online ConQuest Manual. It includes lots of key information, including the formatting of output file:
https://conquestmanual.acer.org/s4-00.html#show
Quote
For maximum likelihood estimates and weighted likelihood estimates (estimates=mle or estimates=wle):
The file will contain one row for each case that provided a valid response to at least one of the items analysed (one item per dimension is required for multidimensional models). The row will contain the case number (the sequence number of the case in the data file being analysed), the raw score and maximum possible score on each dimension, followed by the maximum likelihood estimate and error variance for each dimension. The format is (i5, nd(2(f10.5, 1x)), nd(2(f10.5, 1x))). If the pfit option is set then an additional column is added containing the case fit statistics. The format is then (i5, nd(2(f10.5, 1x)), nd(2(f10.5, 1x)), f10.5)
10
Questions and Answers / Re: Very different WLE Person separation reliability and EAP/PV reliability
« on: October 13, 2020, 10:49:29 PM »
Hi Isa,
I can only see your labels file attached. Can you attach the rest of your files?
If you are trying to explain a population parameter, like average learning gain over time, I would recommend you use PVs. Point estimates, including EAPs will result in biased secondary analysis. See for example:
https://doi.org/10.1016/j.stueduc.2005.05.005
I can only see your labels file attached. Can you attach the rest of your files?
If you are trying to explain a population parameter, like average learning gain over time, I would recommend you use PVs. Point estimates, including EAPs will result in biased secondary analysis. See for example:
https://doi.org/10.1016/j.stueduc.2005.05.005
11
ConQuest News / Re: ACER ConQuest Manual and Command Reference
« on: July 06, 2020, 11:15:34 PM »12
ConQuest News / Understanding Rasch Measurement Theory
« on: June 17, 2020, 07:34:31 AM »
The next iteration of Understanding Rasch Measurement Theory starts on Monday 13 July 2020. This is a 100% online, 10 week, part-time, masters level course.
Led by world-renowned researcher and psychometrician Professor Geoff Masters, this masters level course will unlock the practical understanding you need to get the most from Rasch.
Find out more about enrolments, course outline and contents, and what to expect from an ACER online course below.
https://www.acer.org/au/professional-learning/postgraduate/rasch
Led by world-renowned researcher and psychometrician Professor Geoff Masters, this masters level course will unlock the practical understanding you need to get the most from Rasch.
Find out more about enrolments, course outline and contents, and what to expect from an ACER online course below.
https://www.acer.org/au/professional-learning/postgraduate/rasch
13
Questions and Answers / Re: Very different WLE Person separation reliability and EAP/PV reliability
« on: May 26, 2020, 04:11:45 AM »
Thanks for the files.
I'd strongly encourage you to upgrade to a more recent version of ConQuest (version 5 is new this year).
Your model seems to run fine and converges happily. The estimated reliabilities are the same in a new build of ConQuest (attached).
The WLE reliability is low primarily because there are very few items in dimensions 2 and 3. Reviewing your WLE file, there are many cases where the max possible score for dimension 2 and 3 is 4.00. This can be interpreted as meaning there is little information available to space cases out along the latent trait. Your PV reliability is getting a boost because of the multidimenisfal structure. Your latent traits are highly correlated (> 0.8 ) and provide a lot of information that reduces the uncertainty in dimension 2 and 3. I think it's fair to conclude that this data is more useful for estimation of population parameters than individual abilities.
I'd strongly encourage you to upgrade to a more recent version of ConQuest (version 5 is new this year).
Your model seems to run fine and converges happily. The estimated reliabilities are the same in a new build of ConQuest (attached).
The WLE reliability is low primarily because there are very few items in dimensions 2 and 3. Reviewing your WLE file, there are many cases where the max possible score for dimension 2 and 3 is 4.00. This can be interpreted as meaning there is little information available to space cases out along the latent trait. Your PV reliability is getting a boost because of the multidimenisfal structure. Your latent traits are highly correlated (> 0.8 ) and provide a lot of information that reduces the uncertainty in dimension 2 and 3. I think it's fair to conclude that this data is more useful for estimation of population parameters than individual abilities.
14
Questions and Answers / Re: Very different WLE Person separation reliability and EAP/PV reliability
« on: May 21, 2020, 07:31:51 AM »
we will need to see you data and outputs to check this. Can you also let us know what version of ConQuest you are running?
Alternatively, send through a reproducible example: https://conquest-forums.acer.edu.au/index.php?topic=1239.0
Note, you don't technically need your recode AND your score statements, see: https://conquestmanual.netlify.app/s4-00.html#score
This is will give your items 1, 5, and 30, properties like in the ordered partition model.
Alternatively, send through a reproducible example: https://conquest-forums.acer.edu.au/index.php?topic=1239.0
Note, you don't technically need your recode AND your score statements, see: https://conquestmanual.netlify.app/s4-00.html#score
Code: [Select]
codes 0,1,2,3;
score (0,1,2,3) (0,0,0.5,1) ( ) ( ) !items (1);
score (0,1) (0,1) ( ) ( ) !items (2,6,7,8,12,16,19,20,23,25,26,31,32,35,39,42,43,44,45,48);
score (0,1) (0,1) ( ) ( ) !items (4,17,18,34,38,46,49);
score (0,1,2) (0,0.5,1) ( ) ( ) !items (11,47);
score (0,1) (0,1) ( ) ( ) !items (10,21,24,33,37);
score (0,1,2) ( ) (0,0,1) ( ) !items (5);
score (0,1) ( ) (0,1) ( ) !items (3,5,9,14,27,36,40);
score (0,1) ( ) ( ) (0,1) !items (13,15,22,28,29,30,41);
score (0,1,2) ( ) ( ) (0,0,1) !items (30);
This is will give your items 1, 5, and 30, properties like in the ordered partition model.
15
Questions and Answers / Re: EAP/PV reliability estimates
« on: May 04, 2020, 06:21:37 AM »
If you are comparing "model 1 factor" to "model 4 factors", I think it makes sense to impose the same scoring regime in both models (0,1,2,3,4).