Differential Item Functioning

[KatharinaBorg]

Hi,


I am calculating DIF analyses with dichotomous items from TIMSS comparing two German versions of the test to identify a set of anchor items for further research.
The manual describes quite well how to formulate the model statement and how to interpret the output, but due to the fact that I am writing a statistics thesis I am furthermore interested in details about the applied mathematical models and test statistics.

Concretely, I would be glad to get some reference papers to

a)   the underlying assumptions and model specifications when computing “model item – dif.variable + item*dif.variable”
b)   the computed “Chi-square test of parameter equality” (e.g., ConQuest Manual, Figure 8-2)
c)   test specifications for the ESTIMATE values for the item*dif.variable term (ConQuest Manual, Figure 8-2, TERM 3)
[Of special interest would be a reference proving that “If zero is not element of [ESTIMATE – 2*ERROR; ESTIMATE + 2*ERROR] it is an indicator for the existence of DIF”.]


Thanks and best wishes,
Katharina

[Eveline Gebhardt]

Hi Katharina

a) item*dif.variable gives estimates of differences parameters.  eg  difference in item difficulty for boys and girls.  Null hypothesis of no difference is parameter=0.  Dividing parameter by its standard error is Wald test of this null hypothesis

b) This is a form of Hedges d.  eg Hedges and Olkin 1985   (statistical methods for meta-analysis).  The null hypothesis is that all the dif parameters are zero

c) This is the same as the wald test above     abs(est/error) > 1.96 is same as the confidence interval approach you describe here.

Eveline

[KatharinaBorg]

Hi Eveline,

thank you for your response, it helped me a lot!

I read about the Wald test (Lord 1980) – is it correct that the Wald test is applied to include the difficulty parameter as well as the discrimination parameter in testing for DIF? That would mean that the estimated model (item – dif.variable + item*dif.variable) is a 2PL model, right? In that way the model would be appropriate to detect non-uniform DIF of an item?!

Thanks and best wishes,
Katharina

[Eveline Gebhardt]

Hi Katharina

The default is a one-parameter model in ConQuest, so I think that is what you have run. The DIF parameters are the differences in item difficulty for one group and the average item difficulty. We usually look at both significance (DIF/SE) and at the size of the DIF. Depending on how strict you are a DIF of 0.15 or 0.25 is usually seen as substantive (if significant).

You can plot ICCs by group to see if the DIF is uniform. For this, you need to run the model without DIF (model item;) and define your DIF variable as a group variable.

Eveline

[A.Kirsch]

Hi Eveline, hi guys,

I know that the topic is a bit older, but I read it a few days ago and I'm a bit unsure now.

Based on the DIF tutorial by Adams & Wu (2010) and many others (e.g. Le, 2009), I have so far assumed that DIF is the absolute difference between the two estimated difficulties of the two groups. So it should be (abs.) sum of the both group-specific values given in term 3 for item i (deviation from the average difficulty of the item i), shouldn't it?

Thus, EST/SE would only be the significance of the deviation of one group from the average difficulty of the items, while the significance of DIF in relation to the item-specific Wald-Test could be determined by Chi square = [Est(1)/SE(1)]^2 + [Est(2)/SE(2)]^2 = 2 (Est/SE)^2, df=2-1.  Where EST corresponds to values in term 3, 1=Group 1 and 2=Group 2, or?

Thanks
Alex

[dan_c]

I think that is sound logic.

In the case of a binary DIF variable this will be equivalent - depending on specification you will either get a single DIF parameter (deviation of this group from the item parameter estimate for the omitted group), or you will get two (deviation of this group from the mean of the groups, where the second group is constrained to be the negative sum of the first).

In the case of more than two groups, then your proposed chi square test is reasonable.

[A.Kirsch]

Hi Dan,

thank you for your response.

I got confused when trying to merge the significance test of DIF [Chi Square = (EST/SE)^2] described in the tutorial with the item specific Wald-Test (e.g. Draba, 1977, Strobl, 2012)*, which is defined as W= DIF^2 / SE(1)^2+SE(2)^2.
If I put the estimated parameters and standard errors for both groups into this formula, the result is twice (EST/SE)^2.

But why?


*https://www.rasch.org/memo25.htm

[dan_c]

Can you insert a table with your working?

In this case I would have thought that the Wald static is given by xsi/SE and is testing the null that the item by group interaction for that item and that group is 0.

The link you give to memo 25, is using two seperate calibrations (one for each group) and calculating the difference in xsi and the SE of the difference by hand.

Just another thing to keep in mind, if you are using "quick" errors, you may be under-estimating the SE (and increasing you chance of observing DIF). See the manual for a discussion.

[A.Kirsch]

Hi Dan,

Perhaps my mistake is that I tried to think of the two group-specific parameters as calibrated separately?

Tables and calculation are attached.

[dan_c]

I notice you are using the ETS DIF categories. ConQuest will give you those directly using the mh command:

https://conquestmanual.acer.org/s4-00.html#mh

Note that to calculate mh, you can't estimate a facet for your grouping variable, but rather specify a group in your syntax.

If I imagine your "TERM 2: level" effect is zero, then it looks like to me, your item parameter for the omitted level for SUPK01 is -2.012, and the the item parameter for the group "low" should be (-2.012 + 0 + -0.755) = -2.767. If your term 2 is not zero, then you need to consider that you have subtracted-out the mean shift amongst the items first, and then estimated the item-specific shift for that group. If you only have two levels in your term 2, then you can just take the Wald: abs(-0.755/0.247) =~3.05, p < 0.05. If you take two seperate calibrations, I think the usual approach would be to mean-centre the calibrations first (which has a similar effect to subtracting out the main effect for group) unless you had some other way of establishing the metric (e.g., an anchored item). Otherwise your are assuming the two calibrations are producing meaningful 0 values which may or may not be reasonable (e.g., if you can assume parallel/equivalent samples by group?).

Perhaps you can share your syntax and data?

[A.Kirsch]

Hi Dan,

yes, you are right, I wanted to apply the ETS categories here too. I did the MH with other grouping criteria as well. But for the grouping criterion I applied here (high/low testscore), a MH was not possible, because testscore/ability is already the matching variable of the MH (?!)

Thanks a lot. Now it is clear. I simply disregarded the mean shift in my considerations,
but in principle I have understood the procedure.