Author Topic: about empirical standard errors  (Read 858 times)

hklee1973

  • Newbie
  • *
  • Posts: 3
    • View Profile
about empirical standard errors
« on: April 18, 2016, 03:48:15 AM »
Hi,

I ran rasch model with 122 samples using ConQeust 3.0. When I requested the empirical standard errors, ConQuest didn't provide errors for some items. Case constraint and Montecarlo method was used. Could you explain why this happen? I need more accurate errors because the quick errors were too big.

Thanks,
Hye

Eveline Gebhardt

  • Administrator
  • Full Member
  • *****
  • Posts: 103
    • View Profile
    • Email
Re: about empirical standard errors
« Reply #1 on: April 19, 2016, 01:26:24 AM »
Hi Hye

I am not sure why this was not provided by CQ. Could you please download the newest version of ConQuest and try again? You will not need to enter your licence again. If you get the same results, could you email me your command and data file so I can have a closer look?

Best wishes
Eveline
eveline.gebhardt@acer.edu.au

hklee1973

  • Newbie
  • *
  • Posts: 3
    • View Profile
Re: about empirical standard errors
« Reply #2 on: April 21, 2016, 07:03:05 PM »
Eveline,

I am working for UC Berkeley BEAR Center, so I don't have a personal license code for ConQuest. Anyway, I downloaded the newest trial version and ran the same analysis. ConQuest gave the empirical errors, but they were even larger than quick standard errors. Were they supposed to be smaller than the quick error estimates? I used default estimation method this time. I tried to attach the command and outcome files, but an errors message kept popping up. If you need to see them, can I send them by email?

Thanks,
Hye

Eveline Gebhardt

  • Administrator
  • Full Member
  • *****
  • Posts: 103
    • View Profile
    • Email
Re: about empirical standard errors
« Reply #3 on: April 26, 2016, 02:06:23 AM »
Yes, please send them and the input files (command and data) to eveline.gebhardt@acer.edu.au.

Cheers
Eveline

shentong

  • Newbie
  • *
  • Posts: 14
    • View Profile
    • Email
Re: about empirical standard errors
« Reply #4 on: October 18, 2016, 02:46:31 AM »
Hi,

I came cross the same problem.  I was running the Rasch rating scale model with 123 samples using the latest version of Conquest. When I set the constraints=cases, the empirical SE was larger than the quick SE, but it is supposed to be smaller than the quick SE, right? When I set the constraints=items, ConQuest only provided the quick SE but not empirical SE. 
Additionally,  the EAP/PV reliability statistics were consistently changing each time I run the model, while the WLE Person separation reliability was quite stable. Could you explain why these happened? Thanks in advance for your help!


Eveline Gebhardt

  • Administrator
  • Full Member
  • *****
  • Posts: 103
    • View Profile
    • Email
Re: about empirical standard errors
« Reply #5 on: October 20, 2016, 10:24:21 PM »
Hi Shentong

Could you please email me your input files? I'll take a look and reply.

Cheers
Eveline

Eveline Gebhardt

  • Administrator
  • Full Member
  • *****
  • Posts: 103
    • View Profile
    • Email
Re: about empirical standard errors
« Reply #6 on: October 23, 2016, 11:11:52 PM »
Here some responses to both your questions:

Empirical standard errors are the default in ConQuest. If CQ chooses quick standard errors, there is probably a problem in your model. One issue could be that ConQuest needs a zero category for rating scale responses. If the data is coded as 1,2,3,4,5, you could add the command "score (1,2,3,4,5)(01,2,3,4);"

Empirical standard errors are not necessarily larger or smaller than quick standard errors.

If the model converges, WLE estimates should be (close to) identical each time you run the same model. Plausible value results differ slightly because of the measurment error. Measurement errors are larger for short tests. If differences in results seem large between runs, your model may not have converged well. For most simple models, adding "keeplastest=yes" to the SET command will fix the problem. Without this option, CQ takes the results from the iteration with the lowest deviance. Usually this is the last iteration, but not always, in which case CQ takes the results from an iteration before the parameters were converged. It is important to export a log file and examine if the results are acceptable.

Hope this helps.

Best wishes
Eveline

shentong

  • Newbie
  • *
  • Posts: 14
    • View Profile
    • Email
Re: about empirical standard errors
« Reply #7 on: October 30, 2016, 08:15:00 PM »
 Hi Eveline,

Thanks for your reply! It's been deeply appreciated!

When I added the "keeplastest=yes" to the SET command and rerun the command file, the empirical SE did show up. But when I changed the command "set constraints=cases;" to "set constraints= items;", the empirical SE were missing again.

Interestingly, when I worked on another dataset, still using the Rasch rating scale model, the empirical SE were showing when "set constraints= items;" but missing when "set constraints= cases;". Could you explain why these things happened?

Best,

Tong

Eveline Gebhardt

  • Administrator
  • Full Member
  • *****
  • Posts: 103
    • View Profile
    • Email
Re: about empirical standard errors
« Reply #8 on: November 02, 2016, 05:56:47 AM »
Hi Tong

I see ConQuest complains about a negative variance and possible model misidentification when using constraints=items during the estimation of the empirical errors. I am not sure yet why, but I am looking into it. The data looks fine, but CQ may struggle with the small number of cases and items. I'll get back to you when I know more.

Eveline


Eveline Gebhardt

  • Administrator
  • Full Member
  • *****
  • Posts: 103
    • View Profile
    • Email
Re: about empirical standard errors
« Reply #9 on: November 03, 2016, 12:40:08 AM »
Hi Tong

I'm sorry, I'm not sure why it happens. I think the problem is in the highest step parameter (the one that is constraint). Your test is short and sample small, which can lead to estimation problems.

I think the best solution is to use constraints=cases. If you subtract the average item difficulty from the the student abilities and from the item abilities, the results will be the same as constraints=items.

Another option is to use Quadrature as estimation method. That seems to work with constraints=items.

I'll let you know if I find out more.

Cheers
Eveline