MTurk: getQualificationScore returning "QualificationType does not Exist" for Base Qualifications Type (approval rate, etc)

Question

I've got a bunch of results from a Mechanical Turk HIT that I had done, and I want to try and rank all my answers by how trustworthy that Worker was. To do that, I'm trying to get each submitting worker's accept rate from MTurk.

I do this using the MTurk API QualificationRequirement type, "Worker_PercentAssignmentsApproved", whose type_id is 000000000000000000L0.

To get the worker's rating for percent approved, I call GetQualificationScore, which returns to me this very angry message:

 <GetQualificationScoreResponse>
   <OperationRequest>
     <RequestId>e5a68cbf-f8a4-4fc6-8d47-8198a138fe12</RequestId>
   </OperationRequest>
   <Qualification>
     <Request>
       <IsValid>False</IsValid>
       <Errors>
         <Error>
           <Code>AWS.MechanicalTurk.QualificationTypeDoesNotExist</Code>
           <Message>
             QualificationType 000000000000000000L0 does not exist (1287717938474) 
           </Message>
         </Error>
       </Errors>
     </Request>
   </Qualification>
 </GetQualificationScoreResponse>

Perhaps I had some formatting issues and was calling the method wrong.

To test that, I called GetQualificationType (which takes the same type of argument for the type: (QualificationTypeId, The ID of the Qualification type, String)).

Here's the thing: that works perfectly.

So, here's my question: Does Amazon just not permit access to base qualification/stats about workers that have done HITs for requesters? Or is there a way to get at this data that I haven't thought of?

Note I'm using a slightly-modified version of boto to make these calls, but have found boto to be buggy/lacking for this particular stuff. At this point I'm +/- working on the XML call level.

Answer 1

Correct, you can't query a Worker's system Qualifications directly. You can create HITs with Qualification requirements based on system Qualifications, but that's it.

Currently, the only way to gather specific data about a Worker is to analyze your own results for HITs and Qualification tests. You can store the results of your own analysis in Quals you create yourself and update them over time, and use those Quals as the basis for Qual requirements for your HITs.your HITs.

-DAN@AWS, Amazon, responding to this question back in 2006.

So, it looks like this can't be done directly. Sadface.