Skip to content
This repository has been archived by the owner. It is now read-only.

Issue with requesting by batch (extra info in annotations) #120

Closed
torfason opened this issue Jul 7, 2017 · 2 comments

Comments

Projects
None yet
2 participants
@torfason
Copy link

commented Jul 7, 2017

Hi, I had an issue with requesting results by batch from MTurk using GetAssignments. The symptom is as follows when I request results for batch 123456, which exists:

  > d <- GetAssignment(annotation="BatchId:123456;")
  Error in GetAssignment(annotation = "BatchId:123456;") : 
    No HITs found for HITType`

The issue is that there is extra information in the annotations field, and GetAssignment() by annotation requires an exact match. After examining the HITs with SearchHITs I find that there is information about OriginalHitTemplateId in the annotations field. After updating the query to add that, the GetAssignment() request works:

  > d <- GetAssignment(annotation="BatchId:123456;OriginalHitTemplateId:9988776655;")
  60 of 60 Assignments Retrieved

I wrote a function, GetAssignmentsByBatch(), that also works for my case:

GetAssignmentsByBatch <- function(batch.id)
{
    stopifnot(is.character(batch.id) && length(batch.id)==1)
    
    result.SearchHITs <- SearchHITs(verbose = FALSE, return.qual.dataframe = FALSE)
    unique.annotations <- unique(result.SearchHITs$HITs$RequesterAnnotation)
    annotation <- grep(paste0("BatchId:",batch.id), unique.annotations, value=TRUE)
    if (length(annotation)<1)
    {
        stop(paste0("Batch ", batch.id, " not found in search results"))
    }
    stopifnot(is.character(annotation) && length(annotation)==1)
    result <- GetAssignment(annotation=annotation)
    return(result)
}

Using this function, I get the results that I expected:

 > d = GetAssignmentsByBatch("123456")
 60 of 60 Assignments Retrieved

If it had been possible to search by wildcard (or, perhaps the best solution would be if GetAssignment() used a contains by default, rather than an exact match, to select the assignments to return), this would have worked. I file this as an issue because I found nothing in the documentation to suggest there was a better way that I should have used.

@leeper leeper added the enhancement label Jul 7, 2017

@leeper

This comment has been minimized.

Copy link
Member

commented Jul 7, 2017

Thanks, yes, only exact matching is currently implemented (here but also everywhere that annotation is used in the package). I will try to implement this, though the answer may be a regex option rather than a wildcard (so that it's more general).

It may take some time as it will require changing a lot of code. In the meantime, you can use SearchHITs() to retrieve the HITId values that you want (by grep()-ing against the RequesterAnnotation column) and then pass those values into GetAssignments(hit = ) iteratively (which is all this is actually doing internally).

@torfason

This comment has been minimized.

Copy link
Author

commented Jul 8, 2017

Thanks for the response, yes regex + perhaps a warning in the documentation could help people out. It's tricky, I guess I just have a special use case that results in the OriginalHitTemplateId to be included in the annotation field.

@leeper leeper added the wontfix label May 23, 2019

@leeper leeper closed this May 23, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.