[API-users] Limits on the occurrence search API

[Tim Robertson]

Hi Nico

> - Shouldn't the secretariat update the documentation to say the limit is 200.000 instead of 1.000.000 ?

Yes

> - Is there a plan to incrementally remove/lessen this limitation ? 200K is quite a limit compared to the 500M+ total records available.

Not until the Lucene committers fix their deep paging issues.  This is a common problem in Lucene and you’ll find similar limitations on the likes of Google search results paging etc.
Paging over more than 200k records is not all that sensible in general though, when a far more efficient mechanism exists with the download API.

Cheers,
Tim


> 
> Le 12/02/15 20:20, Markus Döring a écrit :
>> Hi Nico,
>> see anserws inline below.
>> Makrus
>> 
>> 
>> 
>>> I'm having some doubts about how the hard limits work when using the occurrence search API. I think this is a documentation/error message issue rather than a technical one:
>>> 
>>> Here is what the doc says:
>>> 
>>> - max value for "limit" = 300
>>> - max total records = 1.000.000 (does that means limit * offset?)
>> no, it means offset + limit
>> and like you discovered below the limit is actually at 200.000 not 1 million
>> 
>> 
>>> What my tests says:
>>> 
>>> - http://api.gbif.org/v1/occurrence/search?hasCoordinate=true&limit=300&offset=3335
>>> => works, but shouldn't since 300*3335 = 1000500 total records
>> the offset is not in pages but absolute records. So you actually are looking at records 3335 to 3634
>> 
>>> - http://api.gbif.org/v1/occurrence/search?hasCoordinate=true&limit=1&offset=200000
>>> => "maximum offset allowed is 200000", but offset IS 200000 (message should be "less than"? 199999 works) Also, this limit on the offset parameter should be stated in the doc page.
>> I believe we have not updated the docs and the 1 million limit actually is 200.000 which you can see here in your example!
>> 
>>> - http://api.gbif.org/v1/occurrence/search?hasCoordinate=true&limit=300&offset=199998
>>> => "maximum offset allowed is 200000": here, we probably hit the 1.000.000 total limit, but the error message refer to the undocumented (and not hit here) max offset limit.
>>> 
>>> Could you clarify a bit ? 
>>> 
>>> All the best,
>>> 
>>> Nico
>>> 
>>> 
>>> _______________________________________________
>>> API-users mailing list
>>> API-users at lists.gbif.org
>>> http://lists.gbif.org/mailman/listinfo/api-users
>> 
> 
> _______________________________________________
> API-users mailing list
> API-users at lists.gbif.org
> http://lists.gbif.org/mailman/listinfo/api-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gbif.org/pipermail/api-users/attachments/20150226/15a4bfa3/attachment.html>