- API-users - lists.gbif.org

wrong faceted API results/counts (at least in our datasets) and lowercased facet values (everywhere)
by Herbario SANT 15 Feb '17

15 Feb '17

Hello. I am trying to play with faceted results from the occurrence api, but returned values are very odd IMHO. Perhaps I am misunderstanding how faceting should work? Or there might be some problem with the indexing of these particular datasets. I am pretty lost. This is what I found: *(1) RESULTS COUNT NOT MATCHING SUM OF ALL FACET COUNTS* I put a simple example so everything is returned in one page. http://api.gbif.org/v1/occurrence/search?facet=ScientificName&limit=10&coll… The count value is 4, the number of results is 4. But the number of facets is 1, and its count is 2. The faceted term ScientificName is a mandatory field, so no null values should happen. I would expect every occurrence having a value for it. And the number of values is short, so everything is returned in one request (no paging needed). So, in such a case shouldn't the sum of facet counts be equal to the number of results? Why the count of the faceted name is not 4? *(2) LOWERCASE FACETS (facets values not matching results values):* Look at the same api request above (plant names) results: Scientificname: "Generic_name specific_name (Basionym_Authors) Name_Authors" facets: name: "generic_name specific_name (basionym_authors) name_authors" Why are the facets names always in lowercase? I would say that is an error which shouldn't happen. But I reported it some days ago and got no answer, so I wonder if this is the intended api behaviour. http://dev.gbif.org/issues/browse/PF-2758 Not only scientific names are lowercased. This also happens to collectionCode in the next question. *(3) FACETING COLLECTIONCODE VALUES of a single institution fails depending on filtering parameter used to match the institution (code or uuid):* Our institution (uuid= def87a70-0837-11d9-acb2-b8a03c50a862 , institutionCode=SANT) serves datasets from 4 collections, which should sum up more than 100000 records. Why do I get only 2 of our 4 datasets faceted in the following request, which uses our publishingOrg uuid? (uuid should be the preferred option to do this, as code might not be unique for our institution) http://api.gbif.org/v1/occurrence/search?facet=collectionCode&limit=0&publi… Why do I got 4 of 4 if I filter the request using institutionCode instead? (fortunately, nobody else uses the same institutionCode yet, so numbers are correct) http://api.gbif.org/v1/occurrence/search?facet=collectionCode&limit=0&insti… And why do counts differ for the same facet value (sant-lich) in those two requests? (9960 in the 1st request, 10007 in the 2nd one) Why are facet values lowercase again? ("sant-lich" instead of "SANT-Lich") *(4) FACETING SCIENTIFICNAME FAILS FOR SOME DATASETS, but works as expected for others: * More than 1000 faceted Scientificnames returned for our SANT-Lich and SANT-Algae collections. Both of them look correct results: http://api.gbif.org/v1/occurrence/search?facet=ScientificName&limit=0&colle… http://api.gbif.org/v1/occurrence/search?facet=ScientificName&limit=0&colle… But no facets returned for SANT-Bryo (which contains several hundred distinct scientificname values): http://api.gbif.org/v1/occurrence/search?facet=ScientificName&limit=0&colle… And only 7 facets for SANT scientificnames (should be over 10 thousand, as this is by far our largest dataset): http://api.gbif.org/v1/occurrence/search?facet=ScientificName&limit=0&colle… Other than the lowercase facets issue (2), I couldn't reproduce issues 1,3,4 in other institutions datasets. So I wonder if all this is somehow related to a wrong indexing of our IPT. Has anyone else detected these problems? Thanks a lot in advance for your help David -- David García San León Herbario SANT Universidade de Santiago de Compostela

3 4

dataset inflated occurrence counts?
by Stoner, Dan 13 Feb '17

13 Feb '17

Regarding this dataset: http://www.gbif.org/dataset/e0dbf705-cec4-4dce-a152-fc4ebe14674d The count of occurrences in GBIF appears to be (at least) double what is expected. Screenshot: http://imgur.com/a/A9C8M The occurrence count endpoint in the API displays the same value as the GBIF portal: http://api.gbif.org/v1/occurrence/count?datasetKey=e0dbf705-cec4-4dce-a152-… or, 146509 occurrences. However, the actual DwC-A seems to have only a bit over 50k rows. The description for that dataset reads "The vertebrate fossil collection at the Royal Ontario museum consists of over 72,000 catalogued and databased fossil specimens of all vertebrate classes" Further digging indicates that ROM changed their occurrenceID format. A sample that appears to be the same specimen catalog number, same actual specimen, but exists twice in GBIF, once for each occurrenceID format. http://www.gbif.org/occurrence/1211615291 http://www.gbif.org/occurrence/1234628259 Thanks, Dan Stoner iDigBio / ACIS Laboratory University of Florida

1 0

dataset activity api request paged by date
by Herbario SANT 10 Feb '17

10 Feb '17

Hello Where can I find a list of all possible api parameters for this api request?: /occurrence/download/dataset/{datasetKey} It looks similar to the data portal "dataset download activitiy" url, which uses an offset parameter for paging download activity using the links at the end of the page. http://www.gbif.org/dataset/{datasetKey}/activity?offset=10 But ... what other filtering options are there when using the api? I wonder if there is a way to request activity related to a certain date or date range (or to a single year, month). Some examples: year=2016 yearRange=2014,2016 date=2017-01-31 dateRange=2016-12-01,2016-12-31 Of course, I am meaning the "download date", not the "occurrence date". Thanks a lot for your help. -- David García San León Herbario SANT Universidade de Santiago de Compostela

3 3

GBIF Backbone
by godfoder 10 Feb '17

10 Feb '17

iDigBio is nearly complete with the process of integrating the GBIF Backbone into our backend and the process so far has been fairly painless. Thank for all the hard work you've put into building up this resource for the community. I do have one (hopefully minor) change request: Can you add in the parsed values for specificEpithet and infraspecificEpithet into the taxon file where they are available? Background: We're using a multi-term fuzzy match in elasticsearch to resolve taxonomy from records against the backbone, this is performing fairly well, and the more terms we add to the match criteria the better it becomes. The scientificName field in most specimens doesn't include authorship information, so is a relatively poor match overall against the scientificName field in the backbone. Most specimens supply the specificEpithet field though, so I was including it in our matching criteria until I realized that that field is missing from the backbone entirely. Thanks, - Alex

2 2

A couple of questions about the download API
by Doug Palmer 09 Feb '17

09 Feb '17

Hi, I'm a newbie to the GBIF API and I have a couple of questions about downloads. As background, I'm looking at the best way of getting data from GBIF into the Atlas of Living Australia (the bits we don't have already) properly split up into source datasets for attribution. My questions: With the occurrence download limits, am I right in thinking that the limits refer to the number of download requests being processed simultaneously? Can the downloads be of any size in terms of number of records and detail? Instead of being notified by email upon completion, is it possible to poll the status of a download? Doug Palmer

2 1

Interest for a mock server?
by Nicolas Noé 24 Jan '17

24 Jan '17

Hi everyone and best wishes! Over the years, I have several times written code that consumes the GBIF API. So far, proper unit testing of this code has been problematic, because I had to test it against the "real" GBIF API, with several drawbacks (fragility, test needs network connectivity, speed issues, ...). I think other libraries (pygbif for example) probably face the same problems. I am therefore thinking about creating a GBIF API mock server, that could be run locally and mimicks requests/responses cycles of api.gbif.org, to make testing our tools in isolation easier. It would have the following characteristics: * Be a (localhost-runnable) server rather than a mock library, so it can be used by code in any language * Mock the responses from api.gbif.org (starting with the most common stuff: read-only queries on occurrences, registry, ...) * Be super easy to install and run locally I'm thinking of implementing it in Golang, since it seems to fit those requirements quite well and I'd like to build real-world experience with this technology. Ideally, I'd like to write that is not only useful for myself, so here is my question: would you be interested in using such a mock server for your developments? Or isn't that so usefull? Maybe you know existing tools that can already be used to solve this problem? Thanks for your feedback! Nicolas

1 0

Download API: support many predicates?
by Scott Chamberlain 14 Jan '17

14 Jan '17

Hi! A rgbif user is asking me about a particular download query, see https://github.com/ropensci/rgbif/issues/245 TLDR: AFAICT this can't work as the API is set up right now. Or am I wrong? That is, he wants to get data for records that match - 10 taxa with their taxonKey value - 3 different basisofrecord values - country = US - hasCoordinate = TRUE - hasGeospatialIssue = FALSE - year range between 1949 and 2011 - month range between 3 and 8 For the taxonKey values, I think he needs to combine those with *OR* operator: "predicate": { "type": "or", "predicates": [ { "type": "equals", "key": "TAXON_KEY", "value": "2475470" }, { "type": "equals", "key": "TAXON_KEY", "value": "2475470" } ] } However, he should combine the other parameters (basisofrecord, hasCoordinate, year, month, etc) using *AND* AFAICT. *BUT*, AFAICT the API only allows one *predicate* key so you can't do what I think is needed here. That is, to say (taxonkey = 1 OR 2 OR 3) + (country = US AND hasCoordinate = true, etc.) I tried sending two *predicate* keys in the API and only the second goes through. Any guidance much appreciated :) The best solution I can think of is to do the query he's doing, but do one for each taxon. Seem like a good solution? Best, Scott

2 2

Secure (HTTPS) access to the GBIF API
by Matthew Blissett 10 Jan '17

10 Jan '17

Hi all, I'm pleased to announce that the GBIF API is now accessible securely — simply replace http:// with https://. http://api.gbif.org/v1/occurrence/search → https://api.gbif.org/v1/occurrence/search This is necessary for anyone embedding the API in a secure website, or where organizational policy requires using HTTPS. Using the secure API is optional, the non-secure HTTP version will continue to work as before. However, the new GBIF.org website <https://demo.gbif.org/> will only be accessible using HTTPS, as this will soon be a requirement¹ for any website accepting passwords. Please let us know if you encounter any problems with this, Many thanks, Matthew Blissett ¹ https://security.googleblog.com/2016/09/moving-towards-more-secure-web.html and https://blog.mozilla.org/tanvi/2016/01/28/no-more-passwords-over-http-pleas…

1 0

Downloading large numbers of datasets
by Coggins, Clint 08 Dec '16

08 Dec '16

Hi Matt, Thanks so much for your help. I was able to successfully get the file from your link. I did notice there was not "Content-Length" header in the original download response. Maybe that has something to do with the wget trouble. > Subject: Re: [API-users] Downloading large numbers of datasets > Message-ID: <14c80817-da32-23c4-e1ee-eded21481079(a)gbif.org> > Content-Type: text/plain; charset="windows-1252"; Format="flowed" > > Hi Clint, > > You're first to report a problem, though trying to download the file > (64-bit Linux, wget) within the GBIF network also failed for me. > > I've copied it with a different method to here: > http://download.gbif.org/2016/12/0039949-160910150852091.zip -- on a > plain Apache server, so wget's "--continue" option should work if necessary. > > The MD5 checksum is e976523c9e6c7ec0cd9d3cb30030020b and the size is > exactly 43,184,530,448 bytes. > > If downloading that doesn't work, I could split the file into chunks. > We'll also look into why the download failed [1]. > > Cheers, > > Matt Blissett > > [1] http://dev.gbif.org/issues/browse/POR-3199 > > > On 07/12/2016 14.53, Coggins, Clint wrote: >> I'm trying to download all the occurrence data with COUNTRY=US >> >> http://www.gbif.org/occurrence/search?COUNTRY=US >> >> I've requested a download file, which is here >> http://www.gbif.org/occurrence/download/0039949-160910150852091 >> >> However, I've been having a lot of trouble getting the download to >> complete since it is so large(43.2GB). I've tried various browsers and >> also wget on both Linux and Mac. It typically fails with a network >> error in the browser. wget displayed strange behavior in that it >> claimed the download was successful after downloading 4.1GB. This was >> on 64 bit linux with ext4, so I don't think there was a filesystem >> limitation. >> >> Any ideas on how to download this data? Does it make sense to write a >> script to use the API to request the datasets one by one to split it up? >> >> Thanks for any help >> >> -- >> Clint Coggins

1 0

Downloading large numbers of datasets
by Coggins, Clint 07 Dec '16

07 Dec '16

I'm trying to download all the occurrence data with COUNTRY=US http://www.gbif.org/occurrence/search?COUNTRY=US I've requested a download file, which is here http://www.gbif.org/occurrence/download/0039949-160910150852091 However, I've been having a lot of trouble getting the download to complete since it is so large(43.2GB). I've tried various browsers and also wget on both Linux and Mac. It typically fails with a network error in the browser. wget displayed strange behavior in that it claimed the download was successful after downloading 4.1GB. This was on 64 bit linux with ext4, so I don't think there was a filesystem limitation. Any ideas on how to download this data? Does it make sense to write a script to use the API to request the datasets one by one to split it up? Thanks for any help -- Clint Coggins

2 1