February 2017 - API-users - lists.gbif.org

API for batch calling GBIF id -> checklist ID
by Yan Wong 22 Feb '17

22 Feb '17

Hi I have a large number of GBIF ids (~ 2 million), for which I wish to know any existing International Plant Names Index IDs (IPNI s). What’s the easiest way to do this? I can use the standard API, e.g. http://api.gbif.org/v1/species/2705290/related for 2705290 = Avena sativa, which gives me an output that contains several IPNIs (in this case, 164949-3, 391732-1, 27200-2, of which the number 391732-1 seems to be the canonical one, although I can’t figure out how to identify it as such from the GBIF API output) It will obviously be tedious (and unwarranted) to make 2 million API calls. What’s the recommended way to do this? Yan

3 3

how to check last date-time dataset was harvested?
by Herbario SANT 20 Feb '17

20 Feb '17

Hi again. Is there any way to use GBIF portal API to know the last date & time a certain resource was harvested by GBIF? (basically, to know if it has the latest info provided by the original source, or it could be outdated) I suspect the info in this url but I don' know how to interpretate them all: http://api.gbif.org/v1/dataset/df3eab30-0837-11d9-acb2-b8a03c50a862 I guess this is the IPT updated time: "modified": "2017-02-16T22:07:41.127+0000" ¿Perhaps this is what I am looking for? "machineTags":[{..."name":"crawl_attempt",..."created":"2017-01-05T12:07:12.525+0000"},...] Thanks a lot David -- David García San León Herbario SANT Universidade de Santiago de Compostela

2 2

Re: [API-users] Is there any NEO4J or graph-based driver for this API ?
by Collins, Matthew 15 Feb '17

15 Feb '17

Jorrit pointed out this thread to us at iDigBio. Downloading and importing data into a relational database will work great, especially if as Jan said you can cut the data size down to a reasonable amount. Another approach we've been working on in a collaboration called GUODA [1] is to build an Apache Spark environment with pre-formatted data frames with common data sets in them for researchers to use. This approach would offer a remote service where you could write arbitrary Spark code, probably in Jupyter notebooks, to iterate over data. Spark does a lot of cool stuff including GraphX which might be of interest. This is definitely pre-alpha at this point and if anyone is interested, I'd like to hear your thoughts. I'll also be at SPNHC talking about this. One thing we've found in working on this is that importing data into a structured data format isn't always easy. If you only want a few columns, it'll be fine. But getting the data typing, format standardization, and column name syntax of the whole width of an iDigBio record right requires some code. I looked to see if EcoData Retriever [2] had a GBIF data source and they have an eBird one that perhaps you might find useful as a starting point if you wanted to try to use someone else's code to download and import data. For other data structures like BHL, we're kind of making stuff up since we're packaging a relational structure and not something nearly as flat as GBIF and DWC stuff. [1] http://guoda.bio/ [2] http://www.ecodataretriever.org/ Matthew Collins Technical Operations Manager Advanced Computing and Information Systems Lab, ECE University of Florida 352-392-5414<callto:352-392-5414> ________________________________ From: jorrit poelen <jhpoelen(a)xs4all.nl> Sent: Monday, May 30, 2016 11:16 AM To: Collins, Matthew; Thompson, Alexander M; Hammock, Jennifer Subject: Fwd: [API-users] Is there any NEO4J or graph-based driver for this API ? Hey y’all: Interesting request below on the GBIF mailing list - sounds like a perfect fit for the GUODA use cases. Would it be too early to jump onto this thread and share our efforts/vision? thx, -jorrit Begin forwarded message: From: Jan Legind <jlegind(a)gbif.org<mailto:jlegind@gbif.org>> Subject: Re: [API-users] Is there any NEO4J or graph-based driver for this API ? Date: May 30, 2016 at 5:48:51 AM PDT To: Mauro Cavalcanti <maurobio(a)gmail.com<mailto:maurobio@gmail.com>>, "Juan M. Escamilla Molgora" <j.escamillamolgora(a)lancaster.ac.uk<mailto:j.escamillamolgora@lancaster.ac.uk>> Cc: "api-users(a)lists.gbif.org<mailto:api-users@lists.gbif.org>" <api-users(a)lists.gbif.org<mailto:api-users@lists.gbif.org>> Dear Juan, Unfortunately we have no tool for creating these kind of SQL like queries to the portal. I am sure you are aware that the filters in the occurrence search pages can be applied in combination in numerous ways. The API can go even further in this regard[1], but it not well suited for retrieving occurrence records since there is a 200.000 records ceiling making it unfit for species exceeding this number. There is going be updates to the pygbif package[2] in the near future that will enable you to launch user downloads programmatically where a whole list of different species can be used as a query parameter as well as adding polygons.[3] In the meantime, Mauro’s suggestion is excellent. If you can narrow your search down until it returns a manageable download (say less than 100 million records), importing this into a database should be doable. From there, you can refine using SQL queries. Best, Jan K. Legind, GBIF Data manager [1] http://www.gbif.org/developer/occurrence#search [2] https://github.com/sckott/pygbif [3] https://github.com/jlegind/GBIF-downloads From: API-users [mailto:api-users-bounces@lists.gbif.org] On Behalf Of Mauro Cavalcanti Sent: 30. maj 2016 14:06 To: Juan M. Escamilla Molgora Cc: api-users(a)lists.gbif.org<mailto:api-users@lists.gbif.org> Subject: Re: [API-users] Is there any NEO4J or graph-based driver for this API ? Hi, One solution I have successfully adopted for this is to download the records (either "manually" via browser or, yet better, using a Python script using the fine pygbif library), storing them into a MySQL or SQLite database and then perform the relational queries. I can provide examples if you are interested. Best regards, 2016-05-30 8:59 GMT-03:00 Juan M. Escamilla Molgora <j.escamillamolgora(a)lancaster.ac.uk<mailto:j.escamillamolgora@lancaster.ac.uk>>: Hola, Is there any API for making relational queries like taxonomy, location or timestamp? Thank you and best wishes Juan _______________________________________________ API-users mailing list API-users(a)lists.gbif.org<mailto:API-users@lists.gbif.org> http://lists.gbif.org/mailman/listinfo/api-users -- Dr. Mauro J. Cavalcanti E-mail: maurobio(a)gmail.com<mailto:maurobio@gmail.com> Web: http://sites.google.com/site/maurobio _______________________________________________ API-users mailing list API-users(a)lists.gbif.org<mailto:API-users@lists.gbif.org> http://lists.gbif.org/mailman/listinfo/api-users

5 11

wrong faceted API results/counts (at least in our datasets) and lowercased facet values (everywhere)
by Herbario SANT 15 Feb '17

15 Feb '17

Hello. I am trying to play with faceted results from the occurrence api, but returned values are very odd IMHO. Perhaps I am misunderstanding how faceting should work? Or there might be some problem with the indexing of these particular datasets. I am pretty lost. This is what I found: *(1) RESULTS COUNT NOT MATCHING SUM OF ALL FACET COUNTS* I put a simple example so everything is returned in one page. http://api.gbif.org/v1/occurrence/search?facet=ScientificName&limit=10&coll… The count value is 4, the number of results is 4. But the number of facets is 1, and its count is 2. The faceted term ScientificName is a mandatory field, so no null values should happen. I would expect every occurrence having a value for it. And the number of values is short, so everything is returned in one request (no paging needed). So, in such a case shouldn't the sum of facet counts be equal to the number of results? Why the count of the faceted name is not 4? *(2) LOWERCASE FACETS (facets values not matching results values):* Look at the same api request above (plant names) results: Scientificname: "Generic_name specific_name (Basionym_Authors) Name_Authors" facets: name: "generic_name specific_name (basionym_authors) name_authors" Why are the facets names always in lowercase? I would say that is an error which shouldn't happen. But I reported it some days ago and got no answer, so I wonder if this is the intended api behaviour. http://dev.gbif.org/issues/browse/PF-2758 Not only scientific names are lowercased. This also happens to collectionCode in the next question. *(3) FACETING COLLECTIONCODE VALUES of a single institution fails depending on filtering parameter used to match the institution (code or uuid):* Our institution (uuid= def87a70-0837-11d9-acb2-b8a03c50a862 , institutionCode=SANT) serves datasets from 4 collections, which should sum up more than 100000 records. Why do I get only 2 of our 4 datasets faceted in the following request, which uses our publishingOrg uuid? (uuid should be the preferred option to do this, as code might not be unique for our institution) http://api.gbif.org/v1/occurrence/search?facet=collectionCode&limit=0&publi… Why do I got 4 of 4 if I filter the request using institutionCode instead? (fortunately, nobody else uses the same institutionCode yet, so numbers are correct) http://api.gbif.org/v1/occurrence/search?facet=collectionCode&limit=0&insti… And why do counts differ for the same facet value (sant-lich) in those two requests? (9960 in the 1st request, 10007 in the 2nd one) Why are facet values lowercase again? ("sant-lich" instead of "SANT-Lich") *(4) FACETING SCIENTIFICNAME FAILS FOR SOME DATASETS, but works as expected for others: * More than 1000 faceted Scientificnames returned for our SANT-Lich and SANT-Algae collections. Both of them look correct results: http://api.gbif.org/v1/occurrence/search?facet=ScientificName&limit=0&colle… http://api.gbif.org/v1/occurrence/search?facet=ScientificName&limit=0&colle… But no facets returned for SANT-Bryo (which contains several hundred distinct scientificname values): http://api.gbif.org/v1/occurrence/search?facet=ScientificName&limit=0&colle… And only 7 facets for SANT scientificnames (should be over 10 thousand, as this is by far our largest dataset): http://api.gbif.org/v1/occurrence/search?facet=ScientificName&limit=0&colle… Other than the lowercase facets issue (2), I couldn't reproduce issues 1,3,4 in other institutions datasets. So I wonder if all this is somehow related to a wrong indexing of our IPT. Has anyone else detected these problems? Thanks a lot in advance for your help David -- David García San León Herbario SANT Universidade de Santiago de Compostela

3 4

dataset inflated occurrence counts?
by Stoner, Dan 14 Feb '17

14 Feb '17

Regarding this dataset: http://www.gbif.org/dataset/e0dbf705-cec4-4dce-a152-fc4ebe14674d The count of occurrences in GBIF appears to be (at least) double what is expected. Screenshot: http://imgur.com/a/A9C8M The occurrence count endpoint in the API displays the same value as the GBIF portal: http://api.gbif.org/v1/occurrence/count?datasetKey=e0dbf705-cec4-4dce-a152-… or, 146509 occurrences. However, the actual DwC-A seems to have only a bit over 50k rows. The description for that dataset reads "The vertebrate fossil collection at the Royal Ontario museum consists of over 72,000 catalogued and databased fossil specimens of all vertebrate classes" Further digging indicates that ROM changed their occurrenceID format. A sample that appears to be the same specimen catalog number, same actual specimen, but exists twice in GBIF, once for each occurrenceID format. http://www.gbif.org/occurrence/1211615291 http://www.gbif.org/occurrence/1234628259 Thanks, Dan Stoner iDigBio / ACIS Laboratory University of Florida

1 0

dataset activity api request paged by date
by Herbario SANT 11 Feb '17

11 Feb '17

Hello Where can I find a list of all possible api parameters for this api request?: /occurrence/download/dataset/{datasetKey} It looks similar to the data portal "dataset download activitiy" url, which uses an offset parameter for paging download activity using the links at the end of the page. http://www.gbif.org/dataset/{datasetKey}/activity?offset=10 But ... what other filtering options are there when using the api? I wonder if there is a way to request activity related to a certain date or date range (or to a single year, month). Some examples: year=2016 yearRange=2014,2016 date=2017-01-31 dateRange=2016-12-01,2016-12-31 Of course, I am meaning the "download date", not the "occurrence date". Thanks a lot for your help. -- David García San León Herbario SANT Universidade de Santiago de Compostela

3 3

GBIF Backbone
by godfoder 10 Feb '17

10 Feb '17

iDigBio is nearly complete with the process of integrating the GBIF Backbone into our backend and the process so far has been fairly painless. Thank for all the hard work you've put into building up this resource for the community. I do have one (hopefully minor) change request: Can you add in the parsed values for specificEpithet and infraspecificEpithet into the taxon file where they are available? Background: We're using a multi-term fuzzy match in elasticsearch to resolve taxonomy from records against the backbone, this is performing fairly well, and the more terms we add to the match criteria the better it becomes. The scientificName field in most specimens doesn't include authorship information, so is a relatively poor match overall against the scientificName field in the backbone. Most specimens supply the specificEpithet field though, so I was including it in our matching criteria until I realized that that field is missing from the backbone entirely. Thanks, - Alex

2 2

A couple of questions about the download API
by Doug Palmer 09 Feb '17

09 Feb '17

Hi, I'm a newbie to the GBIF API and I have a couple of questions about downloads. As background, I'm looking at the best way of getting data from GBIF into the Atlas of Living Australia (the bits we don't have already) properly split up into source datasets for attribution. My questions: With the occurrence download limits, am I right in thinking that the limits refer to the number of download requests being processed simultaneously? Can the downloads be of any size in terms of number of records and detail? Instead of being notified by email upon completion, is it possible to poll the status of a download? Doug Palmer

2 1