May 2016 - API-users - lists.gbif.org

Re: [API-users] Is there any NEO4J or graph-based driver for this API ?
by Collins, Matthew 15 Feb '17

15 Feb '17

Jorrit pointed out this thread to us at iDigBio. Downloading and importing data into a relational database will work great, especially if as Jan said you can cut the data size down to a reasonable amount. Another approach we've been working on in a collaboration called GUODA [1] is to build an Apache Spark environment with pre-formatted data frames with common data sets in them for researchers to use. This approach would offer a remote service where you could write arbitrary Spark code, probably in Jupyter notebooks, to iterate over data. Spark does a lot of cool stuff including GraphX which might be of interest. This is definitely pre-alpha at this point and if anyone is interested, I'd like to hear your thoughts. I'll also be at SPNHC talking about this. One thing we've found in working on this is that importing data into a structured data format isn't always easy. If you only want a few columns, it'll be fine. But getting the data typing, format standardization, and column name syntax of the whole width of an iDigBio record right requires some code. I looked to see if EcoData Retriever [2] had a GBIF data source and they have an eBird one that perhaps you might find useful as a starting point if you wanted to try to use someone else's code to download and import data. For other data structures like BHL, we're kind of making stuff up since we're packaging a relational structure and not something nearly as flat as GBIF and DWC stuff. [1] http://guoda.bio/ [2] http://www.ecodataretriever.org/ Matthew Collins Technical Operations Manager Advanced Computing and Information Systems Lab, ECE University of Florida 352-392-5414<callto:352-392-5414> ________________________________ From: jorrit poelen <jhpoelen(a)xs4all.nl> Sent: Monday, May 30, 2016 11:16 AM To: Collins, Matthew; Thompson, Alexander M; Hammock, Jennifer Subject: Fwd: [API-users] Is there any NEO4J or graph-based driver for this API ? Hey y’all: Interesting request below on the GBIF mailing list - sounds like a perfect fit for the GUODA use cases. Would it be too early to jump onto this thread and share our efforts/vision? thx, -jorrit Begin forwarded message: From: Jan Legind <jlegind(a)gbif.org<mailto:jlegind@gbif.org>> Subject: Re: [API-users] Is there any NEO4J or graph-based driver for this API ? Date: May 30, 2016 at 5:48:51 AM PDT To: Mauro Cavalcanti <maurobio(a)gmail.com<mailto:maurobio@gmail.com>>, "Juan M. Escamilla Molgora" <j.escamillamolgora(a)lancaster.ac.uk<mailto:j.escamillamolgora@lancaster.ac.uk>> Cc: "api-users(a)lists.gbif.org<mailto:api-users@lists.gbif.org>" <api-users(a)lists.gbif.org<mailto:api-users@lists.gbif.org>> Dear Juan, Unfortunately we have no tool for creating these kind of SQL like queries to the portal. I am sure you are aware that the filters in the occurrence search pages can be applied in combination in numerous ways. The API can go even further in this regard[1], but it not well suited for retrieving occurrence records since there is a 200.000 records ceiling making it unfit for species exceeding this number. There is going be updates to the pygbif package[2] in the near future that will enable you to launch user downloads programmatically where a whole list of different species can be used as a query parameter as well as adding polygons.[3] In the meantime, Mauro’s suggestion is excellent. If you can narrow your search down until it returns a manageable download (say less than 100 million records), importing this into a database should be doable. From there, you can refine using SQL queries. Best, Jan K. Legind, GBIF Data manager [1] http://www.gbif.org/developer/occurrence#search [2] https://github.com/sckott/pygbif [3] https://github.com/jlegind/GBIF-downloads From: API-users [mailto:api-users-bounces@lists.gbif.org] On Behalf Of Mauro Cavalcanti Sent: 30. maj 2016 14:06 To: Juan M. Escamilla Molgora Cc: api-users(a)lists.gbif.org<mailto:api-users@lists.gbif.org> Subject: Re: [API-users] Is there any NEO4J or graph-based driver for this API ? Hi, One solution I have successfully adopted for this is to download the records (either "manually" via browser or, yet better, using a Python script using the fine pygbif library), storing them into a MySQL or SQLite database and then perform the relational queries. I can provide examples if you are interested. Best regards, 2016-05-30 8:59 GMT-03:00 Juan M. Escamilla Molgora <j.escamillamolgora(a)lancaster.ac.uk<mailto:j.escamillamolgora@lancaster.ac.uk>>: Hola, Is there any API for making relational queries like taxonomy, location or timestamp? Thank you and best wishes Juan _______________________________________________ API-users mailing list API-users(a)lists.gbif.org<mailto:API-users@lists.gbif.org> http://lists.gbif.org/mailman/listinfo/api-users -- Dr. Mauro J. Cavalcanti E-mail: maurobio(a)gmail.com<mailto:maurobio@gmail.com> Web: http://sites.google.com/site/maurobio _______________________________________________ API-users mailing list API-users(a)lists.gbif.org<mailto:API-users@lists.gbif.org> http://lists.gbif.org/mailman/listinfo/api-users

5 11

Crawling paused and new taxonomic backbone
by Tim Robertson 08 Aug '16

08 Aug '16

Dear GBIF API users, We are pleased to announce that we are preparing to move a new taxonomic backbone into production. This is the first revision of the backbone for quite some years, which will bring the GBIF backbone up to date with latest data sources, such as the Catalogue of Life 2016 (January edition) along with fixes for the many issues you have reported since 2013. You can read more about the backbone taxonomy rewriting on http://gbif.blogspot.com/2016/04/updating-gbif-backbone.html This requires some significant processing within the GBIF.org system as we'll be rerunning all quality control routines on all data - this means other improvements may also take effect across older data. While GBIF.org will remain operational to users during this time, crawling will be paused. We expect to start this process today, and during next week the changes will be visible in production, after which crawling will resume. Thank you all for your patience, and providing the data, feedback and submitting the issues that have allowed this. We'll report here as changes come into effect. More soon, The GBIF dev team

7 12

Re: [API-users] Is there any NEO4J or graph-based driver for this API ?
by Jan Legind 31 May '16

31 May '16

Dear Juan, Unfortunately we have no tool for creating these kind of SQL like queries to the portal. I am sure you are aware that the filters in the occurrence search pages can be applied in combination in numerous ways. The API can go even further in this regard[1], but it not well suited for retrieving occurrence records since there is a 200.000 records ceiling making it unfit for species exceeding this number. There is going be updates to the pygbif package[2] in the near future that will enable you to launch user downloads programmatically where a whole list of different species can be used as a query parameter as well as adding polygons.[3] In the meantime, Mauro’s suggestion is excellent. If you can narrow your search down until it returns a manageable download (say less than 100 million records), importing this into a database should be doable. From there, you can refine using SQL queries. Best, Jan K. Legind, GBIF Data manager [1] http://www.gbif.org/developer/occurrence#search [2] https://github.com/sckott/pygbif [3] https://github.com/jlegind/GBIF-downloads From: API-users [mailto:api-users-bounces@lists.gbif.org] On Behalf Of Mauro Cavalcanti Sent: 30. maj 2016 14:06 To: Juan M. Escamilla Molgora Cc: api-users(a)lists.gbif.org Subject: Re: [API-users] Is there any NEO4J or graph-based driver for this API ? Hi, One solution I have successfully adopted for this is to download the records (either "manually" via browser or, yet better, using a Python script using the fine pygbif library), storing them into a MySQL or SQLite database and then perform the relational queries. I can provide examples if you are interested. Best regards, 2016-05-30 8:59 GMT-03:00 Juan M. Escamilla Molgora <j.escamillamolgora(a)lancaster.ac.uk<mailto:j.escamillamolgora@lancaster.ac.uk>>: Hola, Is there any API for making relational queries like taxonomy, location or timestamp? Thank you and best wishes Juan _______________________________________________ API-users mailing list API-users(a)lists.gbif.org<mailto:API-users@lists.gbif.org> http://lists.gbif.org/mailman/listinfo/api-users -- Dr. Mauro J. Cavalcanti E-mail: maurobio(a)gmail.com<mailto:maurobio@gmail.com> Web: http://sites.google.com/site/maurobio

3 2

Is there any NEO4J or graph-based driver for this API ?
by Juan M. Escamilla Molgora 30 May '16

30 May '16

Hola, Is there any API for making relational queries like taxonomy, location or timestamp? Thank you and best wishes Juan

2 1

timeframe on implementation of dataset "subtype" parameter?
by Stoner, Dan 28 May '16

28 May '16

Hi, Just wondering if dataset subtype will become available in the API anytime soon. This would be very helpful to me as I am looking for SPECIMEN occurrence datasets. The doc that mentions "subtype": http://www.gbif.org/developer/registry#datasetSearch Description: "Not yet implemented, but will eventually allow filtering of datasets by their dataset subtype as given in our DatasetSubtype enum." DatasetSubtype: http://gbif.github.io/gbif-api/apidocs/org/gbif/api/vocabulary/DatasetSubty… Thanks, Dan Stoner iDigBio / ACIS Laboratory University of Florida

2 1

Scientific names questions
by Scott Chamberlain 13 May '16

13 May '16

HI all, Not sure where is best to ask this... so here goes. Let me know if there's a better place. The following are examples some users have highlighted for me as leading to confusion when searching for taxa. 1. Macrozamia platyrachis (http://www.gbif.org/species/4928834) vs. Macrozamia platyrhachis (http://www.gbif.org/species/2683551) Here, the two spellings (with/without h) are accepted, and exact matches. The sci. authority seems to differ with F. M. Bailey vs. F.M.Bailey. The first is from GRIN taxonomy and the second from COL. Anyway, for users e.g., of the R client, this is a bit confusing. I had thought the backbone taxonomy would only have one master taxon key and name for each real taxon, but here it seems like there's two? 2. Cycas circinalis (http://www.gbif.org/species/2683264 ) vs. Cycas circinnalis (http://www.gbif.org/species/3594916 ) Here, the two spellings (with 1 or 2 "n"'s) are accepted, and exact matches. The sci. authorities here are exactly the same. The first is from COL and the second from IPNI taxonomy. 3. Isolona perrieri (http://www.gbif.org/species/3648546 ) vs Isolona perrierii (http://www.gbif.org/species/6308376 ) Here, the two spellings (with 1 or 2 "i"'s) are accepted, and exact matches. The sci. authorities here are exactly the same. The first is from TPL and the second from COL -------- Should I advise users to when searching on the backbone taxonomy to limit to COL to avoid any confusion about names? Best, Scott Chamberlain

3 4