January 2017 - API-users - lists.gbif.org

Re: [API-users] Is there any NEO4J or graph-based driver for this API ?
by Collins, Matthew 15 Feb '17

15 Feb '17

Jorrit pointed out this thread to us at iDigBio. Downloading and importing data into a relational database will work great, especially if as Jan said you can cut the data size down to a reasonable amount. Another approach we've been working on in a collaboration called GUODA [1] is to build an Apache Spark environment with pre-formatted data frames with common data sets in them for researchers to use. This approach would offer a remote service where you could write arbitrary Spark code, probably in Jupyter notebooks, to iterate over data. Spark does a lot of cool stuff including GraphX which might be of interest. This is definitely pre-alpha at this point and if anyone is interested, I'd like to hear your thoughts. I'll also be at SPNHC talking about this. One thing we've found in working on this is that importing data into a structured data format isn't always easy. If you only want a few columns, it'll be fine. But getting the data typing, format standardization, and column name syntax of the whole width of an iDigBio record right requires some code. I looked to see if EcoData Retriever [2] had a GBIF data source and they have an eBird one that perhaps you might find useful as a starting point if you wanted to try to use someone else's code to download and import data. For other data structures like BHL, we're kind of making stuff up since we're packaging a relational structure and not something nearly as flat as GBIF and DWC stuff. [1] http://guoda.bio/ [2] http://www.ecodataretriever.org/ Matthew Collins Technical Operations Manager Advanced Computing and Information Systems Lab, ECE University of Florida 352-392-5414<callto:352-392-5414> ________________________________ From: jorrit poelen <jhpoelen(a)xs4all.nl> Sent: Monday, May 30, 2016 11:16 AM To: Collins, Matthew; Thompson, Alexander M; Hammock, Jennifer Subject: Fwd: [API-users] Is there any NEO4J or graph-based driver for this API ? Hey y’all: Interesting request below on the GBIF mailing list - sounds like a perfect fit for the GUODA use cases. Would it be too early to jump onto this thread and share our efforts/vision? thx, -jorrit Begin forwarded message: From: Jan Legind <jlegind(a)gbif.org<mailto:jlegind@gbif.org>> Subject: Re: [API-users] Is there any NEO4J or graph-based driver for this API ? Date: May 30, 2016 at 5:48:51 AM PDT To: Mauro Cavalcanti <maurobio(a)gmail.com<mailto:maurobio@gmail.com>>, "Juan M. Escamilla Molgora" <j.escamillamolgora(a)lancaster.ac.uk<mailto:j.escamillamolgora@lancaster.ac.uk>> Cc: "api-users(a)lists.gbif.org<mailto:api-users@lists.gbif.org>" <api-users(a)lists.gbif.org<mailto:api-users@lists.gbif.org>> Dear Juan, Unfortunately we have no tool for creating these kind of SQL like queries to the portal. I am sure you are aware that the filters in the occurrence search pages can be applied in combination in numerous ways. The API can go even further in this regard[1], but it not well suited for retrieving occurrence records since there is a 200.000 records ceiling making it unfit for species exceeding this number. There is going be updates to the pygbif package[2] in the near future that will enable you to launch user downloads programmatically where a whole list of different species can be used as a query parameter as well as adding polygons.[3] In the meantime, Mauro’s suggestion is excellent. If you can narrow your search down until it returns a manageable download (say less than 100 million records), importing this into a database should be doable. From there, you can refine using SQL queries. Best, Jan K. Legind, GBIF Data manager [1] http://www.gbif.org/developer/occurrence#search [2] https://github.com/sckott/pygbif [3] https://github.com/jlegind/GBIF-downloads From: API-users [mailto:api-users-bounces@lists.gbif.org] On Behalf Of Mauro Cavalcanti Sent: 30. maj 2016 14:06 To: Juan M. Escamilla Molgora Cc: api-users(a)lists.gbif.org<mailto:api-users@lists.gbif.org> Subject: Re: [API-users] Is there any NEO4J or graph-based driver for this API ? Hi, One solution I have successfully adopted for this is to download the records (either "manually" via browser or, yet better, using a Python script using the fine pygbif library), storing them into a MySQL or SQLite database and then perform the relational queries. I can provide examples if you are interested. Best regards, 2016-05-30 8:59 GMT-03:00 Juan M. Escamilla Molgora <j.escamillamolgora(a)lancaster.ac.uk<mailto:j.escamillamolgora@lancaster.ac.uk>>: Hola, Is there any API for making relational queries like taxonomy, location or timestamp? Thank you and best wishes Juan _______________________________________________ API-users mailing list API-users(a)lists.gbif.org<mailto:API-users@lists.gbif.org> http://lists.gbif.org/mailman/listinfo/api-users -- Dr. Mauro J. Cavalcanti E-mail: maurobio(a)gmail.com<mailto:maurobio@gmail.com> Web: http://sites.google.com/site/maurobio _______________________________________________ API-users mailing list API-users(a)lists.gbif.org<mailto:API-users@lists.gbif.org> http://lists.gbif.org/mailman/listinfo/api-users

5 11

Interest for a mock server?
by Nicolas Noé 24 Jan '17

24 Jan '17

Hi everyone and best wishes! Over the years, I have several times written code that consumes the GBIF API. So far, proper unit testing of this code has been problematic, because I had to test it against the "real" GBIF API, with several drawbacks (fragility, test needs network connectivity, speed issues, ...). I think other libraries (pygbif for example) probably face the same problems. I am therefore thinking about creating a GBIF API mock server, that could be run locally and mimicks requests/responses cycles of api.gbif.org, to make testing our tools in isolation easier. It would have the following characteristics: * Be a (localhost-runnable) server rather than a mock library, so it can be used by code in any language * Mock the responses from api.gbif.org (starting with the most common stuff: read-only queries on occurrences, registry, ...) * Be super easy to install and run locally I'm thinking of implementing it in Golang, since it seems to fit those requirements quite well and I'd like to build real-world experience with this technology. Ideally, I'd like to write that is not only useful for myself, so here is my question: would you be interested in using such a mock server for your developments? Or isn't that so usefull? Maybe you know existing tools that can already be used to solve this problem? Thanks for your feedback! Nicolas

1 0

Download API: support many predicates?
by Scott Chamberlain 14 Jan '17

14 Jan '17

Hi! A rgbif user is asking me about a particular download query, see https://github.com/ropensci/rgbif/issues/245 TLDR: AFAICT this can't work as the API is set up right now. Or am I wrong? That is, he wants to get data for records that match - 10 taxa with their taxonKey value - 3 different basisofrecord values - country = US - hasCoordinate = TRUE - hasGeospatialIssue = FALSE - year range between 1949 and 2011 - month range between 3 and 8 For the taxonKey values, I think he needs to combine those with *OR* operator: "predicate": { "type": "or", "predicates": [ { "type": "equals", "key": "TAXON_KEY", "value": "2475470" }, { "type": "equals", "key": "TAXON_KEY", "value": "2475470" } ] } However, he should combine the other parameters (basisofrecord, hasCoordinate, year, month, etc) using *AND* AFAICT. *BUT*, AFAICT the API only allows one *predicate* key so you can't do what I think is needed here. That is, to say (taxonkey = 1 OR 2 OR 3) + (country = US AND hasCoordinate = true, etc.) I tried sending two *predicate* keys in the API and only the second goes through. Any guidance much appreciated :) The best solution I can think of is to do the query he's doing, but do one for each taxon. Seem like a good solution? Best, Scott

2 2

Secure (HTTPS) access to the GBIF API
by Matthew Blissett 10 Jan '17

10 Jan '17

Hi all, I'm pleased to announce that the GBIF API is now accessible securely — simply replace http:// with https://. http://api.gbif.org/v1/occurrence/search → https://api.gbif.org/v1/occurrence/search This is necessary for anyone embedding the API in a secure website, or where organizational policy requires using HTTPS. Using the secure API is optional, the non-secure HTTP version will continue to work as before. However, the new GBIF.org website <https://demo.gbif.org/> will only be accessible using HTTPS, as this will soon be a requirement¹ for any website accepting passwords. Please let us know if you encounter any problems with this, Many thanks, Matthew Blissett ¹ https://security.googleblog.com/2016/09/moving-towards-more-secure-web.html and https://blog.mozilla.org/tanvi/2016/01/28/no-more-passwords-over-http-pleas…

1 0