November 2016 - API-users - lists.gbif.org

Re: [API-users] Is there any NEO4J or graph-based driver for this API ?
by Collins, Matthew 16 Feb '17

16 Feb '17

Jorrit pointed out this thread to us at iDigBio. Downloading and importing data into a relational database will work great, especially if as Jan said you can cut the data size down to a reasonable amount. Another approach we've been working on in a collaboration called GUODA [1] is to build an Apache Spark environment with pre-formatted data frames with common data sets in them for researchers to use. This approach would offer a remote service where you could write arbitrary Spark code, probably in Jupyter notebooks, to iterate over data. Spark does a lot of cool stuff including GraphX which might be of interest. This is definitely pre-alpha at this point and if anyone is interested, I'd like to hear your thoughts. I'll also be at SPNHC talking about this. One thing we've found in working on this is that importing data into a structured data format isn't always easy. If you only want a few columns, it'll be fine. But getting the data typing, format standardization, and column name syntax of the whole width of an iDigBio record right requires some code. I looked to see if EcoData Retriever [2] had a GBIF data source and they have an eBird one that perhaps you might find useful as a starting point if you wanted to try to use someone else's code to download and import data. For other data structures like BHL, we're kind of making stuff up since we're packaging a relational structure and not something nearly as flat as GBIF and DWC stuff. [1] http://guoda.bio/ [2] http://www.ecodataretriever.org/ Matthew Collins Technical Operations Manager Advanced Computing and Information Systems Lab, ECE University of Florida 352-392-5414<callto:352-392-5414> ________________________________ From: jorrit poelen <jhpoelen(a)xs4all.nl> Sent: Monday, May 30, 2016 11:16 AM To: Collins, Matthew; Thompson, Alexander M; Hammock, Jennifer Subject: Fwd: [API-users] Is there any NEO4J or graph-based driver for this API ? Hey y’all: Interesting request below on the GBIF mailing list - sounds like a perfect fit for the GUODA use cases. Would it be too early to jump onto this thread and share our efforts/vision? thx, -jorrit Begin forwarded message: From: Jan Legind <jlegind(a)gbif.org<mailto:jlegind@gbif.org>> Subject: Re: [API-users] Is there any NEO4J or graph-based driver for this API ? Date: May 30, 2016 at 5:48:51 AM PDT To: Mauro Cavalcanti <maurobio(a)gmail.com<mailto:maurobio@gmail.com>>, "Juan M. Escamilla Molgora" <j.escamillamolgora(a)lancaster.ac.uk<mailto:j.escamillamolgora@lancaster.ac.uk>> Cc: "api-users(a)lists.gbif.org<mailto:api-users@lists.gbif.org>" <api-users(a)lists.gbif.org<mailto:api-users@lists.gbif.org>> Dear Juan, Unfortunately we have no tool for creating these kind of SQL like queries to the portal. I am sure you are aware that the filters in the occurrence search pages can be applied in combination in numerous ways. The API can go even further in this regard[1], but it not well suited for retrieving occurrence records since there is a 200.000 records ceiling making it unfit for species exceeding this number. There is going be updates to the pygbif package[2] in the near future that will enable you to launch user downloads programmatically where a whole list of different species can be used as a query parameter as well as adding polygons.[3] In the meantime, Mauro’s suggestion is excellent. If you can narrow your search down until it returns a manageable download (say less than 100 million records), importing this into a database should be doable. From there, you can refine using SQL queries. Best, Jan K. Legind, GBIF Data manager [1] http://www.gbif.org/developer/occurrence#search [2] https://github.com/sckott/pygbif [3] https://github.com/jlegind/GBIF-downloads From: API-users [mailto:api-users-bounces@lists.gbif.org] On Behalf Of Mauro Cavalcanti Sent: 30. maj 2016 14:06 To: Juan M. Escamilla Molgora Cc: api-users(a)lists.gbif.org<mailto:api-users@lists.gbif.org> Subject: Re: [API-users] Is there any NEO4J or graph-based driver for this API ? Hi, One solution I have successfully adopted for this is to download the records (either "manually" via browser or, yet better, using a Python script using the fine pygbif library), storing them into a MySQL or SQLite database and then perform the relational queries. I can provide examples if you are interested. Best regards, 2016-05-30 8:59 GMT-03:00 Juan M. Escamilla Molgora <j.escamillamolgora(a)lancaster.ac.uk<mailto:j.escamillamolgora@lancaster.ac.uk>>: Hola, Is there any API for making relational queries like taxonomy, location or timestamp? Thank you and best wishes Juan _______________________________________________ API-users mailing list API-users(a)lists.gbif.org<mailto:API-users@lists.gbif.org> http://lists.gbif.org/mailman/listinfo/api-users -- Dr. Mauro J. Cavalcanti E-mail: maurobio(a)gmail.com<mailto:maurobio@gmail.com> Web: http://sites.google.com/site/maurobio _______________________________________________ API-users mailing list API-users(a)lists.gbif.org<mailto:API-users@lists.gbif.org> http://lists.gbif.org/mailman/listinfo/api-users

5 11

Backbone data as SQL
by Köhler Christian 22 Nov '16

22 Nov '16

Hi, we are developing an application to curate taxonomic and morphological data for scientists. At the moment we are evaluating different taxonomic backbones to be used within our application. The GIBF taxonomic backbone seems to be an good choice in regards to quality, number of entries and acceptance. Due to the nature of our application, a web service to browse the taxonomy will not fulfil our requirements. A local copy of the GIBF data as SQL would be an ideal solution. I looked for this data publicly available to no avail. "Harvesting" the GBIF rest api seems not a good option. Are there plans to provide current taxonomic backbone data in the future? Maybe the data is already available, but I failed to find it yet. Regards Chris -- Christian Köhler Tel.: 0228 9122-434 Zoologisches Forschungsmuseum Alexander Koenig Leibniz-Institut für Biodiversität der Tiere Adenauerallee 160, 53113 Bonn, Germany www.zfmk.de Stiftung des öffentlichen Rechts Direktor: Prof. J. Wolfgang Wägele Sitz: Bonn -- Zoologisches Forschungsmuseum Alexander Koenig - Leibniz-Institut für Biodiversität der Tiere - Adenauerallee 160, 53113 Bonn, Germany www.zfmk.de Stiftung des öffentlichen Rechts; Direktor: Prof. J. Wolfgang Wägele Sitz: Bonn

6 8

Demo site updated with species pages
by Morten Høfft 11 Nov '16

11 Nov '16

Hi everyone As you know by now, we are developing a new site. Yesterday we released a new version to http://demo.gbif.org The most notably changes in this version is: Species pages http://demo.gbif.org/species/5230740 http://demo.gbif.org/species/5786064 Integration with IUCN and CITES Dataset pages have had another iteration http://demo.gbif.org/dataset/e635240a-3cb1-4d26-ab87-57d8c7afdfdb Option to explore taxonomy on both checklist datasets and occurrence datasets And last, but not least - new maps. You will notice that it shows hexagons. More styles and projections are coming up. This is all still work in development, but should give you an idea of where we are heading. Hope you like it - Morten Høfft

1 0

Early access to a new GBIF.org site
by Tim Robertson 02 Nov '16

02 Nov '16

Hi all, Sent to the GBIF API users and EU BON lists for information - sorry if you receive this more than once I'm happy to let you all know that we recently opened access to a very early version of work which is underway on a new GBIF.org site. It has limited functionality today, but we've tackled the most challenging issue which is cross content search. We're also working on making a consistent search experience across the various content types GBIF deal with. Some examples: - Gallery of images associated occurrences in a search result: http://demo.gbif.org/occurrence/gallery?taxon_key=5190137 - Cross content search: http://demo.gbif.org/search?q=argentina - Improved faceted search of occurrences: http://demo.gbif.org/occurrence/search?month=5&taxon_key=3026341 - Consistent search of species: http://demo.gbif.org/species/search?q=flabellina - Revision to the general news details visuals: http://demo.gbif.org/data-use/newsroom/uses/2015-feuda-et-al - Revised occurrence detail pages: http://demo.gbif.org/occurrence/1233599085 - Work in progress on dataset detail pages: http://demo.gbif.org/dataset/7bd65a7a-f762-11e1-a439-00145eb45e9a - A homepage which is yet-to-be designed, but with a simple search: http://demo.gbif.org The menu structure is provisional now, as is the majority of the styling (much of which needs revision) but I hope this helps give a sense of the direction. We aim to have this near completion by the end of 2016. There are several pages that have begun but are not ready yet, including the species detail page, and the country / participant pages. There will also be a revision to maps to include (just demos with example styling here): http://api.gbif-uat.org/v2/map/demo1.html (pixel maps) http://api.gbif-uat.org/v2/map/demo2.html (hexagon binning) http://api.gbif-uat.org/v2/map/demo6.html (CartoCSS styling example) http://tinyurl.com/gbif-arctic-html (arctic projection - we support WGS84 (EPSG:4326), Arctic (3575), Antarctic (3031) and Mercator (3857) http://s3-eu-west-1.amazonaws.com/gbif-demo/animalia-time.html (year resolution data on the maps instead of decade) http://api.gbif-uat.org/v2/map/debug (diagnostic view, which shows the vector tile format, for browser side rendering) All of this work in the mapping API is to set GBIF up to implement the species trends for all species (currently demo'ed for Lepidoptera only): http://demo.gbif.org/generic/tools/species-population [Maps are the only /v2 API change, and we'll shortly document the changes and will also continue to support /v1 maps in the API. v2 maps have all basisOfRecord, multiple projections, year resolution, and hexagon binning and will be available as PNGs and as Mapbox Vector Tile format] Once those developments are included in a forthcoming release we'll enable the feedback functions and then iterate over the developments releasing regularly with updates. The work is on GitHub [1], and primarily developed in Node with AngluarJS using the GBIF APIs. We will be deploying all of this on HTTPS and are exploring HTTP2 now to increase performance. Many thanks, Tim [1] https://github.com/gbif/portal16

1 0