June 2016 - API-users - lists.gbif.org

Re: [API-users] Is there any NEO4J or graph-based driver for this API ?
by Collins, Matthew 15 Feb '17

15 Feb '17

Jorrit pointed out this thread to us at iDigBio. Downloading and importing data into a relational database will work great, especially if as Jan said you can cut the data size down to a reasonable amount. Another approach we've been working on in a collaboration called GUODA [1] is to build an Apache Spark environment with pre-formatted data frames with common data sets in them for researchers to use. This approach would offer a remote service where you could write arbitrary Spark code, probably in Jupyter notebooks, to iterate over data. Spark does a lot of cool stuff including GraphX which might be of interest. This is definitely pre-alpha at this point and if anyone is interested, I'd like to hear your thoughts. I'll also be at SPNHC talking about this. One thing we've found in working on this is that importing data into a structured data format isn't always easy. If you only want a few columns, it'll be fine. But getting the data typing, format standardization, and column name syntax of the whole width of an iDigBio record right requires some code. I looked to see if EcoData Retriever [2] had a GBIF data source and they have an eBird one that perhaps you might find useful as a starting point if you wanted to try to use someone else's code to download and import data. For other data structures like BHL, we're kind of making stuff up since we're packaging a relational structure and not something nearly as flat as GBIF and DWC stuff. [1] http://guoda.bio/ [2] http://www.ecodataretriever.org/ Matthew Collins Technical Operations Manager Advanced Computing and Information Systems Lab, ECE University of Florida 352-392-5414<callto:352-392-5414> ________________________________ From: jorrit poelen <jhpoelen(a)xs4all.nl> Sent: Monday, May 30, 2016 11:16 AM To: Collins, Matthew; Thompson, Alexander M; Hammock, Jennifer Subject: Fwd: [API-users] Is there any NEO4J or graph-based driver for this API ? Hey y’all: Interesting request below on the GBIF mailing list - sounds like a perfect fit for the GUODA use cases. Would it be too early to jump onto this thread and share our efforts/vision? thx, -jorrit Begin forwarded message: From: Jan Legind <jlegind(a)gbif.org<mailto:jlegind@gbif.org>> Subject: Re: [API-users] Is there any NEO4J or graph-based driver for this API ? Date: May 30, 2016 at 5:48:51 AM PDT To: Mauro Cavalcanti <maurobio(a)gmail.com<mailto:maurobio@gmail.com>>, "Juan M. Escamilla Molgora" <j.escamillamolgora(a)lancaster.ac.uk<mailto:j.escamillamolgora@lancaster.ac.uk>> Cc: "api-users(a)lists.gbif.org<mailto:api-users@lists.gbif.org>" <api-users(a)lists.gbif.org<mailto:api-users@lists.gbif.org>> Dear Juan, Unfortunately we have no tool for creating these kind of SQL like queries to the portal. I am sure you are aware that the filters in the occurrence search pages can be applied in combination in numerous ways. The API can go even further in this regard[1], but it not well suited for retrieving occurrence records since there is a 200.000 records ceiling making it unfit for species exceeding this number. There is going be updates to the pygbif package[2] in the near future that will enable you to launch user downloads programmatically where a whole list of different species can be used as a query parameter as well as adding polygons.[3] In the meantime, Mauro’s suggestion is excellent. If you can narrow your search down until it returns a manageable download (say less than 100 million records), importing this into a database should be doable. From there, you can refine using SQL queries. Best, Jan K. Legind, GBIF Data manager [1] http://www.gbif.org/developer/occurrence#search [2] https://github.com/sckott/pygbif [3] https://github.com/jlegind/GBIF-downloads From: API-users [mailto:api-users-bounces@lists.gbif.org] On Behalf Of Mauro Cavalcanti Sent: 30. maj 2016 14:06 To: Juan M. Escamilla Molgora Cc: api-users(a)lists.gbif.org<mailto:api-users@lists.gbif.org> Subject: Re: [API-users] Is there any NEO4J or graph-based driver for this API ? Hi, One solution I have successfully adopted for this is to download the records (either "manually" via browser or, yet better, using a Python script using the fine pygbif library), storing them into a MySQL or SQLite database and then perform the relational queries. I can provide examples if you are interested. Best regards, 2016-05-30 8:59 GMT-03:00 Juan M. Escamilla Molgora <j.escamillamolgora(a)lancaster.ac.uk<mailto:j.escamillamolgora@lancaster.ac.uk>>: Hola, Is there any API for making relational queries like taxonomy, location or timestamp? Thank you and best wishes Juan _______________________________________________ API-users mailing list API-users(a)lists.gbif.org<mailto:API-users@lists.gbif.org> http://lists.gbif.org/mailman/listinfo/api-users -- Dr. Mauro J. Cavalcanti E-mail: maurobio(a)gmail.com<mailto:maurobio@gmail.com> Web: http://sites.google.com/site/maurobio _______________________________________________ API-users mailing list API-users(a)lists.gbif.org<mailto:API-users@lists.gbif.org> http://lists.gbif.org/mailman/listinfo/api-users

5 11

Crawling paused and new taxonomic backbone
by Tim Robertson 08 Aug '16

08 Aug '16

Dear GBIF API users, We are pleased to announce that we are preparing to move a new taxonomic backbone into production. This is the first revision of the backbone for quite some years, which will bring the GBIF backbone up to date with latest data sources, such as the Catalogue of Life 2016 (January edition) along with fixes for the many issues you have reported since 2013. You can read more about the backbone taxonomy rewriting on http://gbif.blogspot.com/2016/04/updating-gbif-backbone.html This requires some significant processing within the GBIF.org system as we'll be rerunning all quality control routines on all data - this means other improvements may also take effect across older data. While GBIF.org will remain operational to users during this time, crawling will be paused. We expect to start this process today, and during next week the changes will be visible in production, after which crawling will resume. Thank you all for your patience, and providing the data, feedback and submitting the issues that have allowed this. We'll report here as changes come into effect. More soon, The GBIF dev team

7 12

New Portal/Occurrece API functionalities
by Federico Mendez 20 Jun '16

20 Jun '16

Dear GBIFApiers, we are excited to announce some new features that are now supported by there GBIF Portal and Occurrence API: 1. Full text search is now supported. Some examples of this new function are: * Search by collector (recorded by): * http://api.gbif.org/v1/occurrence/search?q=hobern<http://www.gbif.org/occurrence/search?q=hobern> * http://api.gbif.org/v1/occurrence/search?q=desmet * Search by locality shows accurate results for most of the cases: http://api.gbif.org/v1/occurrence/search?q=santa%20ana * This service provides suggestions when a 'similar/corrected' version of the query could show more accurate results: http://api.gbif.org/v1/occurrence/search?q=kajsdkla&spellCheck=true . * This functionality is also supported by the GBIF portal: http://www.gbif.org/occurrence/search?q=cobra<http://www.gbif.org/occurrence/search> 2. Faceted search * The GBIF portal now supports faceted navigation by: basis of record, type status, data set, country, year and month. Please, see this example: http://www.gbif.org/occurrence/search?basis_of_record=FOSSIL_SPECIMEN . * The GBIF API supports faceted results by: basisOfRecord, catalogNumber, classKey, collectionCode, country, datasetKey, decimalLatitude, decimalLongitude, depth, elevation, establishmentMeans, familyKey, genusKey, hasCoordinate, hasGeospatialIssue, institutionCode, issue, kingdomKey, mediaType, month, occurrenceID, orderKey, phylumKey, publishingCountry, recordedBy, recordNumber, repatriated, scientificName, speciesKey, subgenusKey, taxonKey, typeStatus and year. * The occurrence API supports pagination request in facets using the parameters facetLimit and facetOffset, see this example: * First three country counts/facets: http://api.gbif.org/v1/occurrence/search?facet=COUNTRY&COUNTRY.facetLimit=3… * Next three country counts/facets: http://api.gbif.org/v1/occurrence/search?facet=COUNTRY&COUNTRY.facetLimit=3… 3. A new repatriation filter allows to search for records whose publishing country is different to the country where the record was recorded in. For example, the following link shows the repatriated data of Costa Rica:http://api.gbif.org/v1/occurrence/search?COUNTRY=CR&REPATRIATED=true . Be aware that this is still in a experimental phase and its definition could change in the future. Please refer to the API documentation http://www.gbif.org/developer/occurrence and don't hesitate to contact us if you have further questions! Best regards, GBIFS Informatics.

7 11

birdhouse meets GBIF
by Nils Hempelmann 08 Jun '16

08 Jun '16

Dear all Here is an current problem to solve :-) for the birdhouse WPS and GBIF developers. There is a species distribution process in the birdhouse WPS, processing Climate data based on occurrence coordinates from GBIF. Currently the GBIF data are given to the process via an url of the GBIF csv (which had to be generated manually before). I was implementing a direct occurrence search from birdhouse to GBIF with pygbif. Worked fine BUT: there is a limit of 300 records which is far to less to train the Species distribution model. So here the question: Is there a way to request the species occurrence coordinates somehow directly (e.g. like a WPS request)? (The previous conversation is following) And some link for the background informations: Birdhouse: http://birdhouse.readthedocs.io/en/latest/ GBIF: http://www.gbif.org/ Species distribution process: http://flyingpigeon.readthedocs.io/en/latest/tutorials/sdm.html Birdhouse architecture: https://github.com/bird-house/birdhouse-docs/blob/master/slides/birdhouse-a… Merci Nils -------- Forwarded Message ------- Subject: Re: pygbif for occurrence coordinates Date: Thu, 2 Jun 2016 09:00:49 -0300 From: Mauro Cavalcanti <maurobio(a)gmail.com> To: Nils Hempelmann <info(a)nilshempelmann.de> CC: wps-dev(a)lists.dkrz.de, Carsten Ehbrecht <ehbrecht(a)dkrz.de>, Wolfgang.Falk(a)lwf.bayern.de, Scott Chamberlain <myrmecocystus(a)gmail.com> Nils, No, there is no other way to get at once a larger number of records using the GBIF API (although one could be able to achieve this by sequentially querying the server using "batches" of 200 or 300 records each). As I said, there are good operational reasons for the GBIF developers to have imposed such limit. As of your other question, it should better be put to the GBIF developers themselves (in the discussion list, so that we can all benefit from the answers! :-)) With warmest regards, -- Dr. Mauro J. Cavalcanti E-mail: maurobio(a)gmail.com <mailto:maurobio@gmail.com> Web: http://sites.google.com/site/maurobio Em 02/06/2016 08:49, "Nils Hempelmann" <info(a)nilshempelmann.de <mailto:info@nilshempelmann.de>> escreveu: Hi Mauro Oh ... No way to set it unlimited? The 'manual via browser' option and parsing the returning csv is the current status. or any alternative to pygbif? If I understood it correctly, GBIF database is organized as Web Server, so I gues there should be a way to connect to the birdhouse WPS, am I right? (put the web-dev list in copy) Merci Nils On 02/06/2016 12:40, Mauro Cavalcanti wrote: > > Nils, > > That's a limit imposed (for good operational reasons) by the GBIF > API. If you want a larger number of records, you'll have to > download them "manually" (that is, via browser) and then parse > locally the csv file returned from the GBIF server. > > Hope this helps. > > Best regards, > > -- > Dr. Mauro J. Cavalcanti > E-mail: maurobio(a)gmail.com <mailto:maurobio@gmail.com> > Web: http://sites.google.com/site/maurobio > > Em 02/06/2016 04:48, "Nils Hempelmann" <info(a)nilshempelmann.de > <mailto:info@nilshempelmann.de>> escreveu: > > Hi Scott > > works fine. Thanks a lot. Just have a question for the search > limits: > > The maximal records seems to be limited to 300. Is that on > purpose? > And requesting more than 200000 gives a 'bad request' > > Merci > Nils > > > In [68]: len( occurrences.search(taxonKey=key, > limit=100)['results']) > Out[68]: 100 > > In [69]: len( occurrences.search(taxonKey=key, > limit=300)['results']) > Out[69]: 300 > > In [70]: len( occurrences.search(taxonKey=key, > limit=3000)['results']) > Out[70]: 300 > > In [71]: len( occurrences.search(taxonKey=key, > limit=200000)['results']) > Out[71]: 300 > > In [72]: len( occurrences.search(taxonKey=key, > limit=200001)['results']) > --------------------------------------------------------------------------- > HTTPError Traceback (most > recent call last) > <ipython-input-72-2f7d7b4ccba0> in <module>() > ----> 1 len( occurrences.search(taxonKey=key, > limit=200001)['results']) > > /home/nils/.conda/envs/birdhouse/lib/python2.7/site-packages/pygbif/occurrences/search.pyc > in search(taxonKey, scientificName, country, > publishingCountry, hasCoordinate, typeStatus, recordNumber, > lastInterpreted, continent, geometry, recordedBy, > basisOfRecord, datasetKey, eventDate, catalogNumber, year, > month, decimalLatitude, decimalLongitude, elevation, depth, > institutionCode, collectionCode, hasGeospatialIssue, issue, q, > mediatype, limit, offset, **kwargs) > 251 'collectionCode': collectionCode, > 'hasGeospatialIssue': hasGeospatialIssue, > 252 'issue': issue, 'q': q, 'mediatype': > mediatype, 'limit': limit, > --> 253 'offset': offset}, **kwargs) > 254 return out > > /home/nils/.conda/envs/birdhouse/lib/python2.7/site-packages/pygbif/gbifutils.pyc > in gbif_GET(url, args, **kwargs) > 17 def gbif_GET(url, args, **kwargs): > 18 out = requests.get(url, params=args, > headers=make_ua(), **kwargs) > ---> 19 out.raise_for_status() > 20 stopifnot(out.headers['content-type']) > 21 return out.json() > > /home/nils/.conda/envs/birdhouse/lib/python2.7/site-packages/requests/models.pyc > in raise_for_status(self) > 842 > 843 if http_error_msg: > --> 844 raise HTTPError(http_error_msg, response=self) > 845 > 846 def close(self): > > HTTPError: 400 Client Error: Bad Request for url: > http://api.gbif.org/v1/occurrence/search?taxonKey=2882316&limit=200001&offs… > > In [73]: > > > On 02/06/2016 02:22, Scott Chamberlain wrote: >> Fixes to docs and fix to download_get are up on Github now. >> Will push new version to pypi soon. Let me know if some thing >> still don't work for you after reinstalling from github >> >> S >> >> On Wed, Jun 1, 2016 at 4:51 PM Nils Hempelmann >> <info(a)nilshempelmann.de <mailto:info@nilshempelmann.de>> wrote: >> >> Hi Scott and Mauro >> >> Mauro sended me a snipped of code which worked fine: >> https://github.com/bird-house/flyingpigeon/blob/develop/scripts/pygbif_occu… >> >> I found the example here: >> https://github.com/maurobio/pygbif#occurrence-data >> >> and here: >> https://github.com/sckott/pygbif#occurrences-module >> >> Thanks a lot very great. That enables a lot ;-) >> I ll keep you posted >> >> merci >> Nils >> >> >> >> >> >> On 02/06/2016 01:45, Scott Chamberlain wrote: >>> And where are those example from exactly? I don't see >>> those examples searching the repo (which includes all >>> docs). >>> >>> `pygbif.name_suggest` wouldn't work because >>> `name_suggest` is a method in the `species` module, so >>> you'd have to do pygbif.species.name_suggest, or `from >>> pygbif import species`, then `species.name_suggest` >>> >>> Looks like occ.get(taxonKey = 252408386) is a >>> documentation bug, that should be `key` instead of >>> `taxonKey`, a copy paste error. will fix that. >>> >>> The `occ.download_get` call has a small bug, will fix that >>> >>> All other calls work for me >>> >>> S >>> >>> On Wed, Jun 1, 2016 at 4:34 PM Scott Chamberlain >>> <myrmecocystus(a)gmail.com >>> <mailto:myrmecocystus@gmail.com>> wrote: >>> >>> Hi, >>> >>> What version of pygbif are you on? And what version >>> of Python> >>> >>> Best, Scott >>> >>> On Wed, Jun 1, 2016 at 4:02 PM Nils Hempelmann >>> <info(a)nilshempelmann.de >>> <mailto:info@nilshempelmann.de>> wrote: >>> >>> Hi Mauro and Scott >>> >>> Was checking out pygbif. seems to be a very >>> useful tool. >>> >>> Can you help me with the syntax (or forward me >>> to the appropriate person >>> ;-) ) >>> The given snippets of code are outdated. >>> >>> I am basically just looking for the occurrence >>> coordinates: >>> here is my first try : >>> >>> import pygbif >>> occ = >>> pygbif.occurrences.search(scientificName='Fagus >>> sylvatica') >>> occ['count'] >>> >>> ... and further? ;-) >>> >>> the examples in the docu are throwing errors: >>> >>> key= pygbif.name_suggest(q='Helianthus >>> annuus',rank='species')['key'] >>> pygbif.search(taxonKey=key[0]['key'],limit=2) >>> >>> from pygbif import occurrences as occ >>> occ.search(taxonKey = 3329049) >>> occ.get(taxonKey = 252408386) >>> occ.count(isGeoreferenced = True) >>> occ.download_list(user = "sckott", limit = 5) >>> occ.download_meta(key = "0000099-140929101555934") >>> occ.download_get("0000099-140929101555934") >>> >>> >>> Thanks >>> Nils >>> >> >

4 8